Hybridrenderer with VR

I have a grocery store full of items that the user can pick up in VR. I'm trying to convert my items into entities to get better rendering performance. I'm using ConvertToEntity with Convert and inject GameObject. I need the original prefab to hang around because it has all the VR interaction scripts on it as well as my physics. I then keep the entity positioned with the prefab using CopyTransformFromGameObject. This seems to be working however I don't want to show the renderer of the prefab because now it's being rendered twice. Once as a prefab and once as an entity. If I try to turn the MeshRenderer off via a script on the prefab when the entity is created the entity ends up without a renderer. I can get around this by using a coroutine with a WaitForSeconds(1) before deactivating the MeshRenderer. However in the profiler I see a ton of mesh memory usage until the MeshRenderer is deactivated. Is there a better way to go about this?