Why are cross-(sub)scene references not allowed? (Or why can't I reference pre-converted entities?)

I've been trying to get an entity conversion working for a while now, and have had to use pretty dirty/hacky, badly performing solutions to get a reference to an entity within a subscene outside of that subscene (I need an entity reference in a MonoBehaviour). Currently, a subscene is seemingly the only way to convert an entity before runtime and seems to be used in a way that prefabs currently are (for example, in the megacity demo everything is a subscene IIRC), but I'm not really sure why it wouldn't just be possible to convert a prefab before runtime and just reference the resulting entity the way you would with a GameObject that converts at runtime. I'd actually imagine that would be easier to actually implement as the entity has an index and its components if you convert the entity before runtime (which is why we can inspect the entities with DOTS editor without having to press play), so it wouldn't have to run GetPrimaryEntity on the converted GameObject to get the Entity reference, it would just be able to directly reference it. Am I wrong about this? If not, why can't I simply reference an entity in a SubScene the same way I would reference an entity that converts at runtime?

Most importantly, is there any other way to convert an entity before runtime that does allow me to reference it in a MonoBehaviour or are we stuck using the Convert To Entity component for now(which Unity is already saying will be deprecated at some point, and is slower than pre-runtime conversion)?

 

Yes you are.

Entities in subscenes are remapped at runtime when merged into the default world from the streaming world. Nothing outside of the subscene would have any knowledge about the result of the remapping, since that can be runtime-dependent. If you want a MonoBehaviour to reference an entity in a subscene, then the MonoBehaviour needs to be a hybrid component within the same subscene.

 

Fair enough, but why couldn't those references be fixed after the remapping? You can reference an Entity directly for a ConvertToEntity component, even though that has to go through an entire process of conversion first yet that is completely possible. All the remapping would do is change the entity index, right? Why not just change all references to that index to the same new index, or at the very least have some kind of GetPrimaryEntity that can get the post-conversion index using the old index? That's essentially what I'm doing right now with my own hacky solution.
Or better yet, why doesn't the remapping happen before runtime as well, as all of the needed subscenes are already in my scene? I'm not manually loading in new subscenes during runtime and if I did, I obviously wouldn't be able to reference one of those entities in a component anyways. Or if I were to place everything in the same single subscene that remapping wouldn't really be required in the first place.

I wasn't aware hybrid was even possible in subscenes. How do I do this? Convert and inject isn't possible and MonoBehaviour components get destroyed unless I keep the subscene open for editing. Also, would said MonoBehaviour be allowed to have child objects or does it not have a GameObject at all? Because the whole point of my hybrid solution is having an animated mesh as a GameObject representation and just have it follow the Entitys position., which would obviously need a bunch of child objects for the rig.

 

How would Unity know which references to fix?

That's because the entity already exists at that point, and you are just modifying it by adding components during the rest of the conversion process. For runtime conversion, there is no remapping.

Subscenes can be instanced.

conversionSystem.AddHybridComponent()

See this thread: https://forum.unity.com/threads/using-classic-animator-with-ecs-subscenes.1086356/
The solution isn't great. But if you can get into the habit of closing subscenes before entering playmode, you might get some mileage out of it.

I personally don't use subscenes, but that's because I only convert a small amount of prefabs at runtime and then instantiate thousands of them procedurally. I do that because A) I'm bad at networking and finding designers that can populate game worlds, and B) the baked data can easily desync from assemblies and it is a pain to debug.

 

How would it not? If it somehow is not aware of its components that have an Entity field (which it obviously does, as you can have an Entity field in a data component with [GenerateAuthoringComponent]), you could just have some sort of EntityReference GameObject/authoring component that fixes the index/entity reference after conversion. That's exactly what I'm doing right now and it actually works as intended for my use case. I basically have to assign my own unique ID to each entity I'm trying to reference, and then when the game starts a system checks for every entity that has one of these IDs and sends it to every game object in my scene that has an "entity reference receiver". It's probably pretty slow the way I've implemented it right now, but it's the hacky solution I've had to come up with.

See, that's the part that doesn't make sense to me. The entity already exists for a pre-runtime converted object as well, it's just that the index gets remapped because of the whole subscene instantiation. So what if we were able to just able to directly convert objects in the main scene without this subscene nonsense? Sure, instantiate and remap my subscenes if I want to manually load and close those in code or whatever I would need them for, but just let me pre-convert and properly reference entities that are directly in my scene.

Anyways thank you for the help and linking that thread, it seems to not work in every situation but it looks something like this is all we have for now. I'll link him to the solution I came up with, maybe he'll have more luck implementing IK with what I've come up with.

You seem to be pretty active and have a lot of experience with DOTS, do you believe we can make proper games with DOTS in its current state? I've been trying to implement it for months now and all it's done is give me headaches rather than give me this amazing experience that I was promised in 2019.

 

Yes, you can, and we're pretty happy with this.

 

How complex are the games you make? Do you have animation rigs on your characters, sounds, or physics? It seems to me like the whole "hybrid" solution is still very necessary at the moment. Yet, at the same time Unity seems to want to step away from the "hybrid" thing and try to focus on pure DOTS though it seems to not be possible to make a game with pure DOTS yet unless you want every object in your scene to be either animated through a shader or not at all.

 

You can read through this pinned on DOTS forum thread:
https://forum.unity.com/threads/unity-dots-case-study-in-production.561229/
Our page on steam in my signature (video and screenshots a bit outdated, we will release EA in this year and will update them).
We have our own physics, animation, navigation solutions which we built purely on DOTS. And that's the awesome part, that we don't use general solutions from unity and building exactly what we need, optimised exactly for our purposes.

 

Because it doesn't know which subscene instance it was referencing. And that would all require a search through every UnityEngine.Object in the game through every field using reflection. That's crazy slow.

I'm using it as a foundation to build some custom tech that will eventually result in features that no game engine today has. You can find some small POC games I built mostly using DOTS (all but 1) here: https://dreaming381.itch.io/

 

Thanks for the link, that's very helpful.
If I'm not mistaken, it seems like you have a team that would have been willing to build these solutions in other game engines or even have written your own if you didn't use DOTS, correct? Or would you have done the same with MonoBehaviours/relied on classic unity systems?
I'm currently researching how viable DOTS is as a substitution for 'classic' unity, including for smaller indie teams. So not having to write all of your own systems like this would definitely weigh into that conclusion.

 

You're absolutely right, that is exactly how I'm fixing the references right now (Resources.FindObjectWithTypeAll). Though I still don't see why we couldn't just directly convert game objects in our scene and not need to use subscenes in the first place, seems like that would fix this issue.

Thanks, that looks great! I appreciate you including the source code as well, that will definitely help my research

 

You ended up with pretty hefty generalizations.

It is okay to say that you or your team doesn't want to work on physics/audio/whatever, so ECS isn't for you. Unity says exactly the same, if you expect a full-blown ecosystem behind you, ECS is not for you just yet.

But this doesn't mean it's bad or other people can't use it to make their games. You always need to keep in mind, you aren't the only customer of Unity, they are catering for many-many different people, teams, and capability.

I guess ECS isn't for you at the moment.

 

We can do this in MB land, BUT it will require us to write our own DOTS from scratch, and because Unity already did that - why we should? They give us a tool that gives us everything we need to focus on building specific game systems we required, without writing our own engine inside an engine or new engine. We reuse their rendering (partially), we reuse their editor workflow, we reuse their UI and new Input System (with our wrappers for DOTS), we reuse their Addressables and just focused on the things which really matter for gameplay.

 

The thing is, I'm currently writing my graduation thesis on the viability of DOTS for small Unity development teams, so sometimes I kind of have to think in generalizations. I also can't just say "ECS isn't for me at the moment", I have to explain when DOTS should or should not be used. So far, it seems like what Mike Geig said years ago still applies: that DOTS is a tool you can use to solve a specific problem, overcome some kind of specific technical hurdle. But in my opinion, its far from the "performance by default" Unity was promising at Unite 2019.

Don't get me wrong, I'm definitely not saying its bad, DOTS is obviously extremely powerful and I do think it has the potential of becoming the new standard some day. But my focus right now is on whether or not DOTS, in its current state, is viable for the average small Unity team. Correct me if I'm wrong, but I don't believe the average Unity team would have the time or knowledge to be able to write all these systems themselves. So at the moment it looks to me that the answer will be "no, it is not viable", but I'd have to prove that it is not, and more importantly have to do my best to try to prove the opposite conclusion as well. So I'm just trying to make sure that there isn't some sort of more user friendly way to do these kinds of things, rather than writing a whole bunch of boilerplate code and workarounds, before I come to that conclusion.

 

You're writing your thesis on the viability of a non-released feature? Are you sure about this? Because it has no value.
Also, you're writing a thesis and you drop a line that "I don't believe [...]". You shouldn't write a thesis on your belief. Where is your statistics on the "average Unity team"? What Unity team? Professionals? Hobbyists? What is the goal of your thesis? (Other than get you through the school...)

 

"Performance by default" != "Features by default". Just because there are few use cases that are supported doesn't make the phrase any less true for the use cases that are supported.

That's not provable in the general case. It is use-case and team-specific. Some teams are filled with people who pick up the concepts of DOTS easily. Others, not so much. Some projects require custom solutions regardless of whether they are built in GameObjects or Entities. Others could leverage existing GameObjects functionality but require a custom Entities solution.

Root scene conversion is a promised feature. Whether that will support unconverted GameObjects is unspecified. That would solve the problem more elegantly though.

 

How does it have no value? If a company I'm working for can already start to use DOTS in their production workflow (as a lot of people here claim is already possible), wouldn't that be a valuable technology to be aware of, and to have experience with when the first verified release comes out? If there are already companies out there using this, wouldn't it be necessary to at the very least look into it?
As far as what kind of team, the research is specifically about the company I'm currently working for which is between 2-4 Unity programmers but could expand in the future. Potential expansion is why I was calling it "the average Unity team" but the important thing in my case would be the context of the Unity team of this company.

 

Maybe "prove" was a bit of of an overstatement, the point is I have to explore in what contexts DOTS would or wouldn't be a good solution. How do you know whether your team would or wouldn't pick up DOTS easily? What project would need some hybrid solution, or when should you stay away from DOTS entirely? Those are the kinds of questions I will have to answer to come to some sort of conclusion on whether or not DOTS should be used (and if they DO use it, how the best way to learn to do so would be). It's obviously not gonna be a conclusion like "DOTS is good" or "DOTS is bad" because like you said, it all depends on the context.

I'm glad to hear that. Do you have a link to an up-to-date roadmap for DOTS?

 

I don't know. Where I'm coming from, thesis is not like a page long memo we usually write when we evaluate the viability of a tech at my company. But maybe I'm wrong here, it's just strange to me.

When the first verified version comes out your thesis will be severely outdated since you're writing about the current situation.

Your company is your company, without hard data about Unity's clients you have no way of knowing how an average Unity team looks like. And further more, you will need to decide what do you mean by Unity team at the first place, I mentioned above, hobbyists are a big part of Unity, do they count? Or only the studios? Only the indies or AAA too? How do you determinate the average size and more importantly the inclination of the developers towards writing their own systems? No data becomes true just because you think it is true.
Well, if the viability of DOTS or ECS at your company is worth a thesis, then it's okay, just don't generalize it, because it won't be true automatically.

 

Nope. Got that fact from the docs about Convert To Entity.

 

Most other graduates write their thesis on something like "what functionality does this software need", I actually wanted to do something that interests me and is actually researching some new technology rather than a single specific usecase. Maybe DOTS doesn't need a whole thesis written on it to you, but for some people a 30-page explanation on what kinds of problems I ran into and what information I learned in the 5 months spent researching and prototyping with DOTS might actually have some value, even if the full release of DOTS might be slightly different. This company and my school both agreed it had value and was an interesting subject, so that's all that really matters to me.

I have no idea what you're even trying to say here. Should I just stop trying to write my entire thesis then because everything I think is true might not be?

Obviously I'm defining the context and the value of this research very clearly, which is why I had to write a 20-page research plan in my first month that had to be approved by both the school and the company before I was even allowed to start. Also, just so you know, if someone is in a stressful situation while in the middle of their graduation project, "your thesis subject sucks" is awful advice. Maybe in your mind you're trying to help but all you have tried to do so far is discourage me.

 

Well, good for you then...

Dude, I am a random guy on the internet. I'm neither your father nor I'm your advisor, it is not my job to encourage you.

I'm saying you're waving around phrases like the mentioned 'average unity team' and 'they don't want to implement stuff'. I'm saying, that it's not true just because you're saying it. I'm honestly interested where do you get your data because I don't have it. So if you know hard data about Unity's client structure and hard data on how many professionals, studios, indies versus hobbyists they have in general, and any kind of measure of their proficiency or inclination to make their own systems, and it is shareable, please do!

 

It wasn't your job to discourage me either, yet you took the time to do that. Even if I wasn't writing a thesis on this subject, imagine if someone came in here trying to learn DOTS only to get shut down by someone saying "DOTS isn't for you" rather then actually answering their question or pointing them in the right direction. I also already explained why I used those terms but you obviously didn't read that so you can just stop replying to me and go watch this video instead

 

Over-generalizations are misleading and often harmful.

I have no issues with someone asking questions and researching the viability of DOTS, but I do have an issue if they spread their conclusions in a far more generalized manner than what their use case is.

 

Congratulations, elegantly stepped over the 'show your data or explain why' part.
I sincerely hope your thesis will be okay for your school though...

BTW, just to be clear, you waved around a bunch of unsubstantiated data, then I asked you where those are coming from. At this point you started to wave around the fact that you're writing your thesis about the subject as some sort of self-describing authority. As if the fact that you have an assignment in school would give your assumptions any merit. Instead of either give something tangible, or at least explanation why do you think what you stated or rethink your position on those statements.

I am really open to the possibility that you're right. I doubt it, but I'm open, so if you have a shred of any kind of evidence, please, show us.

 

@TWolfram This is just an idea so take or leave it.

In my opinion the two best replies you got were about people supplying data points by specifying how they use dots by linking to source or specifying which packages they use. Maybe the best idea is to request similar objective data (assuming people are willing!) and use that to help figure out whatever you're trying to do. Some sort of survey asking which DOTs packages and other things people are using, if they have a game that is live, pre-launch dev or just experimental indie dev, for example. Maybe we would find out, for example: 90% of DOT developers using ECS but only 5% using DOTs Animation, 35% using hybrid renderer, only 20% are developing for mobile etc, could be interesting (I've entirely made up those figures obviously).

Something interesting might come out of it and people can form their own conclusions from the data.

 

The conclusion will be aimed at the use case of the company I work at. Again, maybe I shouldn't have used the words "the average Unity team", I did this because the team is still growing, but all my report will really care about is the use cases this specific company could run into. All that I'm doing is documenting my experience and give advice to this company based on that experience and the information I've gathered while trying to make prototypes in DOTS. I'm not making any blanket statements on whether or not anyone should be using this, it is only advice based on my experience with making prototypes and the information I've gathered in that time.

What statement exactly are you referring to? I've never made any statement claiming to know anything with certainty, that's why I always use words like "it seems" or "I think". I'm always open to the possibility that I am wrong about something and rereading what I've said the only thing I think you could be talking about is what I consider to be "an average Unity team", which I've already admitted was the wrong term to have used. So what are you talking about? What "unsubstantiated data" did I wave around exactly? You're the one telling me DOTS is not for me, not knowing anything about my use case. All I've said is that I'm trying to research the viability of DOTS as a replacement for classic Unity, which yes I already said I should have clarified was more about this specific company.

Also, I have never claimed to be any sort of authority. In fact, me writing my thesis on this only indicates that I am essentially a noob at being a professional programmer, and have only tried to verify my assumptions by people who know more than me. That's the whole reason I'm asking questions on this forum in the first place. But your immediate hostility makes it difficult for me to actually ask questions because sometimes I feel more like actually writing code than arguing with hostile forum users.

 

A survey of DOTS users was definitely on my to-do list. A question on what packages they use exactly sounds like a great idea, thanks! I'd be especially interested to see how many people actually use Unity Physics, that's been the most disappointing/unstable part so far in my experience.

 

Thank you for clarifying this. I'm a contextual person, so with this new understanding of your goals I will be able to answer your questions more effectively moving forward.