Feedback DOTS and Job System SUCKS - provide proper access to multithreading (!)

EDIT:
This is now "solved", in terms of that i get a separate thread.
I will edit this post on the weekend to proper explain the issue
but without ranting. Besides the struggles with ECS and Jobs,
i really love unity and believe the developers did a great job
with the engine in general.

So after 6 months of daily trying this tech, i have not managed to simply move enemy units
towards the player, on a separate thread. Neither with ECS nor with Jobs/Bursty.

No matter which i used, JOBS, ECS... i hate it, it sucks! Extremly clumsy to use, nothings working, change
something here, break something there... these tech's have worst "usability" i ever experienced.

Unless Unity does not provide proper accessibility, then this tech will stay niche forever
and only be used by a small handfull of devs.

Its already a crime it took gameEngine devs so long to "figure out" that people "maybe need
multithreading", cant believe this isn't pure standard in 2023.

And then the offered solutions suck and scare the regular devs away.

Absolutely disgusting to use tech.

 

The job system has been in use and out of experimental for a while. It's just ECS that is new.
I haven't used ECS, but with the Job System you can use https://docs.unity3d.com/ScriptReference/Jobs.IJobParallelForTransform.html

 

I don't really see things the way you do. And I doubt the developers at Unity do either. Could you provide some examples of the issues you are having? And can you elaborate on what "proper" would look like?

 

Thank you, i tested 3 variants for IJobParallelForTransform alone (one was the direct example from the unity docs, just using positional movement, instead of velocity applied to y axis, like in the example). All 3 approaches failed with either the units not moving, or errors. I would post them here but not sure if people are interested in debugging walls of text.

EDIT: i believe i had an example working once in the first month, but then found out this runs on mainthread. But not sure if i still have that at hand, as i was aiming for separate thread, i most likely dumped it.

 

If you can Complete() some code will still run on the main tread btw, since it has to wait untill completion.
Other than that if you have multiple transforms in the array it should be multithreaded at least

 

What you are trying to do is very simple, but DOTS / multithreading is a bit of an advanced topic. If you have invested 6 month on a daily basis, I can see how this is frustrating and I honestly think you would best be off with a tutor. The gap is likely too big to be bridged by forum posts. There is for sure room to make the current DOTS universe more customer friendly, but I honestly think you are fighting with yourself here…

 

"Unity AI tutors, coming to you for just hundreds of dollars per month".

Ah, now I see why Unity abandoned their whole engine in favour of their new pet features.

 

Ah the AI stuff is haunting me…a real person tutor, just try to PM a nice bloke from the forum and see if this gets you somewhere… I assume OP has tried to read the manual and watched tutorials and it got him nowhere… speaking to someone who knows…it will be a chain of questions

 

Here is an example of what i mean with "inaccessible", thats DOTS though, not Jobs (but imo it applies for Jobs too):



Do you see how much gibberish one has to write, just to simply transfer a value from ECS to Mono world?
This is not much different for Jobs + Bursty as it seems.
...


Getting back to the jobs,
For jobs i would expect an API from unity that eases things up and allow me to call a method on separate thread; just like i would call a coroutine: StartThreaded(MyMethod())

Also, seeing this guy, he seems a very experienced dev, but he also has to use a "workaround" to achieve (TRUE) multiple threads, which consists of nesting multiple coroutines:

(i think he wouldnt do this if there was a better way)

My attempts (code following):
i am not sure if you guys want to see all this, but here are my 3 approaches on how to move units multithreaded:

Approach 1:
Code -> https://pastebin.com/DMRkQKaR (UNITY docs example, taken from here: https://docs.unity3d.com/ScriptReference/Jobs.IJobParallelForTransform.html )

Errors:

..whereas for enemyUnits IDE is talking of "has to be non-nullable type" - i believe i cannot access that custom class on low level? And for Schedule(), it is saying "there is no boxing conversion available", was not able to fix that, dead end for me.

Apporach 2:
Code -> https://pastebin.com/GW8JeR6F

Errors: none, just cannot get the code to accept player position from outside the job struct, in the example it is using [SerializeField],which of course is not accepted in Burst world (just left that uncommented in the code as reminder where i was stuck). In fact, i tried with several ways to feed the value into the job struct, but had no chance. Dead End.

Approach 3:
Code -> https://pastebin.com/h74f2Jht
Error with Schedule again, "EnemyManagerBURST cant be used as typeparam "T", there is no boxing convention available"

(examples end here)
.....

There were 2 completely different versions, that have most likely been dumped when i was angry. One seemed to work, but when looking at compiler it was running on mainthread. Dead End, dumped.

These are only tests regarding "how to move unit towards player on separate thread". I have done severall other experiments, also tested DOTS prior 1.0 as well as v1.0, but same as with jobs, just didnt work.

Here are 4 more links, leading to posts of me on Unity Discord server,
dating back to January 2023, fighting with DOTS:

JOBS and Reference Types: https://discord.com/channels/489222168727519232/1070060126809116692
DOTS Feedback: https://discord.com/channels/489222168727519232/1068902623975055361
Accessing GameObject in Monoworld from ECS world: https://discord.com/channels/489222168727519232/1068902623975055361

My current solution:
i moved multithreading to GPU, so currently the GPU takes care of unit movement and obstacle Avoiding via compute shaders. Works good, but comes with other challenges, which atm i still find easier to tackle than all the Job system related multithreading mechanics. But its a shame that people are forced to go this way if there is multithreading.

Thanks and best!

 

Thats another issue in my eyes - unity forces me to follow strict multithreading scheme, but why for F***s sake,
they just should let me allow firing a separate thread at my will and let me take care of waiting for completion, race conditions, etc.

 

You can schedule a crap ton of IJob's and disable all safety (;
C# threads also could work, but stuff like transform if off the table since that's managed code.
The job system is just a lot more safe and easier to get up and running. Together with burst it might even be faster than manual threading.

 

I thank you very much for the input,
and i am sorry i have no positive response other than that i have tried that too;

- tried disabling thread safety:
https://docs.unity3d.com/Packages/c....4/api/Unity.Collections.LowLevel.Unsafe.html

- i even tried multithreading with plain C# but unity blocks this
(when asking for help, i was told there is no way around, if you know one, please mind sharing more info on that).

Maybe people now start to understand why atleast from my view, the implementation of that tech sucks. And i believe i will not be the last one struggling with this, as unity made that tech not accessible.

Thanks and best!

 

I fixed your "Approach 3" -> https://pastebin.com/VAQxWzgB

The main issue was, that you weren't using a

TransformAccessArray

but a

NativeArray<TransformStruct>

instead, which would also work - but not as parameter when scheduling a

IJobParallelForTransform

.

Also I changed the way the array is filled - because you can't do that while the job is running. So I'm only updating the arrays used in the job when the job is completed.

Another issue I still see with this code is that delta time is the wrong time to move those enemies. Instead you should measure the time between job completions (with

Time.time

or

System.Diagnostics.Stopwatch

) and use that instead.

Edit: one more thing - you have to add

Code (CSharp):

1. if (_moveJobHandle.HasValue)
2. {
3.     _moveJobHandle.Value.Complete();
4. }

inside

OnDestroy()

before Disposing the Arrays.

 

I am surprised people even care of my long post, i thought it is too much to read,
so thank you very much for looking into this and for even correcting my mistakes,
i seem to be lucky as this is not taken for granted.

How did you know that the NativeArray is not accepted as Parameter? Where did
you learn this info from? The main concept is covered in docs, but where to get
the info about such small details? Monoworld seems intuitive, if something doesn't
work there, i can still find the error, but with JobSys i reached many dead ends
where even extensive googling wouldn't come up with fresh ideas for finding the
error. I really wonder how some of you guys managed to dig so deep into this.

I am currently looking into the corrections and will report back when tested,
thank you very much already for helping, i really appreciate this!

EDIT:

Mother of god... It's truly working now ✔



I swear by god man i would kiss your bare feet right now... Thank you so much!!!

Since half a year, finally i see a Bursty Job being executed on a separate Thread...
i almost had a heart attack when i saw this lol. This was the biggest stone i ever
faced in my whole game dev life, and now its gone, i still try to grasp this moment...

I will send you a tip for that, it is not for granted that someone fixes my code, you
shall deserve compensation for your good deed!

I still think Unity should make this tech easier accessible, but maybe my rant was
to hard judging, i mean, besides this one thing i really love unity. I will edit my op post
on the weekend and try to phrase it better. Maybe it would help if unity provides more
examples for that tech, more youtubers covering that, etc.

Thanks & Best.

 

Crossing the boundary between ECS and GameObject worlds is awful. I 100% agree with you there. If you can stay fully in the ECS realm, it is a lot better, though still not quite as easy as GameObjects.

As for jobs, multi-threading is actually very difficult to do both safely and fast. The only programming language the delivers both consistently that I know of is Rust. Every other solution either gives you a footgun, is as complex as Unity's jobs system (or worse), or is easy to use but uses very slow techniques that sometimes make it slower than just doing things on a single thread.

A big part of the job system is understanding the error messages. They can be a bit cryptic when you are starting out. But once you understand the vocabulary, they will point you in the right direction. But to build up that initial vocabulary, honestly the best resource is to ask here.

For example, you got caught up by a type not being "non-nullable". What that actually means is that your type is a class or an interface and gets passed around by reference. Really, it needs to be a struct or a plain-old-data type. And if it is a struct, the same rules apply to all fields within the struct recursively.

"There is no boxing conversion available" means that an argument to a method or an assignment is invalid, and it couldn't do any sort of cast to resolve it.

It sounds like you have something to go off of now. But if you have any more questions about things you don't quite understand, just ask! In general you will find people on Discord have a bit less patience than people on the forums, as forums are more conducive to longer-form discussion. We've all had those frustrations of learning the tech. But once it clicks, it is very good (in places, you still need to choose your battles wisely ).

 

Thanks a lot for the kind words, i am really surprised how helpfull the people here are!

Thanks to your explanations, i now understand the error messages. Here though i wonder,
where did you get the "translation" for the cryptic error messages back when you learned this?
There must be an initial source where all this info comes from, besides docs?

Thanks and best!

 

I heard you can pay to get help from unity directly, is it that?

Thats correct, i consumed any info i could get, i read the docs, i googled, i tested, i watched all DOTS and Jobs videos on channels like WaynGames, TurboMakesGames, CodeMonkey, asked chatgpt if it sees errors in my code, asked on unity discord, etc.. the list is indeed long.

 

Time and experience. I wish I had a better answer, but I have been doing this for too long now. At this point, I can only evaluate a learning resource by how correct it is, rather than by how valuable it is for someone starting out.

 

My IDE (Visual Studio) told me

IDK how the documentation is today but back when I played around with the JobSystem and Burst (Q1 2020) for my custom physics engine for physically correct Building Destruction

and later with ECS (Q1 2022) for Hordes of Enemies, fully procedurally animated because I just wanted to try this approach

there wasn't really any documentation. Just examples here and there, a bunch of forum posts, some answered by Unity-Devs, some YouTube Videos that were immediately outdated because the API changed rapidly .. it was all trial and error.

Meanwhile I've also used DOTS at my dayjob and also I tried helping a friend with his C++ Game a few years back which required me to learn some basics of memory management and that also helped with understanding allocations better. (in the end I couldn't help him because it was 20k lines of spaghetti code in one file and 10k more lines in another one - the whole game was in two files basically)

So yea, experience might also play a role here I guess - or I was just lucky, I can never tell the difference. xD

 

The DOTs system is quite the pain to work with and generally I do just avoid it because speed of development is far more important to me than speed of execution almost all of the time. But the reasons it all works the way it does makes more sense when you look at ot from a historical perspective. First and foremost, the core API in the engine itself was never designed to work with multithreading. It can be argued either way I'd this was a good idea or bad but at this point they have too much momentum behind it to change that now. Add into the mix the complexities of dealing with marshalling between the native side of the engine and the managed as well as the design limits of C# and how that has changed over the years and you start to realize the very technical issue that surely cropped up.

The design and public interface of these systems has changed more times than I can honestly remeber and it's pretty much always been for good reasons that involve technical limits, usability issues, or both. This isn't to say it's perfect. But I think it's at least understandable. It's a compromise that lets people who really need it to have access while still leaving the fundamentally approachable systems in place. Personally, I have yet to have need of it other than the very rare spot optimization and so I have no qualms with advising most to stay away from it other than to achieve very specific goals or to simply learn so they have it in their toolbelt for such occasions.

EDIT: It'd be nice if I could type on my phone for more than 10 seconds without the site auto posting whatever I've written so far. Is this a common issue on these forums with androids using chrome?

 

Lol, careful what you wish for. Have you ever prpgrammed something larger multithreaded with the basic constructs like Monitors, Mutexes or even Semaphores?
I have (for a Master thesis). It's hell. Am glad that Unity abstracts so much away.