Search Unity

Feedback Particle System C# Job System support

Discussion in 'Particle System Previews' started by richardkettlewell, May 1, 2018.

  1. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    karl_jones likes this.
  2. ifurkend

    ifurkend

    Joined:
    Sep 4, 2012
    Posts:
    349
    I missed the revelation of Unity C# Job System back in mid-2017, so I am kinda confused and I think many other effect artists here like me who began learning C# (and HLSL) primarily for expanding the variety of Shuriken-based effects would have a funny time to comprehend why this Job System is introduced in the first place. The manual says nothing. There is an hour-long official video about this topic with no timestamp for us to look for vital info. How to use it in the right way or the possibilities it opens are nowhere to be found.

    Amateurishly I can see the benefits of getting greater control over manually assigning jobs for multi-threaded CPU, but the best-practice part is often left uninstructed by Unity, resulting the effect artists taking ages to figure out the programming and worst being blamed by programmers for coding not in the fashion/mindset of a true programmer when there is no dedicated technical artist around. So I just want Unity to give us in-depth instructions (which is currently none) of how it works and what pitfall to avoid when dealing with such programmatic feature by effect artist.
     
  3. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
  4. ifurkend

    ifurkend

    Joined:
    Sep 4, 2012
    Posts:
    349
  5. dmarfurt

    dmarfurt

    Joined:
    Apr 28, 2013
    Posts:
    9
    Can a brother get a link up in here?
     
  6. ifurkend

    ifurkend

    Joined:
    Sep 4, 2012
    Posts:
    349
  7. Endar

    Endar

    Joined:
    Aug 13, 2014
    Posts:
    3
    I will have to try this out. I did make a job implementation using 2018.1 that changed the color of particles based on their direction. With this experimental job for particle systems, I should be able to make a more efficient version of the job.
     
    richardkettlewell likes this.
  8. Endar

    Endar

    Joined:
    Aug 13, 2014
    Posts:
    3
    Here is what I have for it so far. It seems to work.

    Code (CSharp):
    1. using System.Collections.Generic;
    2. using Unity.Collections;
    3. using UnityEngine;
    4. using UnityEngine.Experimental.ParticleSystemJobs;
    5. public class ParticleColorByDirection : MonoBehaviour
    6. {
    7.     public ParticleSystem ps;
    8.     public List<Vector3> directions = new List<Vector3>();
    9.     public Color [] colors;
    10.     NativeArray<Color> nativeColor;
    11.     NativeArray<Vector3> dir;
    12.     public bool moveTowardsNewColor;
    13.     public bool useDeltaTimeLerp;
    14.     public float lerpValue= 0.5f;
    15.     void Update ()
    16.     {
    17.         var job = new ParticleDirectionColor();
    18.         dir = new NativeArray<Vector3>(directions.ToArray(), Allocator.Temp);
    19.  
    20.         nativeColor = new NativeArray<Color>(colors, Allocator.Temp);
    21.         job.colors = nativeColor;
    22.         job.colorDirections = dir;
    23.         if (useDeltaTimeLerp)
    24.             job.delta = Time.deltaTime;
    25.         else
    26.             job.delta = lerpValue;
    27.         job.moveTowards = moveTowardsNewColor;
    28.         ps.SetJob(job);
    29.        
    30.     }
    31.     private void LateUpdate()
    32.     {
    33.         nativeColor.Dispose();
    34.         dir.Dispose();
    35.     }
    36.     struct ParticleDirectionColor : IParticleSystemJob
    37.      {
    38.         [ReadOnly]
    39.         public NativeArray<Color> colors;
    40.         [ReadOnly]
    41.         public NativeArray<Vector3> colorDirections;
    42.         [ReadOnly]
    43.         public float delta;
    44.         [ReadOnly]
    45.         public bool moveTowards;
    46.         public void ProcessParticleSystem(JobData particles)
    47.         {
    48.             var startColors = particles.startColors;
    49.            
    50.             int closestDirectionIndex = -1;
    51.             float closestDist = float.MaxValue;
    52.             Vector3 particleDirection;
    53.             var startCo = particles.startColors;
    54.             var velocities = particles.velocities;
    55.             for (int i = 0; i < particles.count; i++)
    56.             {
    57.                
    58.                 particleDirection = new Vector3(velocities.x[i], velocities.y[i], velocities.z[i]).normalized;
    59.                 for (int x = 0; x < colorDirections.Length; x++)
    60.                 {
    61.                     float dist = Vector3.Distance(particleDirection, colorDirections [x].normalized);
    62.                     float ang = Vector3.Angle(particleDirection, colorDirections [x].normalized);
    63.                     float weight = 1f - (ang / 180f);
    64.                     if (closestDist > dist)
    65.                     {
    66.                         closestDist = dist;
    67.                         closestDirectionIndex = x;
    68.                     }
    69.                 }
    70.                 if (moveTowards)
    71.                     startColors [i] = Color.Lerp(startColors [i], colors [closestDirectionIndex], delta);
    72.                 else
    73.                     startColors [i] = colors [closestDirectionIndex];
    74.             }
    75.         }
    76.      }
    77.    
    78. }
     
    richardkettlewell likes this.
  9. shaun

    shaun

    Joined:
    Mar 23, 2007
    Posts:
    728
    It appears to work fine, but from what I can see it simply offloads the Job into one of the worker threads.

    If you're already using the Job System processing an IJobParallelFor, this results in a performance decrease because your IParticleSystemJob gets run on a random worker after the IJobParallelFor.

    The timeline below shows a parallel job running a 2-octave Perlin noise on a position, and then feeding that to a particle system job. There's 90,000 particles in this instance.

    Screen Shot 2018-07-05 at 17.04.59.png
     
    Endar likes this.
  10. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    I guess it would be better if the particle job was a for-each too, and you could choose the granularity? Instead of one big job.
     
  11. shaun

    shaun

    Joined:
    Mar 23, 2007
    Posts:
    728
    Yep, that would be super helpful. Most of the time, if you're directly manipulating particles it's because there's a lot of them, so being able to split the for loop across multiple workers would give a huge boost.

    Out of curiosity, how would you dependency chain jobs together? In my case I want to parallel-for a bunch of work, then throw the result over to particle system job. The SetJob method doesn't seem to support JobHandle dependsOn, and theres no job.Complete().
     
  12. AnttiX

    AnttiX

    Joined:
    Oct 31, 2017
    Posts:
    1
    What is the status of this? I'd be very interested in manipulating particle positions in an IJobParallelFor.
     
  13. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    We didn’t get it finished in time for 2018.3, so we are looking at a 2019.1 release now.
     
    AnttiX likes this.
  14. defic

    defic

    Joined:
    Apr 29, 2013
    Posts:
    16
    I am really looking forward for the new release. Is this feature included in 2019.1?

    At the moment I am calculating positions for particles in a job and then setting the particle positions afterwards in a normal loop.

    It already gives a lot of performance running the position calculations with burst and in parallel, but when I can modify the positions directly it is going to be great! :)

    Further question: When ParticleSystem positions are updated internally?
     
    Last edited: Jan 10, 2019
  15. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Yes it is :) Hope it works well for you, all feedback welcome!
     
  16. defic

    defic

    Joined:
    Apr 29, 2013
    Posts:
    16
    I tried the 2019.1 alpha and it was a nice way to manipulate particle data. However, I was hoping to take advantage of parallel jobs with particles (this was mentioned as upcoming feature in Unity Berlin particle video by Karl Jones) and it does not seem to be possible at the moment. Any word when this could be coming?

    The way to use the ParticleSystemJob was also unconventional from normal jobs. F.e. it is not possible to set dependencies to other jobs, but I guess that's because of how the particles are updated internally.

    Also I am interested to hear any word on user made particle system modules :)

    In live Unity 2018.3 project I manipulate particle positions by having a local particle array and using SetParticles method. I am not using GetParticles as I want to be in control of the particle array order (it is unsure where Unity engine internally moves the newly emitted or dead particles). Texture sheet animation module works when decreasing deltatime from particles remainingLifeTime. However rotation over lifetime module does not work. Is there a way to get it to work in this scenario? One option might be querying the rotation per particle using the modules MinMaxCurve.Evaluate(time, particleRandomSeed), but it feels like waste of performance.
     
  17. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Both of these points relate to the same limitation in how the job is scheduled currently. Right now, we simply call the c# job from the end of the internal particle update job. That is 1 job, hence the c# job is also 1 job. And because it's internal, you don't get to see the JobHandle, so no dependencies/waiting for completion.

    What I'm hoping to do, is change the API to work something like this:

    - You declare a MonoBehaviour function, similar to how you declare OnParticleCollision today. That function is called something like OnParticleJobScheduled. It gives you the JobHandle of the internal job. You are then free to schedule whatever you like from that handle: eg a foreach job with a granularity of your choosing, or just 1 job like how it works today.
    - Allowing you to do the scheduling will also give you a JobHandle back, to create your own dependency chains, like you can do with all other c# jobs.

    Karl believes that we could provide a custom module UI in the Inspector as new module(s), with each entry backed by a C# job, to let you build custom modules. I believe that you can already build custom modules using C# jobs, and as a programmer needs to write the job/module, that same programmer might as well just add the UI in the MonoBehaviour script too. So I'm not really sure there is much for us to add to make this possible (except i want to be able to attach multiple c# jobs to a particle update, whereas right now it's just 1.

    When we make the improvements to the API, I may try writing some example "custom modules" to prove/disprove the concept.

    New particles are emitted at the end, and when a particle dies, the live particle at the end of the array is copied into the slot of the dead particle, to maintain a contiguous array. It's hard for you to track this in script, but it is possible via the Custom Data API: see the example about adding a unique ID to each particle: https://docs.unity3d.com/ScriptReference/ParticleSystem.SetCustomParticleData.html

    Maybe one day we will have a proper event system...

    I'm not sure, sorry :(
     
    defic likes this.
  18. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    1,948
    Is it possible to process collision events inside IParticleSystemJob?
    E.g. static / dynamic collisions when particle collides with something.

    I'd love to manipulate particles that collided with something. That would be great.
     
    Last edited: Feb 16, 2019
  19. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Hey, thanks for the feedback. No, it’s not possible right now, but it is a great idea. I’ll add it to our roadmap for this feature as we try to get it out of experimental status.
     
    Djayp, ReadyPlayGames and TerraUnity like this.
  20. sstrong

    sstrong

    Joined:
    Oct 16, 2013
    Posts:
    1,203
    Are their any details on what was released with 2019.1? The Release Notes only have a one liner.

    Also, are there any samples or doco?
     
  21. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Last edited: Apr 23, 2019
    TerraUnity likes this.
  22. Opeth001

    Opeth001

    Joined:
    Jan 28, 2017
    Posts:
    294
    Is it possible to use particle systems with ECS + C# job? Maybe like bullets or explosions effects.
    If yes please give me any link to a doc or just any example.
     
  23. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Hi, no it’s not possible as far as I’m aware.
     
    Opeth001 likes this.
  24. Opeth001

    Opeth001

    Joined:
    Jan 28, 2017
    Posts:
    294
    I think hybred entities will do the job then.
    Thank you.
     
    richardkettlewell likes this.
  25. ifurkend

    ifurkend

    Joined:
    Sep 4, 2012
    Posts:
    349
    Finally got sometime to study Unity Job System (which difficulty is incredibly dwarfed by ECS), but right now (2019.1f) it feels bit redundant to cache all 3 axes of a NativeArray3 to 3 different float arrays and Set them back individually after we're done manipulating them, I wonder if IParticleSystemJob could provide something like TransformAccess of IJobParallelForTransforml.

    I just stress-tested with simple velocity-based homing logic with 100k particles in a bare scene, but the performance actually drops compared to the classic way of get-set particles. As the ParticleSystem.UpdateScriptJob is scheduled sequentially.

     
    Last edited: Apr 28, 2019
  26. TigerFudo

    TigerFudo

    Joined:
    Nov 30, 2012
    Posts:
    18
    Here is my codes using IParticleSystemJob in an ECS project:
    I passed the pawns' positions to the Job .
    Job:
    struct FollowEffects : IParticleSystemJob
    {
    [ReadOnly][DeallocateOnJobCompletion]
    internal NativeArray<UPostion> Positions;
    [ReadOnly][DeallocateOnJobCompletion]
    internal NativeArray<URotation> Rotations;
    public void ProcessParticleSystem(ParticleSystemJobData particles)
    {
    ......
    }
    }

    Below is some codes in my ComponentSystem:

    void OnCreateSystem()
    {
    .....
    particleJob = new FollowEffects ();

    particlesystem.setJob(particleJob);
    ......
    }

    protected override void OnUpdate()
    {
    ....
    particleJob.Positions = LivingPawns.ToComponentDataArray<UPostion>(Allocator.TempJob);
    particleJob.Rotations = LivingPawns.ToComponentDataArray<URotation>(Allocator.TempJob);
    ....
    }
    When I run the project ,I got array data leak' error.
    How can I pass ComponentDataArray to the particle Job correctly ?
     
  27. jjxtra

    jjxtra

    Joined:
    Aug 30, 2013
    Posts:
    1,013
    Is it possible to do custom sort of particles using a job? The built in Unity distance sort falls short with alpha blended mesh particles. Is get/set particles still the only way?
     
  28. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    ̶Y̶e̶s̶ ̶g̶e̶t̶/̶s̶e̶t̶ ̶i̶s̶ ̶t̶h̶e̶ ̶o̶n̶l̶y̶ ̶w̶a̶y̶ ̶t̶o̶ ̶d̶o̶ ̶̶t̶h̶i̶s̶.̶ ̶̶T̶h̶i̶s̶ ̶n̶e̶w ̶f̶e̶a̶t̶u̶r̶e̶ ̶w̶o̶n̶’̶t̶ ̶h̶e̶l̶p̶ ̶y̶o̶u̶ ̶̶y̶e̶t̶,̶ ̶̶a̶l̶t̶h̶o̶u̶g̶h̶ ̶̶m̶̶̶̶a̶y̶b̶e̶ ̶̶i̶t̶ ̶c̶a̶n̶ ̶b̶e̶ ̶u̶s̶e̶d̶ ̶t̶o̶ ̶d̶̶i̶s̶p̶a̶t̶c̶h̶ ̶̶t̶h̶e̶ ̶̶n̶e̶c̶e̶s̶s̶a̶r̶y̶ ̶̶j̶o̶b̶ ̶̶̶c̶̶h̶a̶i̶n̶ ̶o̶n̶c̶e̶ ̶̶w̶e̶ ̶̶a̶l̶̶l̶̶o̶̶w̶̶̶ ̶̶̶̶a̶c̶c̶e̶s̶̶s̶ ̶̶̶t̶o̶ ̶̶t̶h̶e̶ ̶̶j̶o̶b̶h̶a̶n̶d̶l̶e̶.̶.̶̶̶̶̶

    EDIT: Actually, it was a bit early in the morning and I wasn't thinking about this correctly. My first answer was in the context of being able to dispatch a multi-job multi-threaded sort, which you can't do at the moment. However, you could totally perform a single-threaded sort in the job. you have access to all the elements and could swap them via any sorting algorithm you like :)
     
    Last edited: May 15, 2019
    Djayp likes this.
  29. Djayp

    Djayp

    Joined:
    Feb 16, 2015
    Posts:
    69
    Talking about it, any news of the job handle ? =)
     
  30. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Soon :) there will be a new experimental build available once it’s ready.
     
    Jes28, Noisecrime and Djayp like this.
  31. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    586
    Is it correct that IParticleSystemJob cannot be compilled with Burst or is that a bug?
     
  32. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    I will speak to the Burst team about why this is the case. Thanks for raising it.
     
  33. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    I've spoken with the Burst team and they have advised me how to fix this. The fix is simple and is working well :)
     
  34. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    586
    That's great! Thanks Richard. Can't wait to sprinkle some more [BurstCompile] here and there and see the huge speedups :)
     
  35. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Djayp and pvloon like this.
  36. pvloon

    pvloon

    Joined:
    Oct 5, 2011
    Posts:
    586
    Woah, wasn't even going to ask for those other features but exactly what I need :) Amazing!
     
    richardkettlewell likes this.
  37. Djayp

    Djayp

    Joined:
    Feb 16, 2015
    Posts:
    69
    Thank you ! I'll test it this whole weekend :)
     
    richardkettlewell likes this.
  38. Djayp

    Djayp

    Joined:
    Feb 16, 2015
    Posts:
    69
  39. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Oh no, indeed that was reported internally too on our latest 2019.3 alpha :(

    You can comment out the line in your package cache, or copy the package to a local folder, comment it out, and point the package manager at the local folder.

    Annoying, sorry.. I will make a new build as soon as this is fixed (unless the fix is made purely in package code).
     
    Rallix and Djayp like this.
  40. Djayp

    Djayp

    Joined:
    Feb 16, 2015
    Posts:
    69
    Thank you ! No problem, just to let you know ;)
     
    richardkettlewell likes this.
  41. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    I've just released a new version with the following improvements:
    * Support passing a JobHandle to the Particle System Schedule function, in case the particle job has prior dependencies
    * Fix job scheduling to ensure jobs are kicked off immediately, rather than not being processed until their results are needed (gives much better performance)

    Still waiting for the Burst fix.. it sounds like it's going to be a fix in the Burst package.
     
    Djayp likes this.
  42. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    There is another new version out, with a fix for errors when trying to pass custom containers (eg NativeArray) to a particle job. Plus we added support for the safety system, which provides you with errors when you (for example) try to write to invalid indices from a ParallelFor job.
     
    Jes28, pvloon and Djayp like this.
  43. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Hey all, this feature has been released in the forthcoming 2019.3.0a9 build! You are welcome to continue posting feedback in this forum, and also in the 2019.3 alpha/beta forum if you prefer.
     
    Djayp likes this.
  44. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,407
    Forgive my ignorance here. So is the basic idea to get the best performance use the api to set custom particle data, manipulate it via jobs, and disable all of the associated particle system component options?
     
  45. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Hey :)

    The idea is to be able to modify all particle data in a much faster way than with the old ParticleSystem.GetParticles and ParticleSystem.SetParticles APIs.

    You can modify any of the data (position, velocity etc), or, as you say, you can also setup and manipulate custom particle data.

    You might be able to get faster performance than the built-in components by re-writing them in jobs (eg Velocity over Lifetime, etc) but we haven't tested this, and it's probably not a good use of your time - I would expect the differences to be quite small. The real power of this feature is to write custom code for your particle systems that is not possible via the built-in components.
     
    TeotiGraphix likes this.
  46. TeotiGraphix

    TeotiGraphix

    Joined:
    Jan 11, 2011
    Posts:
    144
    This is interesting, I just happened on this and probably will try it with a visualizer I am making using particles.
     
    richardkettlewell likes this.
  47. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    603
    @richardkettlewell - Is there an ECS launch point for particle system Jobs or is it just the MonoBehaviour callback for the time being?
     
  48. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,189
    Hey, there are no plans at the moment for an ECS hookup with the particle jobs.
     
  49. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    603
    Hybrid it is then, just wanted confirmation. Thanks for the quick response!
     
    richardkettlewell likes this.
  50. TeotiGraphix

    TeotiGraphix

    Joined:
    Jan 11, 2011
    Posts:
    144
    This is great, I am just finishing up my engine code and I can actually give performance numbers from before and after as well, can't wait to try this in my framework.
     
    richardkettlewell likes this.