Feedback Wanted: Visual Effect Graph

I have been playing around with this and the Realsense D415 camers and have found it very cool! The realsense seems more stable in 2018.3 than it was in 2018.2, which is nice.

So far, I've got a simple version of what I want to happen working. I want particles to be emitted in the shape of the user that is X distance from the camera. My problem, and I'm wondering if anyone else has solved this, is how to generate a full greyscale image from the depth stream of the camera.

@JulienF_Unity screen grab was very helpful, and it sounds like @racarate and @jashan are both doing similar things.

Here's what I have so far:


The particles emit based on the raw depth stream. The problem is, the image is not a full depth image, just black and dark red. I can see that the librealsense examples use shaders to parse out the depth into color maps, but I can't figure out how to get those color maps as an input to the particle system. Works fine with the raw image and this function:

Code (CSharp):

1.    public void SendToFXThing(Texture tex)
2.     {
3.         thisVFX.SetTexture("CamTexture", (Texture2D)tex);
4.     }

I simply add a new binding in the depth sample. I do not know OpenGL well enough to figure out how to parse the data myself, and I'm also concerned that since I'm on a Mac I would rather be using Metal with this new VFX system.

Anyone here have some experience in converting a the Z16 depth stream to a greyscale depth image?

 

Just came here to say that VFX Graph now finally works in 2019.1a11. That is very cool!

I got a little stuck with things not related to VFX Graph (my main project is still on 2017.4, and now I need to make these two things work together, plus upgrade to the new SteamVR Input system which apparently is even more of a nightmare than upgrading Unity ;-) ).

Most likely I'll have to use VFX Graph with "vanilla 3D" (aka legacy render-pipeline) ... aside of upgrading a legacy project to HDRP, the other issue is that HDRP gives me around 2.5ms per frame overhead that I'd rather consume with more particles.

 

VFX graph wont work with legacy pipeline.

 

I'm not able to get VFX Graph working in 2019.1.0a11.
This is the only error I see:

Code (CSharp):

1. Shader error in 'Hidden/VFX/System 1/Quad Output': 'EvaluateAtmosphericScattering': no matching 2 parameter function at /git/jens/vfxgraphtest/vfxTest2019.1.0a11/Library/PackageCache/com.unity.visualeffectgraph@5.2.1-preview/Shaders/RenderPipeline/HDRP/VFXCommon.cginc(80) (on d3d11)

My setup is a simple as I could make it: New project with HDRP. Add VFX Graph from Package Manager (5.2.1).
Switch Scripting Runtime Version to .NET 4.x Equivalent in Player settings to stop most of the console errors.

Create a new vfx graph object. The error shows in the console and on the shader object when you expand the vfx object.

Help?

 

I've been experimenting with something similar. I've been using OpenCV (the Enox version from the Asset Store) to do threshold operations on the depth texture. Unfortunately it's all happening on the CPU so it's kinda slow, but there are tons of useful tools to fix up the depth texture. I've been using Photo.inpaint to fill holes in the depth texture, then Imgproc.threshold to convert it to black and white.

I've been looking at faster ways to do this and I just found Graphics.Blit this lets you supply a texture and a material and render the result into a destination RenderTexture. I haven't had a chance to try it, but it could be a fast way to do some of these operations.

Another thing to be aware of is the Z16 depth texture you get from the camera contains integers that are represent how many millimeters that pixel is from the camera. The range is technically 0-65535, but realistically 200-10000. Unity is expecting depth textures to contain floating point values between 0-1. I haven't tried it yet, but I think that's going to need to be part of the conversion if you want to use the depth texture block that's available in VFXGraph.

p.s. For stability, I've been setting the RsDevice object in the scene to use "Unity Thread" while in the editor. Multithread should work fine for standalone builds though.

 

Thanks! I think Graphics.Blit might be what I can use. I actually did this in another project using an extra camera, but it seemed so hacky I'd like to find a simpler way. The performance was actually pretty good.

The technique I used is in my project on Github. The webcam texture is shown on a polygon in the scene, a camera is set to look at the polygon and render to a rendertexture. The rendertexture is set to read/write and then fed into another script to convert it to a tex. The nice thing about this particular technique is downsampling is 'free' by setting the resolution of the rendertexture. For blob tracking I typically don't need all those pixels and for that project it worked well. So, the camera looking at the webcam feed can render at 90X40 pixels and I don't need to care what resolution the webcam's native feed is.

I'm going to try to use Graphics.Blit directly from the materials in the realsense project. They are doing threshold, color conversion in a shader and have feathering and other things exposed as parameters that effect the shader. I'm just not sure if I can grab the texture after the shader does it's work using Graphics.Blit. Hence, the hacky solution with a camera might be how I do this again...

 

I made a first attempt using Graphics.Blit yesterday and gave up and used the camera pointing at a quad technique instead. *shrug* Good luck!

You can use a RenderTexture directly as the Texture in the VFXGraph's Sample Texture 2D node, BTW.

 

Thanks! Did not know that.

It took me a while to figure this out, but if anyone else is interested, here's a screen grab of things working as I wanted it to. I made a second orthographic camera, changed the UI to World Space and positioned the camera to look at the UI texture. It does render the texture after shading, so I can now filter particles based on color or depth using the shaders provided in the realsense Unity project. Turned out I didn't need to write any code, just drag the Render Texture into the VFX node.

 

Just found an even nicer way to use the depth stream. I'm not even sure how this works exactly, but the camera that is rendering to a RenderTexture is pointing at the RawImage of a UI object connected to the rsDevice Depth stream. As you can see, the UI is a flat image in the editor.

But...once wired up to the Project On Depth block, the depth values are coming through to generate particles in 3d space! It's pretty nice! You can tweak the Project On Depth to get exactly the foreground object you want. It's a little hard to see in the screenshot, but my hand is closer to the camera than my body.



This is exactly what I was hoping to do! By tweaking the Depth X,Y on the block, I can isolate only my hand as an emitter of particles.

 

How would I go about saving my custom settings in a Visual Effect asset? I've exposed a number of inputs and when I change them during Play they are not saved. I was going to make a script that points to the Visual Effect Asset but I can't see how to access it.

Also, since the asset is in the Project area, i would think it would be like a PostProcessing effect and save the settings during play mode.

 

I was able to get VFX Graph working in 2019.1.0a11 by updating HDRP to version 5.2.2!

 

Cool - I found this (after searching for a little while). So that's Project Settings / VFX / Render Pipe Settings Path.

Unfortunately, both in Unity 2018.3.0b12 and Unity 2019.1.0a11, I get:

Code (CSharp):

1. Shader error in 'Hidden/VFX/System 1/Quad Output': invalid subscript 'xd' at /GameDev/Experiments/LegacyVFX/Library/PackageCache/com.unity.visualeffectgraph@4.6.0-preview/Shaders/RenderPipeline/Legacy/VFXCommon.cginc(49) (on d3d11)
2.  
3. Compiling Vertex program with UNITY_PASS_FORWARDBASE
4. Platform defines: UNITY_ENABLE_REFLECTION_BUFFERS UNITY_USE_DITHER_MASK_FOR_ALPHABLENDED_SHADOWS UNITY_PBS_USE_BRDF1 UNITY_SPECCUBE_BOX_PROJECTION UNITY_SPECCUBE_BLENDING UNITY_ENABLE_DETAIL_NORMALMAP SHADER_API_DESKTOP UNITY_COLORSPACE_GAMMA UNITY_LIGHT_PROBE_PROXY_VOLUME UNITY_LIGHTMAP_FULL_HDR

Is there a way to fix this? Turns out that for the time being, we can't move our project to HDRP, so Legacy is what we need to have working.

Full code that's causing the shader compilation error:

Code (CSharp):

1. float VFXSampleDepth(float4 posSS)
2. {
3.     return _CameraDepthTexture.Load(int3(posSS.xd, 0)).r;
4. }
5.  

 

I tried using Graphics.Blit again and it seems my problem might have been that the material supplied as the third parameter can't be using a ShaderGraph shader (HDRP?). My shader was simple so I rewrote it as an unlit shader and the blit works. This seems like a bug, if there are Unity people checking this thread I'm on 2018.3f1.

 

I'm looking for how to get the direction (like a forward vector) of a particle. Anyone know how? I've tried using the "direction" node and the "axis" node, but neither seem to work. Any help would be appreciated.

 

Was anyone able to get it working with DirectX 12?

Thanks!

 

How do we "lerp" between point caches?
Like what this gentleman did.

 

I made it 2 week ago. Video in Russian, but there are subtitles.

 

Is there a way to get the sample scene for this demo:
https://i.imgur.com/I6biWna.jpg?1

 

... so, while the Legacy render pipeline template is there, it's currently broken. I have just filed this to FogBugz:

(Case 1110700) Visual Effects Graph 5.2.3 has Legacy RenderPipeline Setting - but creates Shaders with Compilation Errors

Is there maybe a workaround so we can use Visual Effects Graph with the Legacy render pipeline?

 

Are there any way to rotate spawn position(for box, cone, sphere (chunk of it) etc.) inside the graph

 

How do you guys make anything in vfx graph when it bugs out all the time?

Ive had sliders dissapearing, blocks just getting stacked on top of one another, ui partially or fully corrupting.
Mind you i just started using in 5 minutes ago and already encountered 10 bugs.

 

Those bugs might be new - last time I checked, it was fairly stable (it did sometimes stop showing the connections, which was easy enough to fix by restarting Unity). I did notice some Editor UI issues in the package manager in Unity 2019.1.0a12 (the most recent alpha). If what you see is related to those, resizing the panel is a quick workaround (in the package manager, it restored those issues reliably).

 

Ditto, I have the same problems @pointcache does.

Along with this one:


Almost looks like that artifacting you get when your GPU is about to go.

EDIT: This only happens if a point cache is connected to the input of the "From Map" node. Otherwise it's fine.

 

Any chance we might see Translucent Particle Shadows in the next update, i love the fact that particles can be lit with light sources and shadows, but i have been trying to get soft shadows for smoke particles for a very long time. something like present in unreal.

 

Is there a way to control the seed for the random node? I'm using it for some random positions of particles, and would like them to have the same position every time the effect is played. Being able to set the master seed per graph I think would solve my problem.

 

This would be really amazing, even if only for unlit Outputs or points.. VEG is an unbelievable match for VR, and appears robust enough to be considered for production...

Would it be unthinkable, for example, to expose the graph results as compute buffers via a special output node?

 

I cannot seem to use set lifetime from curve. Using 2018.3, it throws a shader error. Has anyone else had this happen?

 

This is somewhat solved by using a random number for set lifetime which I think in some ways makes more sense as the graph I am not sure what the x and y axis would actually be?

 

Can I get a quick example of how the line output is actually meant to be used? Like for example, in the demos with long sweeping curved lines?

 

Hi I downloaded the last version of VFX and I have a problem, no particules appear in the unity viewport. unity 2018.3.0f2

 

Franckitou

Try creating a new project with the HDRP Template.

 

Hi,

is there a starting point to generete hair with collision against an sdf with visual effect graph.

Probably close to this interpolating technique out of hair strands.
https://forum.unity.com/threads/hair-tool.420492/

 

Any news regarding issue with legacy/standard pipeline ? I hope it will be fixed.

 

Hi folks,

Fluvio (fluid simulation) developer here. We're currently working on applying realtime fluid sim to the VFX graph. Is there any way for us to store neighboring particle information in a buffer, then access that data in a later step?

Use case: We're currently looking at getting particle neighbors. We currently do the following steps in our shader (non-VFX-graph) implementation. Each step happens as sequential parallelized steps, on a per-particle basis.

1) perform spatial partitioning of particles, stored in an infinite grid (GPU-optimized layout of a flat array, won't go into details here)
2) perform a neighbor search using the infinite grid, store X closest neighbors for particle Y in an flat array sized X * Y
3) perform physical simulation on each particle Y, reading information from each neighbor and writing a force vector for particle Y.
4) integrate forces into velocities

So in our case, as a bare minimum we need three buffers:

1) spatial partitioning grid: arbitrary (user-configurable) size * max neighbor count
2) neighbor buffer: particle count * max neighbor count (this one is generally pretty large but very doable w/ current GPUs)
3) force buffer: same size as our particle count for accumulating forces

Note - we're fine digging into the internals of blocks if this isn't possible with multiple blocks at this time. Right now it's the only blocker we see for adding full fluid sim to the VFX graph. So even if not possible with the current API or blocks, I'm happy to hear of any hacky ways we could experiment with this or explore.

EDIT: Thought through this and rephrased the question to better fit what we're doing.

 

So I found out how to read from other particles. While I think this is undocumented , looks like

attributeBuffer

, and all of its data, is accessible in a custom block. Unfortunately there seems to be no way (without patching generated shaders) to add additional buffers, but I might be able to work around it with textures instead.

Some requests, based on what I've found so far:

1) Are there plans to make custom blocks not internal, so that users can implement more complex behaviors that would take dozens or hundreds of nodes/operators? Right now I'm working around this by patching in a file adding a friend assembly once the package installs. This is simple enough, and I know everything's early, but I'm hoping exposing this API is on the radar—one of the things we really struggled with in the old particle system was the fully sealed-off UI.

2) Unless I missed this, I'd love to be able to add other custom buffers (not just textures) for use within blocks or as inputs. EDIT: I just realized that blocks support includes, and experimental custom attributes exist. This should work fine!

3) I would really like the ability to override the default Euler integration. From my scan, it seems this is an invisible block added. We use Euler, Verlet, and some other obscure integration methods for simulation. EDIT: I see this is configurable ("none" can be chosen in the inspector for the visual effect)

4) I noticed a

mass

parameter in the generated shader code, always set to 1. Is that editable at all? It seems it's used with velocity field forces. For consistency purposes, I'd love to be able to set that mass value to the same mass we use for fluid simulation.

Digging through the implementation, this is some great stuff! The way the graph compiles down to clean shader code is top-notch, and it really shows how far Unity's HLSL compute pipeline has come

 

Fixed? Its not broken. It has never been for built in, and that has been made clear from the get go, including in this thread.

 

Actually some confusion on this subject was probably caused by a number of posts on the first page of this thread.

This being the most obvious example: #23

 

Ahh! This makes sense now, thanks!

 

@VladVNeykov thanks for clarification. So we can definitely forget the legacy render path of the package. Meanwhile i fell back on the mesh gpu instancing option of the particle system, it works great.

 

I'm trying to create a collider/attractor plane at the bottom of the viewable screen. Is this currently possible?

 

Hello, does anybody know how to change sorting mode? Like at shuriken particle system? I've got wierd issue with smoke particle

 

Huge thanks for the link and the additional info @ThomasVFX!

I actually got an initial release out here:
https://github.com/thinksquirrel/fluviofx
https://twitter.com/getfluvio/status/1082598344466333696

Will definitely take a look into that repo- we had to hack together a few things to make our initial release work and I'd love to clean some of that up/keep in tandem with the API as it develops.

I do have one bit of feedback though. I actually ran into a bit of a wall trying to implement spatial partitioning for particles. Basically, we need to be able to access a read/write buffer or texture _somewhere_ on the GPU in order to store grid positions and neighboring particles. Attributes haven't worked here because the sizes don't correlate to particle system capacity at all. Do you know of any way we could do this?

Also, are particleIds and indexes guaranteed to be the same? If not, would love to have some way to access the current index in a block somehow. I wasn't sure if I missed that or not. We've gotten around this so far by patching the shader files after they are generated, but it's a pretty ugly workaround.

 

@JulienF_Unity that's how i understood it, anyway that's good to know that it is useable again at the moment.

 

I feel like an idiot.....but is there an equivalent to Rotation Over Time\life? I assumed it was the "Set Angle over life" block but that doesn't do anything no matter how much I adjust the curve. I feel like this should be pretty simple and that I'm just missing something obvious.