Showcase Lattice Visual Scripting for ECS

Hey everyone! I wanted to start sharing updates on a project I've been working on full time for a while now. This system has been a long time coming, and is actually the 3rd iteration of several designs I've had for visual scripting engines.

Like many of you, I love a lot about ECS, but have found that writing gameplay code is like pulling teeth. So for our next game at Pontoco, I've been working on a node-based system to help with that. Lattice Visual Scripting is a general visual scripting environment, tightly tied to Unity's ECS. It takes inspiration from Houdini, UE Animation Blueprints, and Bazel.

Lattice is currently in a pre-pre-alpha state. The groundwork is here, and the design is solid, but there remains a ton of work to be done before it's anywhere near production-ready. Notably, ground has not yet been broken on the C# compiler, which will be necessary for any meaningful performance. I'll be posting updates as I work on it, so stay tuned. I'm happy to answer questions in the meantime.

Some of my goals of the project are:

Visual scripting built for programmers. Lattice is intended to be used alongside C#. You write nodes as plain C# functions, and stitch the larger architecture together with nodes. The magic of Lattice is to abstract away the 'glue code' between your meaty logic, freeing you up to write relatively simple C#. There are no 'add/subtract' nodes in Lattice -- that sort of thing is better suited to text-based coding.

Fill the missing gaps in the ECS user experience. Writing gameplay code in ECS is a struggle. We'd like to make this as painless as possible. To that end, it's a unified system that supports several types of game logic:

• Bake-Time: Nodes can execute at bake time, and package custom data into the entity. Ex: Drop a node to get a prefab reference.
• Entity Authoring: Lattice Graphs can add ECS Components and define logic between entities. Ex: Defining how several sub-entities making up a complex character parent entity interact.
• Gameplay Code: You write declarative, high-level nodes like a generic "Ability Cooldown" node.
• Presentation Logic: Lattice is well-suited to animation blueprint-like behaviors. Ex: This entity is Red when list 'Keys' is empty.

Prioritize Visibility / Debugging. Debugging is the slowest part of writing gameplay code. We can do a lot better than code-based debuggers. Nodes in Lattice only have a single value -- and they're only ever calculated once. This means you can pause the game and inspect the output of any node in the graph. This is a subtle, but deeply important part of the language, and makes debugging incredibly fast. If a value seems incorrect, you can simply walk up the chain of dependencies to figure out which node went wrong.

Designed for Performance. Lattice is not performant (yet), but it's designed to be quite fast. It will never be as fast as hand-tuned bursted code, but it has a few things going for it. All nodes are written as static C# functions operating on unmanaged data types. All nodes form a data dependency graph, so computation is inherently parallelizable. Lastly, the node language is explicitly designed for inlining and compilation to plain C#. Performance is being set aside for the moment while I iron out the ergonomics, though.

Here are some samples:

A simple Lattice Graph implementing a dash ability.​

The definition of the 'AbilityCooldown' node above.
​

A Lattice Graph implementing a flappy-bird-like platformer.​

Current Status: Largely the big pieces are in place. You can write scripts, build logic, and read and write from ECS Components.

This will likely be a long on-going thread. Feel free to ask any questions as it develops.

 

Very cool! I'm glad other programmers like you acknowledge that visual scripting is more than "for designers that don't know C#".

My feedback on your current ergonomics: I have difficulty understanding what are the inputs and outputs of nodes. I don't know their purpose, type and direction.

Looking forward to updates!

 

Thanks!

Yeah, the interface is much easier to understand with a video walkthrough. A few things though:
- Values flow downwards, like reading a book, or code.
- Types input/output names are visible on hover. I'm considering a color-edges-by-type mode, but it turned out to be quite noisy.

 

Cool, I'm excited to see how this develops!

Personally, I'm a fan of how shader graph shows the names for all inputs and outputs. Makes it easier for me to follow. Hover tooltips are good, but I don't think they should be the only way to see this information


But I also like how you draw the default value inline, right next to the parameter name for optional parameters.

I know adding names to all inputs and outputs would probably require moving the connection points to the left and right of the nodes, which changes your downward flow design, but I still would prefer it working this way.

 

Yeah, I can see that argument. My counter-argument is that once you get to a big enough graph, those names take up a huge amount of space. I'm aiming for a level of detail closer to text code, so nodes need to stay compact.

 

I think it is a good idea to make it toggle-able then, the empty circles will severely kick back the readability of any teaching material which isn't inside the editor. So any screenshots of graphs will be essentially useless to teach anything other than the "I did this" and not "I chose this among these options because...".

 

Yeah, I'll keep that in mind when I get to the stage of working on learnability. Right now, I'm squarely focused on making the tool extremely efficient for experts (ie. me who designed it, haha). Once I'm at a happy level of productivity with the tool, I can circle back to making it easier to teach.

 

Sorry for the straightforward feedback, but I don't think the tool is usable currently for anyone else besides you, who designed it.

The idea is very cool, but as a visual scripting tool it is expected to help visualize things better, so if you are hiding such essential information it ends up going against its own goals.

If you want the tool to have a wider audience, including teammates that didn't work in the tool directly, I think it would be valuable to listen to the feedback here, as so far 4 out of 4 (including me) commenters pointed out the same issue on the tool.

In any case, amazing project, looking forward to see it on GitHub on Asset Store to better follow its updates!

 

Don't worry, I'm very interested in UX and visualization, haha! I have a long history of experience there, and we'll get there.

My philosophy about Visual Scripting is that too much time is spent too early on UX, and not nearly enough time on the 'fundamentals'. Ie. "How does the language operate? How does the type system function? What are the perf characteristics?" The ux-first approach makes for very pretty, very learnable tools that, at the end of the day, can't do much that's useful for me.

The project is still very early, and I'm not too worried about syntax right now. That's all stuff to be designed somewhere down the road once I know the language is useful. But the feedback is always useful!

 

It's been a while, but development has not stopped. In fact, nearly all of the underlying runtime code has been rewritten. Here's some of the latest work:

Lattice IR
The previous version of Lattice executed the authored node graph assets directly. This was clumsy, and limited the ability for the executor to analyze the broader task graph. It also linked the implementation of the runtime to the editing experience, which was a major pain to work with.

The latest version of Lattice compiles edit-time Lattice Graphs down into an IR, or Intermediate Representation. If you're not familiar with compilers, and IR is a different representation of the same input program, but usually with simpler primitives and some amount of information erased. For example, in Rust, for loops are compiled down to simple loop/break statements in their MIR representation.

One interesting simplification in the IR in lattice is that every node has only a single output, where the edit-time nodes may have several outputs. Pragmatically, that means a node like:




Is represented in the IR as:



Multiple output ports for a node are represented as several single-value nodes that 'project' the field from the tuple object. Because of the dependency structure, field nodes you don't use will never be executed.

The IR means the Lattice Runtime is now much simpler, operating on a very trivial number of custom nodes. It also paves the way for group nodes, advanced analysis, and compilation to C# long term. Additionally, this IR forms a full global graph for the entire program:

Global Graph
One of the key features of Lattice is that all scripts are unified into a single global compilation unit. Pragmatically, that means that your scripts can depend on values in other scripts. For example, a "UI Display" node in the UI Lattice Graph can depend on the "Player Inventory" node in the player graph. Or an "Enemy Behavior" node could depend on "Player Velocity".

Broadly speaking, this means that Lattice forms a global ordering of all tasks within the frame. Importantly, this is all done at compile time. There is no runtime scheduling of tasks in Lattice, like in the Job system, or the Render Graph. Lattice Graphs are compiled as you edit them, and the unified global IR is executed at runtime (long term, compiled to C#).

Type Checking
With the IR comes some basic forms of type checking and type inference. This is nice because it also allows type coercion. Ie. You can plug an int? into a bool and it will automatically cast it using the nullable status as the boolean.

Types flow through the global IR, allowing you to also define nodes that pull their type information from their inputs. For example, a generic "PreviousFrameValue" node, which takes an input and returns the last frame's value. This can now properly type-check, instead of having to use the 'object' type.

On the subject of types, Lattice uses the C# type system, with a few extra constraints:
- A single nullable type. C# has several: Nullable<T>, Entity.Null, and managed 'null'. These are all merged under the hood into a singular Maybe<T> type in Lattice type checking. This does not change runtime behavior, Lattice just automatically converts between them.
- Entity is implied to be non-null. Lattice will generate an error if you return Entity.Null for type Entity. You must specifically call this out using Entity?, instead.
- Propagating exceptions. Every node can throw exceptions, and these automatically propagate downwards. If an input node throws, all dependent nodes will also return an error, allowing you to follow the chain upwards to find the error.
- No generics. I'm not opposed, and it's perfectly possible, but implementing them is out of scope at the moment. I'm not in the busines of writing a Hindley–Milner type inference engine (yet. )

But largely, the goal is to stick close to C# because at the end of the day, all nodes are implemented in C# anyway, and need to use those types.

---

That's all for now. Currently I'm implementing features as needed for our new project at Pontoco, for which we're using it to author gameplay abilities (dashes, attacks) and some simple enemy AI (enemy types).

A loose roadmap for now:
- Null propagation. If you pass "int?" to a function "bool greaterThanZero(int input)", the function will automatically be lifted to "bool? greaterThanZero(int? input)" and pass the null value through. This is very helpful for stitching gameplay together where many entities may not exist.
- Group nodes. Think "Sub-Graphs" in shader graph. But with different and hopefully much simpler semantics / editing workflows. Group nodes act sort of like function calls, except they are fully inlined in all instances.

Right now Lattice is still changing way too much to release something to play with, but heading in that direction! For now you'll have to enjoy the updates.

 

Will be curious to get to play with it

 

Lattice has met a major milestone this week: programs now fully compile to flat, plain, .NET IL!

I’m very proud of this work. One of the goals of the Lattice project was to design a visual language that could compile down into fast linear code — effectively just the procedural code that you would write if you were coding it by hand. This is very much inspired by Rust’s tradition of “zero-cost abstractions”. With the new compilation pipeline, the node-graph representation is stripped away entirely. Outputs of nodes become local variables and bodies of nodes become plain static methods.

Originally I planned to generate C#, but IL was a much better option for a few reasons:

• Generating C# requires a Domain Reload to compile which is a non-starter if you're editing scripts while the game runs.
• Generating C# would require shipping a C# compiler at runtime if you want to modify scripts during standalone play.
• C# is a sloppy thing to generate. There are a lot of syntactical concerns you get bogged down in just trying to get something valid.

IL, as it turns out, is trivially easy to emit in-process with Reflection.Emit, and is actually really simple to work with. I use the Sigil library which is a validating wrapper which catches a number of type errors, etc, during generation. Plus, IL can do several things that C# can't like calling private methods, etc, which just makes the whole process smoother.

This is all possible because of the IR representation I added to the compiler a month or two back. For example, my integration test graph blow is represented under the hood as a larger graph of simple primitives..



This IR is critical because it reduces the complexity of the next step of the compilation: code generation. While there may be hundreds of nodes available for user scripts, in the IR there are only 7 distinct types of operators:

• Function (a handle to a static C# method)
• Previous (allows referencing a node value from the previous frame)
• Entity (returns a handle to the current Entity)
• QualifierTransform (allows referencing other entities dynamically)
• Barrier (execution barrier, waits for all inputs to finish)
• Collect (collects several values into an array)
• Malformed (a stub node that returns an error. Used for syntax errors)

The IL Generation step only needs to implement generation for these 7 operators, dramatically reducing the complexity. In fact, the IL Generator is only ~500 lines of code. The final code looks something like:


With the new backend, Lattice is now quite fast — as fast as C#! I still need to do comparative profiling, but I’m fairly confident Lattice is by far the fastest visual scripting system in Unity by a long shot. Bolt, NodeCanvas, and Playmaker all interpret their node graphs. Lattice emits a single static method that executes all lattice scripts in the game in a single pass. The .NET and Mono JITs eat pure static methods for breakfast.

This work also enables some interesting next steps for the compiler:

• Automatic parallelization & jobification (Unity Job system)
• Burst compilation of Lattice Graphs

However, Lattice is now plenty fast for my needs. So for the time being, I’m pivoting back to working on gameplay workflows and UX. We have some needs on the OST project at Pontoco that need some feature work in those areas.

I know ya'll are hankering for something to play with. Getting there as quickly as we can.

 

Oh, and another benefit of the IR: several different tools can compile down into it. For example, I'd like to have a separate interface for defining statemachine-like graphs. Both this and the current value-flow style graphs can compile down into the same representation for the compiler.

You could even imagine letting folks make their own custom tools that compile down into Lattice IR, just for performance.

 

I'm curious! And I have a myriad of questions that pop into my head; I'll limit myself to a few hopefully useful ones:
- How does it integrate with the ECS?
- How/where are the graphs stored?
- Can we invoke/schedule the graphs when/where we want (e.g. jobs)
- How do we invoke the graphs?
- How are its dependencies managed?
- How do you imagine parallelizing the graphs (e.g. what are it's sync points)?

As for visual scripting systems that can compile to C#, uNode is a system that can do this.

As a programmer, the only graph system that I found useful to work with was the shader graph. For everything else I've never really been able to get along with visual scripting. I'd be very curious to see examples of gameplay systems where it shines, so that perhaps one day I will get along with it!

 

Hey thanks for the good questions.

Yeah, I completely agree! It's one of the reasons I started this project -- I truly believe there is a visual gameplay system that would make me more productive, but I haven't found it yet! I'm hoping to show more gameplay examples. It's tough without getting a whole video setup. For now, here's an example weapon behavior I wrote recently for our project. It's a special weapon that boosts the player and can jump off of walls.



A big part of my goal is to have few nodes in a graph. Most of the 'mungy' stuff like math and conditionals is contained within the node bodies in C#.

- How does it integrate with the ECS?
There are nodes for every ECS component. They can be driven (written to) by passing a value in from the top, and read from by reading the value out the bottom. This is from my spring example that drives a box's LocalTransform position with a spring and keyboard input.



One thing that's unique about Lattice -- it has no execution wires! The system is declarative, more like Shadergraph, than Blueprints. You would read this as "The position of the LocalTransform is equal to SpringPosition". Although this seems simplistic, with a couple of extensions this is actually quite nice. (More on this in the future)

- How/where are the graphs stored?
Graphs are just assets right now (ScriptableObject). Similar to Bolt, or C# scripts. You attach them with an authoring component.

- Can we invoke/schedule the graphs when/where we want (e.g. jobs)
Potentially. The system compiles a global graph for the entire process. And you can execute that whenever you like by calling the generated delegate.

That said, I'm currently working on adding 'phases'. Ie. Some nodes that run at earlier parts of the frame, and some that run later. This is useful if you want to write into an ECS component, let a different System run, and then read again from the component later. So in the future, each node in the graph will be associated with a specific time in the frame that it executes.

- How do we invoke the graphs?
Currently it's automatically invoked in the LatticeExecutionSystem. But that really just calls into the execution graph. The graph is just a big static function, so you could theoretically execute it whenever.

- How are its dependencies managed?
Dependencies on the asset side are handled like normal assets. You can also make node dependencies between graphs. For example "Enemy A charges at the player, if they're holding item X".

- How do you imagine parallelizing the graphs (e.g. what are it's sync points)?
Parallelization can go two ways:

• Per Entity: Each node is tagged by which entities it executes for. These can be split pretty naturally like IJobParallelFor, when there are many entities executing the same graphs / nodes.
• Splitting the graph: Because the graph is composed of pure functions, nodes that depend on different values can be executed in parallel. The challenge here is determining how best to split this up, keeping in mind the overhead of scheduling (you wouldn't want to run *every* node in parallel as a job, that's too many jobs).

Keeping everything as static functions makes this really nice because we can introspect the precise data dependencies of each node.

 

Oh, and thanks for the tip off about uNode. I'd skimmed it, but hadn't noticed the C# generation.

 

Awesome! I like the declarative approach. It feels like it is adding a constraint, and while that means it perhaps won't be good at certain things, it also means other things may flow very naturally from it. The declarative approach to evaluating nodes exists in more graph systems (like uNode, too), but there's always an execution wire needed somewhere.

Looking forward to seeing more!

 

Yeah, that's exactly it. It's a bit of a constraint, but there's two escape hatches:

• You can reference any value from the previous frame (ie. carry state forward)
• Node bodies are C# functions, so they can do any looping/recursion/etc you might need.

In practice, it's also fine for Node bodies to do mutable operations like writing to ECS state, static variables, or acting on passed in references. So while the graph is composed of pure functions, they only need to be 'effectively pure'.

These combined, it's not as limiting as it first seems, and you still get the benefits of the declarative structure. (That's what enables the compilation pipeline / analysis)

 

where can i get a test package now？

 

And When.....

 

Glad you're interested. Lattice isn't released yet. The first release will likely be an early-adopter v0.1.0 folks can play around with, although I don't quite know when that will be.

The current status of Lattice is that it is largely an execution engine right now. It's an optimized, ahead-of-time task graph for executing gameplay code. Honestly, if you read the above posts, the UX / editing workflow will seem "low-level" (ie. a lot of very basic C# functions).

However, the vision for Lattice is actually very "high-level":
- Composing big blocks of gameplay behavior.
- Avoid 'noodley' coding in the visual script.
- Declare "what" you want to happen, not "how".

I should probably write up a longer article about my thoughts on visual programming design, honestly.

In any case, I'd like to push on some of those workflows, first. The groundwork is all prepped, but it needs a lot of UI Toolkit work on the editor, before I let folks take a first crack at it.

 

I wrote a new blog post, this time on language design. It's about the differences between "execution graphs" (ie. Blueprints) and "value graphs" (Shadergraph, Geometry Nodes, etc). Specifically, why systems like ShaderGraph feel flexible and composable, and systems like Blueprints almost seem to reject refactoring and organization.

https://johnaustin.io/articles/2024/composability-designing-a-visual-programming-language

Also, a major project update! I've hired @vertxxyz (aka. Thomas Ingram) on a short contract to help flesh out the Editor UI for Lattice. Thomas is great because he's a UI Toolkit expert, which I am not. He's been modernizing the editor architecture, and adding some *really cool* editing UI/UX features.

Our end goal here is to make the tool feel Unity native.

 

@JohnAustinPontoco reading that blog post I feel quite exciting about your approach. I've given up on every visual scripting solution from now and then. But your approach seems fresh and interesting. I would surely try Lattice when it's ready.

 

Hey all! We're getting close to a preview build. I'd love to gather a small group of folks to play around with it, it'd be helpful to get some feedback and see people using it in real-time.

If you're interested, I've opened a channel in our discord:
https://discord.com/invite/Qx4aX6Xkxr