Feedback Unity & DOTS Supercomputer Cluster Tests and Feasability (Success)

Hello all, I'm back from the depths of the Computer Science Cave and have a new project I'm messing around with. With the ongoing chip shortage it's very hard to get any hardware at a reasonable price so I turned to my old hardware and was like, "can't I turn these into a supercomputer or something if I connect them or something?" Turns out you can using MPI (Message Passing Interface). Put simply, you can connect various computers (nodes) together and run code and have it execute in parallel (or not in parallel, it's your choice). However, these libraries were primarily developed in C,C++,and Fortran. I know that microsoft has thier own API call MS-MPI which is in C#, that is only available on Windows. However, a github repo https://github.com/rogancarr/MpiDotNet uploaded by rogancarr shows how to extend and wrap the C/C++ function from MPICH/Open MPI to C#. With this we could theoretically run a unity application on multiple computers utilizing their cores.

3/4/2021 update: MPICH is officially no longer supported by Windows so I will be using MS-MPI v10.+

Below I will list some tests, information, and status of my progress

Linux Tests:
Install and get MPICH to run: Success
run a MPICH ML program on multiple machines: Success
Build DotNetMpi: Success
Test DotNetMpi: Success
Add DotNetMpi to Classic Unity Build: Success
Play DotNetMpi example in Editor: Success
Build DotNetMpi example: Success
Play DotNetMpi example on multiple machines: Only Tested on a local machine with Success
Play & Build example in DOTS enviroment: Success
Play & Build example in DOTS enviroment on multiple machines: Success!

Windows:
Install M̶P̶I̶C̶H̶ MS-MPI on windows 10: Success
r̶u̶n̶ ̶a̶ ̶M̶P̶I̶C̶H̶ ̶M̶L̶ ̶p̶r̶o̶g̶r̶a̶m̶ ̶o̶n̶ ̶m̶u̶l̶t̶i̶p̶l̶e̶ ̶m̶a̶c̶h̶i̶n̶e̶s̶:̶ ̶S̶k̶i̶p̶p̶e̶d̶
Build DotNetMpi: Success!
Test DotNetMpi: Success!
A̶d̶d̶ ̶D̶o̶t̶N̶e̶t̶M̶p̶i̶ ̶t̶o̶ ̶C̶l̶a̶s̶s̶i̶c̶ ̶U̶n̶i̶t̶y̶ ̶B̶u̶i̶l̶d̶:̶ ̶S̶k̶i̶p̶p̶e̶d̶
Play DotNetMpi example in Editor: Skipped because the editor only tests a single MPI process
Build DotNetMpi example: Success!
Play DotNetMpi example on multiple machines: Success

General Future:

Run DotNetMpi example on Windows and Linux machine concurrently: Success using WSL2
Run DotNetMpi demo with communication with DOTS systems: Success on Linux!
Run DotNetMpi demo with each node performing a specific system function: Technically Success by perfoming an MPI_Send and MPI_Recieve calls.

Well that's my plan for now! Please feel free to leave any questions and comments below and i'll be trying to keep this roadmap up to date.

Oh, and I'm currently using 4 computer nodes for my cluster. 2 old labtops and an old desktop giving me 3 ubuntu computer and 3 windows computers (i have 6 120GB drives with ether windows 10 or ubuntu on it)
I will be creating a github with what I have currently soon.

 

I got a DOTS program running on 2 computers in a cluster....Next is to set this thing on on Windows.....The linux tutorial is going to be chonky

 

3/3/2022: Successfully got MpiDotNet to run communication between 2 unity applications. Currently MPI specific functions must be called on Main thread because of Mpi being a class. Though this really isn't an issue since data processing is being done by jobs. The structure I plan on setting up is something like;
Example Program that calculates complex cloth and smoke simulation;

Program A:

//Program A is Rank 0, with 4 core and 4 treads on Computer A
//Program A is treated at the "Main program"
- Initialize DOTS runtume
- onUpdate(){
If Rank == 0 {
- run collect data for cloth;
- pass any nessacary data to Program B
- run cloth simulation
- wait for B to finish
- combine data and finsh frame
}

}


//Program B is Rank 1, with 4 cores and 4 threads on Computer B
- Initialize DOTS runtume
- onUpdate(){
If Rank == 1 {
- run collect data for smoke;
- run smoke simulation
- pass results to Program A
}
}

now I know you can use the GPU and computebuffers to achieve the same thing but that only works on 1 system.
with MPI we can run the program on multiple computers CPUs and (therorectically) GPUs concurrently. One neat little thing that I notices is that is I set the unity program to run on 1 core DOTS will still (as far as i can tell [i need more testing]) use the other cores on the machine.

 

3/4/2022 Update: got MS-MPI to work and with easy setup and very little code change. the only change is in the include and library links. I built the MpiDotNetApp dll. next I will be testing it in Unity

 

OK, so here are the results:

MPICH & MS-MPI can be used in Unity and work with Unity DOTS!

my library currently is within a class (managed data) so you can't use it in a job but using it in a job isn't probably the best idea because it would be like if a DOTS worker thread suddenly wanted to get user input...in parallel (unexpected behaviour and hijinks ensure). So calls to MPI like MPI_Send and MPI_Recive are called on the Main thread but you can still do a Entities.ForEach() like in the example below

Code (CSharp):

1.  protected override void OnUpdate()
2.             {
3.                  MpiDOTSSystemTest.mpi.MPI_Barrier();
4.                  double startTime = MpiDOTSSystemTest.mpi.MPI_Wtime();
5.  
6.                 Dependency = Entities
7.                 .WithName("MPI_Test_Job")
8.                 .WithBurst()
9.                 .ForEach((Entity e,MPI_Data data)=>{
10.                     if(!data.finished){
11.                         int size = 100000000*(data.rank+1);
12.                         int output = 0;
13.                        for(int i = 0; i < size;i++)
14.                             output++;
15.                     }
16.                 }).ScheduleParallel(Dependency);
17.                 Dependency.Complete();
18.                 //Dependency =
19.                 Entities
20.                 .WithName("MPICommunicationSystemtest")
21.                 .WithoutBurst()
22.                 .ForEach((Entity e,ref MPI_Message_Data data)=>{
23.                     if(mpi.GetWorldRank() == 0){
24.                         Debug.Log(string.Format("Sending \"{0}\"",data.message));
25.                         mpi.MPI_Send(data.message,1,1,0);
26.                         Debug.Log("Successfully sent!");
27.                     }else{
28.                         Debug.Log("waiting on receive...");
29.                         int value = -1;
30.                     //    Mpi.MPI_Status status = new Mpi.MPI_Status();
31.                         mpi.MPI_RecvI(ref value,1,0,0,out Mpi.MPI_Status status);
32.                         Debug.Log($"got \"{value}\" from [{status.MPI_SOURCE}] with tag [{status.MPI_TAG}] with error code [{status.MPI_ERROR}]");
33.                     }
34.                 }).Run();
35.  
36.  
37.  
38.                 mpi.MPI_Barrier();
39.                 Debug.Log(string.Format("Rank: {0} finished job in {1} seconds, total avaialble cpus: {2} with type \"{3}\"",
40.                 mpi.GetWorldRank(),MpiDOTSSystemTest.mpi.MPI_Wtime()-startTime,SystemInfo.processorCount,
41.                     SystemInfo.processorType));
42.             }

- I successfully tested using Mpi in Unity from ubuntu 20.04 to ubuntu 20.04 using 4+4 cpu cores
- I succesffuly got MS-MPI to work with Unity on the localmachine (firewall was annoying so i just tested on my localmachine and it ran so I assume it will work if the firewall settings are proerly setup)
- I successfully ran a Unity MPI DOTS Application (in headless mode) from Windows 10 + Ubuntu utilizing Windows WSL2. Since WSL2 runs on on the hardware like a vm I count it as "running on windows" since I can access the CPU cores.

Now I can post my code and create a tutorial because this journey was messy and pretty ridiculous. documention for MPICH was pretty much from the 90s and after MPICH 1.4, MPICH was no longer supported on windows so I had to research and test the few available MPI implementations on Windows.

So in short: You can use supercomputer clustering in Unity. BUT this does not mean any application can be run in a supercomputer cluster.

since I have a bit of old computers lyiung around doing nothing i wanted to know if I could put them to good use and i think i completed my goal is making it possible in Unity.

Github Link: [working on it]
Setup Tutorial: [coming soon]

 

update: so I ported over some more functions into C# like more MPI_Send, MPI_Recv functions which allow you to send and receive more data types and their array counterparts. This of course is just to get you all started because I cannot possibly know what special datatypes you may create and since I'm in the "safe" context, i'm not using pointers...for now. This means any special sauce you create in C# that you want send over will have to be added by you in the dll! All I have currently is the link of the files on github without a tutorial. I will make the tutorial but it's going to take me some time because I have to make 2 tutorials: 1 for windows & 1 for Linux so please bear with me. If something breaks it's most likly that you have to compile the files on your specific machine and if the code runs but gets stuck somewhere you make have to perform a dumpbin and look and see if your function is listed in the shared library. Also make sure you compile the MpiDotNetApp as a dll instead of a console app before you put it inside your Unity project....have fun!
Use this as Tutorial until I create a new one: https://github.com/rogancarr/MpiDotNet
Github Link: https://github.com/Steven9Smith/Unity_MPI