Search Unity

How a team of five created a next-gen MMO with Unity and SpatialOS

Discussion in 'General Discussion' started by ggambetta, Feb 18, 2016.

  1. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,808
    Interesting! I am leading a team of 6 to build a small MMO game and the challenges are definitely there, especially a game that focuses on interaction and community rather than the typical "MMO that really is like a single player game". I can't wait to hear more, especially with regard to larger worlds that are not islands. ;)
     
  2. Farelle

    Farelle

    Joined:
    Feb 20, 2015
    Posts:
    504
    I really would like to understand more about it, but I'm not sure I understand networking well enough yet to ask the right questions of what interests me :) Also my own project that I would love to use with such a technology is still in it's concept phase :eek:

    okay, even though I might ask stupid questions, due to not knowing much, here it goes:

    1. even if the server side is supporting 10k players, would it ever be reached (and feasible) having them occupying the same area, considering that todays computers might graphically not be able to handle it?
    2. what about hardware of servers= are you providing a service for that aswell? How is your network code getting to those servers (in case you don't provide them)
    3. I think you mentioned that the workload is being distributed between workers and uses a kind of cloud system, to handle that, but where are those workers practically? are they physically servers? or just some sort of program running on one machine or several programs on one machine?
    4. could I use spatial OS with as example smartfoxserver?
     
    Teila likes this.
  3. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    But this isn't at all what they're talking about. 10,000 players in a game does not necessitate 10,000 players fully synched on each others' clients. In fact, that's a fundamentally bad idea in any case for a number of reasons - it wastes resources, wastes bandwidth, enables cheats/exploits (ie: you shouldn't send data to a client that it's player shouldn't be able to perceive), and - as others have pointed out - wouldn't actually be useful on the client side because you couldn't see all of that at once anyway.

    So yes, they potentially can claim to have 10,000 player FPS support. There are still plenty of valid questions as to how effective it will be, but the fact that showing it all at once on the client is a challenge doesn't rule it out.
     
    Socrates and Ryiah like this.
  4. GoGoGadget

    GoGoGadget

    Joined:
    Sep 23, 2013
    Posts:
    689
    I'm fully aware of what they're talking about, but they're actually claiming to be able to have "10,000 players in a single massive FPS battle?", not just support for a 10k max-player FPS. I wouldn't have as much trouble if their wording wasn't intentionally spun with PR to lure in inexperienced developers and puff their system up.

    Either way, this is a fairly pointless debate, since no-one will ever release a 10k player FPS with 'SpatialOS', and a number of other people here and on other forums have pointed out the other issues you'll run into when using it in production.

    I'm not saying that they haven't built something impressive, or that it's a totally useless system, far from it. It obviously works well for Worlds Adrift, but there's a big difference between that and a 10k player FPS battle.
     
    Last edited: Feb 22, 2016
    Tomnnn likes this.
  5. JohnnyA

    JohnnyA

    Joined:
    Apr 9, 2010
    Posts:
    4,682
    @GoGoGadget didn't improbable get 20m+ in funding to build this service? Are you confusing the 5 guys from Bossa Studios with the (now) 100 or so from improbable building SpatialOS for the last 4 or so years.

    Its not so unrealistic that you could build something pretty reasonable for that kind of money. If its a sustainable solution for indies, well thats another question, one which the success or otherwise of Worlds Adrift will help to answer.

    EDIT: Added some details, shuffled around.
     
    Last edited: Feb 22, 2016
  6. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    What's the practical difference between those things?

    Anyway, they've said 160km x 160km. If we accept that, then that's a bit over 25,000 square kilometers, which is plenty of space for said "battle" while allowing a really sparse population density.

    So it's probably not going to allow a 10,000 player game of Unreal Tournament, if that's what you're driving at. But a simulated wargame where 10,000 individuals are playing roles in a NATO theatre of operations? I wouldn't necessarily rule it out.
     
  7. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @zombiegorilla Geographically? Zürich. Teams? Two years as an SRE for Dremel (the BigQuery backend -- see paper), two years as a SWE in the account abuse team (the thing that prevents automated logins via web and Android, among other things -- see blog post)
     
    tatoforever likes this.
  8. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @GoGoGadget My offer to have a technical chat still stands :)
     
  9. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @angrypenguin As you correctly point out, density is an interesting issue. That being said, we do support much higher density than 10k players in 25,600 km2.
     
  10. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @JohnnyA We got $20M from none other than Andreessen-Horowitz. SpatialOS itself has been in development for three years. Most engineers come from Google, Facebook, or hedge funds, or from Cambridge/Oxford and the like. More info here.
     
  11. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @Carve_Online You're the Unity experts :) We provide massive world simulation technology with tight Unity integration, then you can figure out the best way to render that in a beautiful and efficient way. Of course SpatialOS is smart enough to send down to clients updates only for the region of space they're interested in, not the whole world.
     
  12. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    Oh, I assumed you would. I just went with those figures because it's quite easy to accept that a distributed network could handle that.

    The real question, as I think @GoGoGadget was driving at, is where between the two extremes I described your system can fit. There are non-software issues, there, as well. You advertise 10,000 players, but how does that actually work? I accept that you can probably get those numbers, but what kind of activity can you have at that scale? A traditional FPS is very sensitive to latency, and you're spreading work out over multiple servers, so... is my ability to shoot people going to vary depending on where in the map I'm standing? Or does it depend whom I'm shooting at? Or..?
     
  13. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @angrypenguin We're talking traditional, sensible-to-latency FPS gameplay. Work is spread between multiple servers within the same datacenter, so they have extremely low internal latency. Whatever latency you get as a client will be dominated by whatever latency you get connecting to the deployment across the Internet, which isn't any different to what you'd get by using a single server. Our tech is still subject to the speed of light ;)
     
    tatoforever and Teila like this.
  14. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    3,967
    I'm interested in SpatialOS, where is the buy/download button? :cool:
     
  15. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,808
    Yes, we would like more information as well. I did a search but could only find articles about you guys, nothing really about the tools.
     
    tatoforever likes this.
  16. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,569
    So you solved the problem of the built in latency hit you take having to go through Update? Most people know you can only access most of the core unity functionality in the main thread. Less are aware that what this means for networking is that you take an automatic latency hit because your data once it reaches unity has to wait until the next Update before it can be processed.

    Most never hit this because the common usage of unity on the server is traffic only hits a single unity instance at most. But it's a real issue in a distributed system where you might have to talk to a number of unity instances to get the data you need to send back to the client. Take a good sized pvp battle with say 50 players or more where the majority of those players are on different nodes, and say you need to do a spherecast over the area they are in.

    The above issue is the wall I hit trying to integrate unity into a distributed system. With each call to any single unity instance having at best a latency hit averaging 2ms, you can only really distribute so much before the latency matters,and you have to go back to localizing stuff. You can decrease the latency by increasing frame rate, but that has a direct impact on cpu usage and also messes with the timing of whatever logic you have running in Update. And it's still not very deterministic because the latency hit is determined by what point in the Update cycle that instance is at.

    Now there is more then one approach to solving the above problem. But there is really only one approach that is efficient, and that requires solving the above issue which I'm hoping you guys have done Every other solution involves duplication of resources or just simply localizing a lot of things and giving up most of the benefits of a distributed system.
     
  17. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,569
    To clarify, that 2ms latency hit was with framerate being boosted way up to around 200 if I remember correctly. Default framerate results in much higher latencies and standard deviations.
     
  18. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    tatoforever and Teila like this.
  19. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @snacktime We don't use Unity as the server running the simulation; that's what SpatialOS does. In this context, Unity engines are workers that connect to the simulation to do something, either computing part of the simulation (e.g. the inside-the-cluster instances we use to compute physics), or visualising its state and sending player inputs (e.g. a Unity game client connected across the Internet). More details here.

    The built-in latency is, as you say, built-in, and we make a point of using unmodified third party software as workers. But this latency is not cumulative, there's no big game loop or central simulation node waiting for lots of physics engines to send their updates upstream before they're processed; the whole system is extremely asynchronous. As you point out, this has some implications about determinism, but with a mostly-deterministic system being synchronised continuously, this works surprisingly well. Look at this blog post for an extreme example of this.
     
  20. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    3,967
    @ggambetta,
    So with that worker distribution system, simulations are mostly bound by client hardware? What happen when a client cannot handle large amounts of simulations, let's say it's a mobile tablet. How do you handle that?
     
  21. BFGames

    BFGames

    Joined:
    Oct 2, 2012
    Posts:
    1,543
    Thing with the 10.000 FPS battle is that i don't doubt SpatialOS could handle it server side, looks really nice. but even with low poly models, with a special 6 bones setup, and animation and model LOD, clients wont be able to handle it on their side with normal computers.
     
  22. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    Sure, but lets say I want to perform a raycast (or series of raycasts) over a long distance and it overlaps the physics areas being simulated by 3 Unity physics workers. (This might not even be a long raycast - it could simply be near the corner of an area.) What might that look like in my code, and what impact does that have on the speed of the query?

    I realise that this might not be specific to your own technology, but it's a use case anyone combining it with 3rd party tools like Unity, as you're advertising, is likely to have to overcome.
     
    Tomnnn likes this.
  23. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,569
    That simply makes no sense in the context of my question.

    Spherecast is just one example out of a class of problems where you have to wait until all results are returned before you use the results. If that spherecast needs to include enough 'things' where they aren't all on one instance, you have to play some kind of trick to make it work.

    At the end of the day I actually gave up trying to solve this problem in the context of distributed systems, and FYI I've been working on this problem longer then anyone at Improbable has. I'm actually fairly convinced that it can't be solved unless you change Unity or you do other things that basically take away most of the advantages of using a distributed system to start with.

    Plus, large scale physics can't be asynchronous, that's not how physics works. Maybe you fake it good enough for some use cases, but saying you can distribute physics using unity is not strictly accurate. There are gotcha's there somewhere and you really should be stating those up front in your materials. That's kind of a giant red flag in my book for a company that's creating a developer's tool/platform.

    Your physics examples are kind of strange actually. Your example is just not typical of problems that those of us making large scale games face. And the ones that are more typical I would say are much more challenging.
     
    tatoforever likes this.
  24. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    My interpretation is that any quirks arising from the way that our "workers" do things are for us to solve.

    I've never tried to solve this, so I could be way off, but my first guess would be making the whole system run in lock-step, and running physics queries at the beginning of the step after they are received. Even that, as you say, results in some built-in latency.
     
  25. Tomnnn

    Tomnnn

    Joined:
    May 23, 2013
    Posts:
    4,137
    That reminds me. @angrypenguin might know. What's the difference between a sphere cast and checking distance to a point? Wouldn't checking for objects being within a radius of a point be less expensive than a sphere cast? Or are they the same thing?
     
  26. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    12,360
    A sphere cast doesn't check distance from a point, it checks distance from a line.

    Edit: You can kind of think of a sphere cast as a "fat raycast". It's a raycast that also has thickness.
     
    Tomnnn likes this.
  27. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @tatoforever The amount of physical things you can have in your game world is not bound by client hardware. The Unity engines used to simulate physics run in the cloud and are managed by SpatialOS. If the load is too high, it just starts new instances (and it also shuts them down when no longer needed, to minimise costs).

    Client hardware does impose some limitations on the amount of things you can show at the same time. Your interactions with the game world won't be the same if you're using a powerful PC or an iPad (or a VR headset for that matter), in fact you could play as a FPS in your PC and have an overhead tactical view on the iPad (or just lower poly models). The interesting thing is that the game world underlying all of that can be really really big (both in terms of scale and complexity).
     
    tatoforever likes this.
  28. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @snacktime Oh, I think I see what you mean. We provide our own version of spatial queries, these happen in the cloud and are based on the canonical state of the simulation that SpatialOS holds. These are very fast because they happen between servers in the same datacenter, and since the physics engines are also in the datacenter, most of the time you don't even have to send it back to clients.

    Regarding distributed physics, sure, the co-simulation example in this blog post is extreme on purpose, to push the limits of what it can do. We hope we'll never see that happening in a real game :) What happens in general is that different physics engines are responsible for mostly disjoint areas of space, but since we don't do shards or regions but have a big continuous region of space, there has to be co-simulation at the overlapping boundaries of the areas of authority of each engine. The example shows how we deal with these areas with co-simulation, and that no artificial boundaries are needed.
     
  29. ggambetta

    ggambetta

    Joined:
    Sep 8, 2015
    Posts:
    25
    @angrypenguin Unity is well supported. We provide a "standard library" that makes it very easy to do things with Unity and SpatialOS, but we have to work within the limitations of Unity.

    Most of the time, at least. For example, coordinates are floats in Unity, and in our tests that limits you to a radius of 5km or so before you start losing precision (assuming 1:1 scale and sub-millimeter accuracy; this is obviously different if you're modelling the solar system at coarse scale). On the other hand, SpatialOS coordinates use doubles, and we found ways to make all this completely transparent to you. In other words, you can make Unity games where you fly through a continuous world 200km wide, even though Unity itself can do 5km.
     
    Socrates likes this.
  30. Tomnnn

    Tomnnn

    Joined:
    May 23, 2013
    Posts:
    4,137
    Oh, right, I sort of remember that. They really need to rename that thing!
     
  31. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,808
    Fly as in overhead, such as a flight simulator? So does that mean that a none-flight simulator is limited to 5km?
     
  32. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,569
    The actual limit varies a lot, 5km is an extremely conservative number for the majority of games it's more like 10km and if you center your world at -10km you have 20km effectively.

    But the problem he is speaking of is one explained fairly well here. It sounds like what SpatialOS does is they have a global coordinate system in doubles, and then each physics engine is probably running with an origin of 0,0 or the center of the area it's covering. It's actually fairly straight forward it just gets difficult to reason about without a good abstraction layer and the right tools, especially if you are debugging stuff.
     
  33. CDMcGwire

    CDMcGwire

    Joined:
    Aug 30, 2014
    Posts:
    133
    I have no intentions of working on a game like this anytime in my reasonable future (but as I've learned over the last year, the future is unreasonable) but this interests me from a computer science standpoint. This kind of thing is something I should look into when I start my research at schoo.
     
  34. Freakyuno

    Freakyuno

    Joined:
    Jul 22, 2013
    Posts:
    138
    I was going to seriously consider it for the project my team is spinning up...but because of this...we're out.
     
  35. Freakyuno

    Freakyuno

    Joined:
    Jul 22, 2013
    Posts:
    138
    @ggambetta - Do you have a Skype username you wouldn't mind sharing with me in a private message? I'd love to discuss more technical details with you, as well as the very real possibility of releasing an MMO on this platform, so we need to talk infrastructure, SLA, money, and tech if you're available sometime in the near future.

    We've examined a few possibilities and I've had some indepth talks with the people at MuchDifferent about Pikko, and it's a no start. We were leaning towards Orleans and our own library...again. We've done it 3 times in the past, we were just hoping to avoid it, but somehow we missed your tech in the initial flyover of what was available...
     
unityunity