Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Unity's networking is DYSFUNCTIONAL.

Discussion in 'Multiplayer' started by Aubrey-Falconer, Nov 4, 2009.

Thread Status:
Not open for further replies.
  1. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    .
    I have been actively developing a multiplayer Unity game for over two years, but have run into showstopping issues with every aspect of Unity's builtin networking library:
    • Every multiplayer action game I have seen that utilizes Unity's built in networking is afflicted by one or more of these issues.
    • Although Larus has provided me with test Master Server builds to attempt to circumvent one of these issues, to my knowledge, other than this, these issues have never been officially acknowledged - let alone addressed - by Unity.

    Here's a representation of various networking discussions I have engaged in on the Unity forums:

    I am now desperate to see one of these two things:
    1. A full featured multiplayer action game (including serialized NetworkViews, buffered RPCs, interpolation, networked physics, etc) that uses Unity's networking and that is not crippled by these bugs. If there is just a bunch of things that I am doing wrong, I would love to know what they are.
    2. An acknowledgment by Unity that the built in networking is currently unsound, and a statement that they are working to fix this asap.
    If neither of the above conditions can be met, I need to find a third party networking system that can serve in the role that Unity's built in library is supposed to fill - as efficiently and cost effectively as possible.


    When in the course of human events it becomes necessary for an integral component of a near perfect system to be abandoned by one of the system's most loyal adherents; a decent respect to the opinions of mankind requires that the causes which impel the separation be enumerated before a candid world:
    1. The current Master Server occasionally drops games so fast that they flicker in and out of existence even when they are registered every couple seconds.
    2. The latest beta Master Server is incredibly unreliable. I have tested the Unity hosted beta server, hosted several versions of the beta server on my own dedicated server, and am now connecting my game to DimeRocker's server. All suffer from completely random (and just as debilitating) instances where they can't be connected to for anywhere from a couple seconds to hours on end. Master Servers can be connected to more reliably from native apps than from web players, and I have experienced instances where one client instance could connect to the master server while another couldn't, so I am currently suspecting that this is a client side issue.
    3. The Connection Tester is completely inconsistent. I had to give up on using the Connection Tester in my game because I got tired of it randomly telling me that I was offline or unconnectable. Does it work for anyone, or should it just be fixed or scrapped?
    4. I have experienced many instances where game servers could be successfully connected to after the 2nd, 5th, or even dozenth failed connection attempt. These obviously aren't network configuration issues - so why couldn't the servers be connected to on the first attempt?
    5. My game is now suffering from a bug that seems to grow more prevalent with each Unity release: NetworkViews are set up properly, and transfer RPCs perfectly - but they absolutely will not serialize with each other. This occurs most often in games that have been running for at least an hour or so, although it can happen at any time.
    6. The networking contains a bunch of general bugs and oddities such as Network.Instantiate buffering it's RPCs but Network.Destroy not clearing them and undocumented NetworkMessageInfo.Timestamp behavior. I am sure we as a community could compile a couple more if there was an opportunity of them being addressed

    My game has an active fan base that would love to help with troubleshooting. Lets squish these bugs once and for all!
     
  2. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    Half the problems you mention sounds like you rely on UTs hosted services (for connection tester / nat facilitator), which you shouldn't as they are not for production use and have no guaranteed availability at all.

    I've been using the dimerocker master + nat + connection tester for over a week with 10 parallel headless clients that run as server, and never had any problems to connect to any of them. The only thing I do aside of relying on unity's own functionality is a remove - add host call to the masterserver because the automatic one does not seem to work reliably.
     
  3. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    I run my own master server reliably and I have helped others to set up master servers on hosted virtual servers as windows service now, been ironing out kinks in my service code but all in all, very positive results.

    Send support your issues, maybe they can help you resolve them.
    I know of a few other people using my master server daily and they don't complain or have any issues that I am aware of, and that particular test server isn't running as a service as of yet, adding more trouble shooting text code into my service before I deploy it up to replace what is their.
     
  4. Ethan

    Ethan

    Joined:
    Jan 2, 2008
    Posts:
    501
    Hi,

    in my multiplayer game i host my own masterserver too, but i have to restart my gameserver at least once a day (at night per batchscript).

    otherwise the server would lose connection to masterserver and disappear or get strange physics behaviour or networking errors ("networkview id not found strange behaviour could occur", players dont move on other clients etc)

    i am using rpc (non buffered), unreliable statesync. and players network.instantiate themselves (not by server)

    i thought about allocating the networkview ids manually, but i read about strange workarounds and bugs with it, so i kept my network.instantiates.

    so most of the time i go fine with this configuration until more players connect and disconnect to the same time, which causes ugly bugs sometimes, which are very frustrating.

    ethan
     
  5. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Thanks all for your replies!

    @Dreamora, Zumwalt;
    I haven't utilized Unity's hosted servers in months - so that certainly isn't the problem. Although not as frequent as when I was hosting a master server myself, I have been experiencing connectivity issues to DimeRocker's server - even over the last couple days. I am very excited at the possibility of accessing a reliable master server, and will greatly appreciate all the information you can give me on what is special about your setups. I have gone back and fourth with support over the master server issue for months, but as yet, still am experiencing issues with every master server I can find. I have 11 open support tickets right now (and many more that were robotically closed without a resolution for 2.5.1), so I am kind of discouraged on the whole support ticket process :)

    I have just decided to sign up for a 11 Windows VPS, and give one more attempt to host my own master server. (My last attempt was running on a decently loaded webserver, so who knows - perhaps that had something to do with it's pathetic connectivity). I will also be building a dedicated headless server version of my game to host on the VPS and see if it helps to alleviate issue 5 enumerated above (which is by far the most serious to me anyway).

    @Ethan;
    When I figure out how to work with a master server reliably, I'll let you know! Your Master Server's availability shouldn't have any correlation at all with your game's physics or in game networking. You should definitely be allocating NetworkViewIDs manually - I was able to solve several annoying and hard to track bugs by eventually taking this route in my game.
     
  6. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    Shoot me an PM with your ip / ms fs port info and tonight when I get home from work I will build you up a ms to run as a service and shoot you a downlod link for it, also with 1and1 make sou you add 4 rules to the windows firewall, they would be 2 tcp rules for the ms/fs ports and 2 rules for the udp ports let me know if you need and info or help and I will gladly type up a document on all of this.
     
  7. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Thanks Zumwalt!
    I will dispatch the pm once my new 11 account is activated.

    Hope you find the referral credit useful ;)
     
  8. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    :) always do, FYI, I'll be giving you full code to the project also, so you can tweak it as you want, I just include the build with it, both debug and release along with installation instructions (command line you use and how you use it) to make it a service. I had offered it up to Unity but they don't seem to have a want or need for this build of mine, which is totally fine and cool by me, I have found that through the logs, 1and1 do a maintenance reboot at around 4AM CST to 5AM CST depending on which bank you are on for the virtual servers and the server outage amounts to 4 minutes in total, with this build you will be getting, and running it as a service, you can track service outages.

    It is up and running the rest of the 23 hours and 56 minutes though. The reboot doesn't happen on every night, just when there are critical windows server updates to push out. As I stated though, I will help support you through the roll out and make sure you get up and running smoothly with the latest build. Make sure you have VS 2003 if you want to rebuild it, because upgrading the project to 2005 or 2008 also causes inherent issues with the master server do to changes in the Microsoft Framework and compilers.

    I will get you that build and link in a few hours with full instructions on how to add it and run it on your Virtual Server.

    PM sent with link.
     
  9. Noname

    Noname

    Joined:
    Nov 9, 2009
    Posts:
    7
  10. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    I think its time for me to jump in here again to give some data on my ongoing tests with the Unity Headless Client of 2.6:

    Number of Servers: I have 10 instances running the network example (modified to run optimal as headless clients and additional code to detect if a port is already in use as well as a few lines of code to handle master server updating)

    Current uptime: The servers have been started 13 days ago. Neither did the server crash or require restarts during that time nor did they ever vanish from the Master Server Listing (kudos go to dimeRocker, I'm using their Master + NAT + Facilitator setup for this test. I would assume Zumwalt's would work too)

    Required CPU resources: I'm only connecting to a few selected servers to get usefull and representatitive data on idle cpu requirements vs in use requirements.

    Idle Server: Use 1s of cpu time per day
    Used Server: Use 2-3s of cpu time per day (I'm spending 5-10mins all 2-3 days on them. The levels are the heavier ones means the buggy fun level and 3rd person which does position sync)


    Server: Is my personal homeserver which is a Core 2 Duo E6600, Windows Server 2008 Standard, 4GB RAM. Connected to the web through my regular vdsl home connection (20'000 / 5'000)


    I hope this data will help others to decide if unity networking is for them or to find out if their own code is the cause, which I'm sure is the cause in various cases.
     
  11. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    As far as I am aware, Robur has been up and running since deploying my solution, last I heard from him, only 1 hiccup but that seemed client related and not master server related. Other than maintenance reboots by my ISP, my server has been up constant for 2+ months, and another company in CA that is using my service code has had a server up and running for at least a week as a service (when I finished the sevice code, would have to look again at the logs) and they have been doing testing with very good results.

    The master is a bridge, I can't seem to stress that enough, however, there does seem to be some sort of flaw or issue if you run it as a windows console application that doesn't seem to exist when running it as a windows service. Granted the server that is running in each of these 3 cases mentioned are 8 way processor machines with 1 gig guaranteed ram, up to 2 gig burst ram. Honestly I never see the processor over 5% utilization with the masters.

    (I don't have access to Roburs master so he would have to verify the service stats), plus my version uses the logging feature and stat feature in the logs, so I am pinging the stats every 10 seconds and rotating the log with the default 5 meg feature, this way if anyone who is using my service build has an issue, they can simply zip up the log files and email them to Unity support as is and Larus can easily read them.
     
  12. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Status update!

    Zumwalt's master server build running on a dedicated 11 vps has been performing flawlessly for me around a week now with an average load of 60 or so active players @ any given time, with the exception of an occasional crash-inducing buffer overflow which he is working to isolate.

    I have created testcase Unity projects which demonstrate the Network.Destroy (and much more important) NetworkView Serialization bugs, and they are in Larus's hands. Larus is also working on a number of other bugs which Leepo has assembled into a list.

    I completed coding the dedicated headless server for my game last night, and will be conducting performance testing. So far I haven't been able to successfully get multiple instances of my headless server to run properly side by side on the vps, but this is probably just a minor thing I did wrong somewhere. Dreamora: Does your "additional code to detect if a port is already in use" look anything like this, or am I on the wrong track altogether? :)

    Code (csharp):
    1. MasterServer.dedicatedServer = true;
    2. Network.useNat = !Network.HavePublicAddress();
    3. var port = 2500;
    4. while(true) {
    5.     info = Network.InitializeServer(10, port);
    6.     if(port < 2600  info != NetworkConnectionError.NoError) port++;
    7.     else break;
    8. }
    9. if(info != NetworkConnectionError.NoError) Debug.Log(info);
    10. else Debug.Log("Server Running on port " + port);
    I can't wait to see how my new dedicated server setup will work in the wild! It looks like this situation may have a positive derivative after all.

    Interestingly enough, when hosting the headless server and two client instances locally on my dev machine, I was never able to achieve less than 90 - 100 ms of latency on serialized packets sent from one client instance to the other. Do "serious" fps games have the same kind of networking overhead - and if not, what are they doing differently?
     
  13. Ethan

    Ethan

    Joined:
    Jan 2, 2008
    Posts:
    501
    Go Larus GO!!! :roll:


    I run 2 - 3 headless servers on one machine, and it works fine for me. I duplicated the gamefolder to change the config (servername, port...) and the games exe file name.

    Just to give you a number:
    My game uses unreliable and it has no problems at all with the latency. A good connection has about ~20-30 ms latency (well 2 x ping) over the internet.
    Did you try unreliable for your timecritic stuff?

    Ethan
     
  14. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    @Ethan;

    I was able to get multiple master servers running on the same vps by instructing them to each utilize a random port between 2500 and 5000. My previously posted init code, which started at 2500 and incremented the port number with each successive init fail, (and which worked perfectly on my dev machine), didn't seem to work at all on the server. The second server would start on port #2500 as well, and both would immediately disappear from the master server.

    You have to be very careful when measuring timestamps: the NetworkMessageInfo.timestamp variable is based upon the send time of that exact network packet. When a client sends a serialized NetworkView to the server, that packet is not forward directly to other clients. The server's NetworkView merely updates itself with the data, and then sends the data to other clients on the next NetworkView serialization. To accurately measure lag, you have to add some logic to the server to preserve the timestamp of the original client's send time. Here's a script I wrote which (I believe) implements this correctly: http://www.unifycommunity.com/wiki/index.php?title=NetworkView_Position_Sync

    I am using unreliable NetworkViews exclusively in Mars Explorer.
     
  15. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    Your main problem likely was that you ignore basic network development rules.
    Generally all port in the 1 to 9999 range count as "reserved" as various services expect various ports to work or will bite you.

    I think if you use ports in the 10000+ range for your initialization, there shouldn't be a problem.

    I think the random port also is kind of troublesome for clients to use the right port. if they are just incremented, its at least a predictable pattern by which they can search
     
  16. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    @Dreamora;

    I'll try beginning @ 10,000 - why doesn't Unity's networking example do this?
    The Master Server stores which port a server is on, so connecting to games hasn't been a problem at all.
     
  17. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    I am still trying to isolate the buffer overflow, the error that is in the logs on the server point to a strlen (string length) return that when a string copy happens, data is going beyond the real length of the string or some data seems to be "locked" by something, as of yet undetermined. I have now went as far as buying a more powerful PC laptop to isolate the problem locally since time between cycles from my desktop to my VPS possibly is also suggesting that the stream of data might have enough packets dropped to confuse the engine (> 10%), I am wondering if the header of the packet that causes the crash is fully intact but the buffered stream is incomplete.

    This should not crash the master server though, so this is taking longer than expected to track down.
     
  18. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Thanks for the update Zumwalt; wish you the best of success!
    Why exactly are you experiencing such terrible packetloss?
     
  19. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    I am dumping large amounts of serialized information into the packets to test for "worse case" and trying to find the hole. I have a tard load of data going back and forth from the server to the client through the master and I have tons of debugging information. I am trying to isolate a repeatable scenario that I can fully document all of the conditions to make the problem reproducable to at least an 92% success rate.
     
  20. eblade

    eblade

    Joined:
    Oct 29, 2009
    Posts:
    7
    *chuckle*
    Ports 1 to 1024 are reserved on all (normal) operating systems, for known and default services. All ports above 1024 should be valid and available.
     
  21. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    State of the Networking:

    My vps hosted master server still crashes every couple days, but Zumwalt is has reported that he should soon have an autorestart system for it ready to go.

    I now have Unity powered dedicated servers beautifully integrated into my game - complete with fully automated connection initialization and load balancing. Unfortunately, although implementing dedicated servers has greatly increased the reliability of connecting to servers and and the average speed of packet transfers in my game, these advantages have been more than counterbalanced by a dramatic increase of the bug where NetworkViews refuse to serialize.

    I can state with certainty that the bug always occurs between a non authoritative server and certain random connected clients (the packets always reach the server from the authoritative client instance correctly). This bug only affects NetworkView serialization - not RPCs, which always get transmitted correctly. This bug is also asymmetrical: The player who joined the server earlier generally can't receive serialized views from players who joined later, but the latter players can almost always receive views reliably from the players that were in the server when they joined. If the earlier player exits and rejoins the server, the issue is reversed. And finally, the bug is highly selective: just because your client isn't receiving the NetworkView packets from a given player doesn't mean that other clients aren't. From my experience, half or so of the players in each game can receive updates from everyone (even players who joined after them), and the other half only receive updates from players that were in the server before they joined it.

    I have created a custom, super simple test project that demonstrates this bug - and sent it to Larus several weeks ago. I really don't know where to go from here, and am starting to get pretty discouraged.

    From my point of view, Unity is committing false advertising when they talk about how great their networking is. I have never seen a serious, physics enabled multiplayer fps type game that used Unity's networking and that worked reliably. I feel like I have been some kind of test subject in a dreadful experiment - where a company releases a flawed product, claims it works great, and then provides no support when someone invests years of their life developing something with it only to find that the fundamental systems they were counting on don't function as advertised.

    Unity: Please prove me wrong!

    -Aubrey
     
  22. eblade

    eblade

    Joined:
    Oct 29, 2009
    Posts:
    7
    out of curiosity, how does the master server have anything to do with the speed of packet transfers otherwise?
     
  23. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    I never said that it did - as you can see, I was speaking of dedicated servers.
     
  24. eblade

    eblade

    Joined:
    Oct 29, 2009
    Posts:
    7
    durrr. I'm retarded. :) sorry.
     
  25. oxl

    oxl

    Joined:
    Nov 21, 2008
    Posts:
    325
    I'm wondering why I never read something about networking improvements in the change logs of any version since 2.1 or some infos in the roadmaps about it.
    I'm pretty curious what will happen with Unity Networking in Unity 3.0+. Maybe they will silently drop it and replace it with something new ( like the Animation Editor in 2.5) ?

    --
    oxl
     
  26. Ethan

    Ethan

    Joined:
    Jan 2, 2008
    Posts:
    501
    Well after almost 2 years of using unity's painful networking stuff i decided to go the raknet plugin way. :x
     
  27. hogus

    hogus

    Joined:
    Jul 9, 2009
    Posts:
    145
    Has anybody here played with Torque up to the point where its networking can be tested to this extent?

    I've got the basic engine up and running, but you never can tell in 30 days what a product is really like, so it's difficult to compare.

    Their community gets just as uptight as this one, so it will probably take a lot of trolling around their forums to get a feel for how well the product is actually working and make a comparison.
     
  28. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    There is no 30 day limitation on the previous indie version anymore, its free, so you can test it as long as you want.


    And Unity Networking will likely not be dropped, because it is already RakNet, one of the strongest available networking layers in a price range most here can pay (the next higher I know is a few ten thousand dollar per title)

    Also the implementation of RakNet is actually fine, it does the job its meant to do and told to do.
    People just seem to ignore those points, try to do much more with it and then are puzzled why it just won't work.
     
  29. oxl

    oxl

    Joined:
    Nov 21, 2008
    Posts:
    325
    Glad to hear ! Though this argumentation makes no sense to me :
    What does the first has to do with that "it is already Raknet" ? Because of the cash Unity may have spent on this ?
    Well, show us one game, where it does the job its meant to do, maybe I was overlooking it. Please don't post mine :)

    --
    oxl
     
  30. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    As for the "what has this to do": well if a more powerfull networking is included, expect to pay more because each sold license of Unity has to pay higher license fees too. Thats what I meant :)



    Project: show me one project that actually attempts to use Unity 2.6, headless client and an optimized environment to do it.
    Don't expect magic, networking is an advanced to pro topic, not a beginner topic as which it is commonly missinterpreted.
    Beginner + networking = lots of frustration and problems (especially if they move away from Network.Instantiate)

    Common claims against the unity networking include bad stability, cpu hunger, losing master server connections and stuff like that and thats just not true.

    I've had 10 headless clients running a modified network example (to cut the stuff thats headless client unfriendly or rewrite it accordingly) for 25 days on my WinServer 2008 Standard (Core2Duo E6600, 4GB RAM) and had 1s of cpu time per day on the unused ones and about 2.5s of cpu time on the server to which I connected for 3 - 5 mins per day to see if still all is working fine.
    At no time, the servers got lost in the master server list (I've my own master server update function, I don't relay on automagic like the masterserver property for this purpose as it does not work).
     
  31. eblade

    eblade

    Joined:
    Oct 29, 2009
    Posts:
    7
    It looks like, to me, from reading the docs, that the people involved in writing the network stuff had no idea how networked gaming works. It also seems to me that the people involved in writing the docs have no idea how documentation works, either.
     
  32. hogus

    hogus

    Joined:
    Jul 9, 2009
    Posts:
    145
    @dreamora:

    I meant that I have the basic torque engine up and running, and was wondering if anyone can compare Unity networking to Torque networking.

    This is very confusing.... so are you saying that everything is working, so long as you don't rely on unity to do it for you?
     
  33. oxl

    oxl

    Joined:
    Nov 21, 2008
    Posts:
    325
    As you didnt't show us one, I would like to show you mine, but I'm on (ex) Indie, so it shall NOT run around headless. :p

    And yes, I know that networking stuff is an advanced feature. It's my daytime job since a decade and I worked with RakNet long before I discovered Unity.

    Anyway, I'm stopping here, this has been discussed to dealth, and its always ending with " Unity networking just works, it must be you, noob.".

    Maybe in my case, its true. Though I doubt that description fits to Robur, Ethan, Zumwalt or the others pulling their hairs of.


    --
    oxl
     
  34. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    @Dreamora:

    I am not requesting "more powerful networking".
    All I am requesting is that the networking work as advertised - or at the very least, that Unity acknowledge the bugs, and that they start providing all the users that the bugs are affecting with workaround suggestions and a roadmap for the bugs to be fixed.

    When you said "People just seem to ignore those points, try to do much more with it and then are puzzled why it just won't work.", were you speaking of me? Are you seriously positing the idea that a 10 player vehicular combat game is beyond the scope of Unity's networking abilities?

    Your most recent post was a bit confusing to me: First of all, you begin on the rather shaky foundation of questioning the intelligence of anyone who discovers a flaw in Unity's networking library. You also used several straw men to completely avoid discussing the issues that are currently the focal point of this thread: NetworkViews occasionally not serializing, and Zumwalt's Master Server crashing.

    It's nice that your headless clients which were connected to 3-5 minutes a day worked well - but do you really consider that proof that Unity's networking is rock solid, and that anyone who builds a real world implementation and runs into serious bugs is obviously just doing something wrong?

    I understand your "show me" paragraph to be a challenge for someone to demonstrate that they are using Unity's networking correctly, and are running into bugs - if so, I believe that Mars Explorer meets all your criteria except for being a headless client - which seems entirely unrelated to the stability of it's networking implementation.

    Thanks for sharing your thoughts.
    Happy coding!

    -Aubrey
     
  35. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    Network service monitoring tool is coming, slowly but coming, originally I wrote it in c++, going to redo it in C# to see if I can get around an issue I found, will get it to you hopefully this week robur.
    Zumwalt

    FYI for those monitoring this thread, the master server crash that Robur is referring to with regards to my version of the master server, is the same thing you would experience if you use the latest beta build v7 from the download link. The only thing I did was make the thing available to run as a service instead of a console application, I changed no core code on its functionality. The problem that is happening that Robur has stumbled across is related to a buffer overflow issue, nothing more.

    Which when this happens, C++ doesn't handle it correctly and shuts down the service (aka crashes the master), if he was still running as a console application (which he can switch to, by simply running the original master console code with the same ports) the crash will still happen. I have tested this, same blooming error, just happens randomly. All I am doing at the moment is creating a watch dog service for Robur that will simply monitor the service on a timer, every 15 seconds it will send a heart beat to the service, if it gets a response, good, if it doesn't get a response, it will simply do a net start command to get the mastser up and running again.

    I will probably also enable an email alert feature to send him an email when the server is down and that action is being taken along with the results of that action.

    The service also has verbose logging going on so he has complete logs to send to Unity of all data transactions up until the point of the crash. (just not the actual physical packet itself, that would be a nightmare). Robur goes days without a crash, but it crashes eventually and their is absolutely no pattern to it other than the same reason which simply points back like I stated, to a buffer overflow problem.
     
  36. Azimuth(kruncher)

    Azimuth(kruncher)

    Joined:
    Oct 19, 2008
    Posts:
    3
    To Everyone Here;
    Thank you all for your contributions to robur's plight! I have no technical knowledge but I have played Mars Explorer consistently for over a year and I'm on this forum to remind anyone that these games are played by people......frustrated people......people that want an immersive experience. Said immersive experience is stopped cold by game bugs. I understand that server crashes are important but I am less concerned with server crashes and much more concerned about bugs inside the game that don't allow all players to interact. I urge everyone to keep at this problem until solved for everyone's benefit. Who's at fault is not important. It's WHAT's at fault, am I right? Attached is only the tiniest sample of player non-connectivity. Based on discussion in the game, all players are moving about the map on their own screens and see all others laying still with the (no connection) sign.
    Thanks!
     

    Attached Files:

  37. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Latest Status:


    My master server crashes due to a buffer overflow every couple days, but Zumwalt pointed me to a windows service feature that autorestarts it as necessary - and it has been working great.

    My dedicated servers are doing well.

    My game's networking is terrible.


    There is a highly debilitating bug in Unity's networking library relating to the serialization of NetworkViews - It has already been discussed in detail here and elsewhere. I provided Larus with a custom Unity project which I believe demonstrates the issue weeks ago. I have hundreds of players - and dozens of highly vocal ones (Greetings Kruncher!) who can attest that this is a serious problem - and has been for years. Unity appears to be simply ignoring me.


    Aside from hearing from Unity, It appears that I have two options:

    1) Give up on Unity, and rebuild my game's networking with a library that actually works.
    2) Figure out a way to work around this issue.

    On the workaround front, the most obvious idea seems to be to utilize RPCs to transport all network data in my game - as they are unaffected by the state sync bug.

    Can anyone offer advice on the practicality of syncing the position, rotation, and velocity data of up to ten players 15 times per second via RPCs? I understand that RPCs are a "reliable" transport mechanism and that they have considerably more overhead than an unreliably serialized NetworkView - could this overhead be prohibitive, or should I be able to switch over to RPCs without any problems? I will be running some experiments over the next couple days to see if RPCs are at all viable.


    -Aubrey
     
  38. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    You aren't talking about the fact that your serialization / bitstream must have a constant size (which can easily be achieved by filling up a potentially smaller buffer with "dummy data")? Cause that wouldn't be a bug, thats a requirement of using delta compression, you can't do delta compression against a variable amount of data.

    What is missing though is a 4th network view mode: reliable (without delta compression) for exactly such circumstances where you just need the data to be received for sure.


    Depends on your latency requirement. Low latency requirements are something you will have to drop if you work with RPCs.


    Should Unity not be what you are looking for, then you have different options in different price classes. The list is starting from the least expensive to the most expensive solution

    1. Free crossplatform .NET: Lidgren (the ENet of .NET)
    2. Windows Standalone: DarkNet

    3. Large scale with lobby, rooms etc, tcp: SFS Pro, Neutron
    Large scale, high performance, low latency udp: Photon

    4. Ultra large scale and expandable to cluster, low latency, secure and optimized: NetDog
     
  39. Azimuth(kruncher)

    Azimuth(kruncher)

    Joined:
    Oct 19, 2008
    Posts:
    3
    Mars Explorer server list restarts are very fast lately, ten-fifteen seconds in my experience.

    Networking on Mars Explorer's dedicated servers has recently degraded to 100% of players displaying the dreaded (No Connection) badge 100% of the time.
    Those in the know have stopped using them.

    We game geeks wish we could help you Aubrey.

    No reply requested. Just keep on keepin' on.
     
  40. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    No Dreamora - I am not using delta compression, and don't need a "reliable" bitstream in which each packet is error checked. Each player's vehicle in my game has it's own NetworkView, which uses an unreliable bitstream to sync positional data according to this script.

    The problem is that at certain times (which seem completely random, and are becoming so frequent as to be the norm), everyone's NetworkViews sync correctly with the server - but never make it to other connected clients. Any RPCs sent by each client are correctly transferred to other clients, but not a single NetworkView serialization ever reaches non authoritative clients from the bug effected players.

    If you would like a firsthand demonstration of this issue - please check out this beta version of Mars Explorer, which includes dedicated servers:
    http://dat.marsxplr.com/211/play

    Join a game, play for a bit, and you should soon see new players who join and have "no connection" appended to their names in their onscreen badges and in the "Active Player's" list. Each player is instantiated with a "non connection" flag, but the flag is cleared the instant that the player's first NetworkView serialization is received. Note that the NC players can fire lasers, deploy and retract their wings, and chat just fine - RPCs are not affected by this issue (and are very helpful to demonstrate that I do indeed have the NetworkViewIDs properly configured).

    Low latency really is a requirement for me - this game includes elements of vehicular combat after all - so I suppose that that rules out using RPCs instead of NetworkView serializations. I will create a test RPC version of Mars Explorer anyway just to see how it works.

    If Unity's networking worked correctly, it would be exactly what I am looking for! It is simple, elegant, integrated nicely with the rest of Unity, and would be more than ample for a project of this scale. Since it doesn't work, if I don't hear from Unity, I guess a switch is in order. Photon and Lidgren both look interesting, I may end up taking one of those routes.

    Has anyone used Lidgren in a Unity FPS project yet? I couldn't find any Lidgren powered projects on the Unity forums, but a simple library library like Lidgren might be just what I need.
     
  41. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    RPCs are always guaranteed to be reliable, independent of the network view mode.

    The network view mode only impacts the component that it is observing.
    does the behavior for your game change if you set the network view to delta compression (that could rule out potential message routing / dropping issues).

    Also, are you using Network.Instantiate or an own network viewid alloc setup?

    And most important: I assume that all prefabs that contain network code are the same between client and server so you have both in the same project but with distinct monobehaviours or isServer / isClient switches? (I'm asking because otherwise the network ids wouldn't be the same and it would fail but I guess you would have seen and fixed this as it paints the editor console red basically. None the less I think its important to mention it.)


    Don't remember seeing any lidgren based games, but thats nothing that really surprises me, as the amount of non-unity networking games that are released or close to release can be counted with one or two hands, that includes projects with all 5 potential options.
     
  42. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    I altered Mars Explorer from utilizing Network.Instantiate to utilizing custom network installation routines a couple months ago due to Unity's extremely poor internal RPC buffer handling, and the NetworkView problem existed before and after the change. Misconfigured NetworkViewIDs are definitely not the issue with this serialization bug - I have never experienced any difficulty sending or receiving RPCs on the NetworkViews that refused to serialize data between the server and other connected clients.

    I hope to release a new Mars Explorer version today or tomorrow that will be able to switch between RPCs, unreliable BitStreams, and reliable BitStreams in game to directly compare the performance of each.
     
  43. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    RPCs actually can not fail without the whole network connection failing as they are reliable UDP.
    They are sent until they succeed or the application goes down.

    Will be interesting to hear about any potential change from the modifications.
     
  44. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    After experimenting with a test Mars Explorer build that can switch between reliable serialization, unreliable serialization, and RPC calls, I have two interesting new developments to report.

    First of all: on my dedicated server (which I hadn't changed any code on for a couple weeks prior to my testing today), my Mars Explorer dedicated servers reliability has been getting worse and worse - and it's now to the point where NetworkViews NEVER serialize under any circumstances in games that are hosted by a dedicated server. Today I uploaded a new version of my dedicated server code and restarted the server - and it didn't help the problem at all. If I host a dedicated server on my computer, it seems to work great. If I host the exact same code on my dedicated server, the NetworkViews of connected players DO NOT serialize. I have no idea why the problem has been getting so much worse of late - but I will be experimenting further.

    Secondly - and even more pathetic: The NetworkMessageInfo.Timestamp of RPCs sent from one client to another through any of my dedicated servers hosted on my dedicated server is always completely messed up. In all the RPCs my game client receives, Network.time - info.timestamp is equal to a number in the negative thousands. There is no explanation for this that I could imagine other than yet another fundamental bug in Unity, and I somehow feel that it is related to all the other bugs that have been plaguing my networking endeavors.

    I am building a system to compensate for the RPC time bug, and will then post a link to my test Mars Explorer build so that everyone can experience what I am discussing firsthand.
     
  45. hogus

    hogus

    Joined:
    Jul 9, 2009
    Posts:
    145
    I've played with lidgren and unity.

    It's relatively easy to set up. Encoding and Decoding messages is only slightly more effort.

    I even went so far as to create a basic lidgren master server that does NAT introductions. UDP punch-through appeared to work well on the test environments I had available to me.

    Although you might have a bit more initial coding, in the long run I think you will have more control with lidgren based networking.
     
  46. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    I would love anything you can share on Lidgren!
    Sample code would be very cool ;)
     
  47. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Allright - my latest Mars Explorer build is online, but it isn't perfect as I haven't yet been able to reliably determine the time each positional update RPC spends in transit from one client to another (so my interpolation logic is currently assuming that players have no latency when the game is in RPC networking mode). For those who are interested, here is Mars Explorer 2.13. The "Host Game >> " button will connect you to a dedicated server by default - you can host a game directly from your client by disabling the "Utilize dedicated Servers for hosting games" option in the Server Settings at the bottom of the lobby window. When in a game, the "Active Players" panel will show you a bunch of diagnostic info on each player in the game. When you are the game's Host, you can adjust the game's networking settings in the Server Settings panel. The settings should be self explanatory for networking experts, but I will be glad to answer any questions. Of particular interest is the complete inability of Unity to serialize the player's NetworkViews in UDP (unreliable) or RDC (reliable) networking modes when the game is connected to a dedicated server. Interestingly enough, the packet serialization works intermittently and the RPC timestamps aren't completely messed up when Mars Explorer game clients are connected to a dedicated server running my dev computer and not my dedicated server. I would be glad to share the source of my dedicated server implementation if anyone (especially Unity) would like to investigate it.

    I am baffled by all the bugs that I find at every turn, and by Unity's complete lack of support. Does Unity systematically ignore anyone who attempts to build anything using their networking? They have been such a great company in every other aspect that I keep expecting them to at least officially respond - but where are they?

    -Aubrey
     
  48. FreelandCJV

    FreelandCJV

    Joined:
    Nov 12, 2009
    Posts:
    221
    That is amazing! I love it! Keep going! Keep improving it! I absolutely can't wait until the next version comes out!!!
     
  49. zumwalt

    zumwalt

    Joined:
    Apr 18, 2007
    Posts:
    2,287
    wow dude i love the map in the corner and what you did with the water and bouyancy! outstanding!
     
  50. Aubrey-Falconer

    Aubrey-Falconer

    Joined:
    Feb 13, 2008
    Posts:
    438
    Thanks! :)

    2.13 is an alpha build that is packed with minor bugs and isn't anywhere near release quality yet, but I am on track for a final release on Christmas day that will include much more than you see now.

    The "Sea" is based on Unity's Island water, with a bunch of tricks that I may get a chance to include in the Sea object of the Whirld 3 release. It currently includes "tropic" and "lava" modes, is easily customizable to any desired look, and it is packed in the game and not individual worlds - so I will be able to keep improving it as technology marches on. The sea physics are really simple - I just assign a bunch of points on each vehicle, then check if each point is submerged and add some forces to it accordingly.

    My latest networking code sends a "ping" rpc between clients every 5 seconds or so, divides the round trip time by two (it doesn't know what the time is on the other client - but it can at least compare the time it receives the return RPC ping with the time it sent the request RPC ping), and then feeds the resulting time to my interpolation code as an estimated player lag. If anyone has a better idea for determining the actual lag of each positional update RPC, I would love to hear it!
     
Thread Status:
Not open for further replies.