Search Unity

  1. Full schedule for #UniteBerlin is now available! Featuring talks on our roadmap, hands-on labs and much more! Check it out!
    Dismiss Notice
  2. Unity 2018.1 has arrived! Read about it here
    Dismiss Notice
  3. Scriptable Render Pipeline improvements, Texture Mipmap Streaming, and more! Check out what we have in store for you in the 2018.2 Beta.
    Dismiss Notice
  4. ARCore is out of developer preview! Read about it here.
    Dismiss Notice
  5. Magic Leap’s Lumin SDK Technical Preview for Unity lets you get started creating content for Magic Leap One™. Find more information on our blog!
    Dismiss Notice
  6. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Unity Multiplayer UNET HLAPI Pro: taking UNET to the next Level!

Discussion in 'Multiplayer Networking' started by vis2k, Aug 11, 2016.

  1. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Download:
    Repository fixes only: https://bitbucket.org/vis2k/hlapi-pro-fixes-only/
    Repository fixes & improvements: https://bitbucket.org/vis2k/hlapi-pro/
    Love it? Become a HLAPI Pro Patreon!
    This is a preview for testing only. All rights reserved until I finish the license file. Use at your own risk. In case of concerns, feel free to inspect the .DLL files with ILSpy!

    Introduction

    Unity is the best game engine in the world, which should make UNET the best multiplayer game development solution in the world, but it's not.

    UNET consists of two parts:
    • The LLAPI is developed by @aabramychev and deserves more credit than it gets. It's nothing short of amazing. We all love your work Alex!
    • The HLAPI was developed by @seanr as an example to showcase the LLAPI. Sean Riley left Unity and everyone's hopes for HLAPI improvements remained mostly (not entirely) unanswered.

    I used UNET for uMMORPG, uMMORPG 2D, uMOBA and VOXL. I love UNET and I believe the architecture is amazing and want to use it for my own MMORPG some day. But the HLAPI code is in desperate need for some fixing and cleanup, as most here would agree.

    Why HLAPI Pro
    HLAPI Pro is a fork of the HLAPI. I will provide a free UnityNetworking.dll file that can be used as a drop-in replacement, no code changes needed. Use it to take your game's networking to the next level, or go back to the original HLAPI DLLs any time.

    The HLAPI Pro goal is Simplicity. The code needs to be short, understandable and stable. Micro optimizations can come later.

    Open Source & Free (mostly)
    HLAPI Pro is free to use by the average user! I can afford to spend a lot of time on this, as my other Assets already pay the bills. That being said, if you love this project then it's only fair to give something back. I haven't made my final decision yet and I am open for suggestions. Possibly:
    • Donations: but who does that anyway?
    • A commercial license: if you make money with it, you might as well give something back
    • A comfort feature: access to the Git Repository, priority support, etc.
    It will be open source as in: you can access the source code, but probably not redistribute it as your own HLAPI Ultimate etc.

    I want this to be used by as many people as possible, hopefully becoming the go to solution for Unity Networking. Hence why this is not a paid Asset on the Store.

    Features:
    • [2017-07-31] Increased the [SyncVar] limit from 32 to 64
      • Problem: one MonoBehaviour could only have 32 SyncVars+SyncLists combined. This is fine until you have a big Player script and no option to use more, leaving you to ugly workarounds.
      • Solution: increased the SyncVar dirty mask size from 4 to 8 byte, allowing 64 SyncVars+SyncLists. Sending around 4 more bytes is definitely worth our sanity.
    • [2017-08-01] Added a small 'welcome to HLAPI Pro' message to NetworkManager.Awake to easily check if HLAPI Pro is used
    • [2017-08-01] NetworkProximityChecker SetVis function replaced with GetComponentsInChildren
    • [2017-08-01] OnSerialize/OnDeserialize safely
      • Problem: the reader/writer is passed through all components. If one messes up, all other components will get strange errors like ReadString/ReadByte out of range
      • Solution: data length is included for each component, buffer copies are passed to each component so that it's impossible to touch any other component's serialization data.
    • [2017-08-01] OnSerialize/OnDeserialize detailed error logging
      • Problem: errors in OnSerialize/OnDeserialize are never shown. If we are lucky, we see ReadString/ReadByte out of range errors, but sometimes just corrupt data.
      • Solution: errors are caught properly before calling OnSerialize/OnDeserialize and log messages give detailed info about the object's name, component's type, sceneId, expected data size, and known causes.
      • Example: 2017-08-01_ondeserialize error log.png
    • [2017-08-05] NetworkTransform reimplemented from scratch
      • Problem: NetworkTransform was bloated with lots of components, all together 2144 lines of code. No interpolation. No NavMeshAgent consideration. High and Low compression modes did the same. No distinction between Kinematic and nonKinematic Rigidbodies. Too many computations.
      • Solution: reimplemented it from scratch. 300 lines of clean and simple code. Transform/Rigidbody2D/Rigidbody3D/NavMeshAgent support (including Kinematic Rigidbodies). Interpolation. 4 compression modes, able to squeeze a Rotation into 2 bytes and still work. Teleport detection. Great Gizmo makes it easy to debug.
      • 2017-08-04_networktransform_b_lots.png
    • [2017-08-06] NetworkManagerHUD via GUILayout
      • Problem: NetworkManagerHUD tried to emulate GUILayout with manual yspace increasing which resulted in way too much code that was way too hard to look at and debug. It also had no background, so white Labels often weren't readable if the 3D world behind it was not dark.
      • Solution: replaced GUI with GUILayout. Saves 50 lines of code. Includes background box for easier visibility.
      • 2017-08-06_networkmanagerhud_guilayout.png
    • [2017-08-06] NetworkDiscovery GUI via GUILayout
    • [2017-08-06] NetworkManager OnValidate warning channel checks
      • Problem: in the UNET code Channels.DefaultReliable is 0 and DefaultUnreliable is 1. So the NetworkManager channels should be of those types, otherwise some functions send data over a channel that wasn't meant for it.
      • Solution: OnValidate checks if first channel is Reliable and second one is Unreliable, shows a Warning if necessary.
    • [2017-08-10] Nondeterministic SceneID bug fixed (most critical HLAPI bug, exists in all HLAPI Projects)
      • Problem: OnPostProcessScene assigned SceneIDs based on the order of FindObjectsOfType<NetworkIdentity>(). The order is not guaranteed, which resulted in different SceneIDs each time the Editor restarts. Reconnecting to a build after restarting the Editor would then cause bugs where UNET tried to serialize the wrong objects, often causing 'ReadString out of range' errors etc.
      • Solution: FindObjectsOfType is sorted by the sibling index path in the Hierarchy. This doesn't change after restarting, only when manually changing the Hierarchy.
    • [2017-08-10] Fixed "ServerDisconnected due to error: Timeout" bug.
      • Problem: NetworkManager.OnServerDisconnect throws an error each time a client times out. A timeout isn't really worth throwing an error, since it happens all the time.
      • Solution: NetworkManager.OnServerDisconnect only throws errors if the cause is not 'Ok' and not 'Timeout'. Also improved the log message because it sounded like the server disconnected itself, even though a client disconnected.
      • upload_2017-8-10_12-56-6.png
    • [2017-08-11] Statically cached Messages removed.
      • Problem: the HLAPI is complicated enough as it is, caching messages statically makes the code so complicated that it's hardly worth an extra allocation. Many of the cached messages are only sent once per client anyway.
      • Solution: messages like 'new CRCMessage' are created when they are needed by the message handler. Less global state, less complexity, cleaner code, easier to debug and to understand.
    • [2017-08-12] Simplified public fields
      • Problem: private variables were wrapped with public get+set properties in many cases, when using a public variable would give the exact same result with less code.
      • Solution: replaced some public get/set properties with public variables. Also replaced some properties with get + private set to keep things simple.
      • Note: there are still a few more to simplify in NetworkManager, NetworkLobbyManager etc., but those are serialized and would break HLAPI compatibility so let's wait a bit longer until HLAPI Pro becomes more popular.
    • [2017-08-14] NetBuffer replaced with C#'s built-in MemoryStream
      • Problem: UNET's NetworkReader and NetworkWriter use a custom NetBuffer class that emulates C#'s built-in MemoryStream to save allocations because the internal buffer can be passed along directly. This is a bit of a philosophical question, I'd argue that being able to access the internal buffer from the outside is a bad idea because any bug will be incredibly difficult to track down. Using the built-in MemoryStream means less code, less complexity and the odds of ever encountering a bug in MemoryStream are virtually zero. Another argument is that computers get faster every year,so a few extra allocations won't matter too much a few years down the road, but any bug would still be terrible.
      • Solution: replaced NetBuffer with MemoryStream. Also changed .Position to 'ushort' to support up to 64KB. Previously it used short.maxValue for packet size checks, which only allows up to 32KB.
      • Note: we can always go back to NetBuffer later when UNET is 100% stable. But for now this seems to be the smart thing to do.
    • [2017-08-14] LogFilter redundant log level definitions removed.
      • Problem: log levels were all defined twice for no reason, once as enum and once as ints.
      • Solution: removed redundant log level definitions
    • [2017-08-20] ConnectionArray: connections list range checks fixed to use < instead of <= to avoid possible bugs where connId might be out of range
    • [2017-08-20] LocalClient: removed unnecessarily complex free message buffering
    • [2017-08-20] Messages: optimized some messages to minimize bandwidth
    • [2017-08-20] NetworkClient: IsValidIpV6 fixed. Invalid IP6 addresses were accepted before but aren't anymore now.
    • [2017-08-20] NetworkDiscovery: string to bytes and bytes to string via proper UTF8 encoding
    • [2017-08-20] NetworkServer: UpdateServerObjects removes null objects immediately instead of every 100 frames.
    • [2017-08-20] NetworkManager: OnClientSceneChanged rewritten from scratch to remove some really weird code.
    • [2017-08-24] 'Did not find target for sync message' Warning text improved to note that this can be completely normal in some cases, e.g. when an arrow is destroyed, but the arrow's movement message comes in a bit later due to how UDP works.
    • [2017-08-25] Fixed Client Scene Switching/Loading
      • Problem: switching a scene would sometimes cause random object spawning errors and 'target not found' messages due to corrupted state.
      • Solution: I analyzed the Client<->Server handshake to find out at which point the client state got messed up. If the server sends a scene change message to the client, then there is no guarantee that this message arrives before the rest of the initialization (like object spawning). If the message arrives during initialization, then the scene will be loaded asynchronously and after the scene was loaded, all the state data like spawned objects are nulled. HLAPI Pro uses a new pausing mechanism that delays all message handling while a scene is loaded, making sure that all the spawning works perfectly fine even if spawn messages are received during a scene load.
      • upload_2017-8-25_14-50-44.png
    • [2017-08-31] Removed unnecessary ChannelPacket class
      • Problem: ChannelPacket was only used by ChannelBuffer and it's pointless to reinvent the wheel
      • Solution: ChannelBuffer uses C#'s built-in MemoryStream now. Saves a lot of code and reduces probability of bugs.
    • [2017-11-16] NetworkConnection.Disconnect doesn't clear .address anymore so it's available in NetworkManager.OnServerDisconnect

    Roadmap:
    • [Continuous]: always apply latest HLAPI fixes from Unity
    • Simplify the code to reduce complexity and number of bugs
    • [SyncVar(bool onlyForOwner)] option
    • Problem: SyncVar always syncs to all observers, not just the owner. It needs an option to only sync to the owner. This will save huge amounts of bandwidth for situations like player inventories that don't have to be seen by anyone around them.
    • Develop an Editor Addon to download and setup the DLLs automatically
    • Improve NetworkTransform interpolation
    • Add other useful components like Name Synchronization, NavMeshAgent Synchronization, NetworkTime, etc.
    • Setup a stress test project and host it 24/7, allow the community to connect and try to crash it.
    • Host a hack-the-server event, see if there are security flaws
    • Possibly introduce Zones/Instances similar to UNET Phase 3
    Usage guide:
    Simply backup the original DLL files from your Unity installation folder and replace them with the HLAPI Pro files:
    • Windows:
      • Backup your Project! Be wise now and don't lose everything if something unexpected happens.
      • replace C:\Program Files\Unity\Editor\Data\UnityExtensions\Unity\Networking content with the downloaded files, but:
      • move the Unity.UNetWeaver.dll to C:\Program Files\Unity\Editor\Data\Managed
      • Restart Unity for the UNetWeaver.dll to be reloaded properly. Rebuild all your clients/servers so they use the same DLLs.
      • Rebuild your server/client.exe so that it uses the DLL as too.
    • Mac:
      • Backup your Project! Be wise now and don't lose everything if something unexpected happens.
      • replace Unity.app/Contents/UnityExtensions/Unity/Networking/ content with the downloaded files, but:
      • move the Unity.UNetWeaver.dll to Unity.app/Contents/Managed/
      • Restart Unity for the UNetWeaver.dll to be reloaded properly. Rebuild all your clients/servers so they use the same DLLs.
      • Rebuild your server/client.app so that it uses the DLL as too.
      • Note: right click Unity.app and select 'Show Package Contents' to see the subfolders.
    • Linux:
      • You'll find it
    Testing:
    I test every single code change with my uMMORPG project, and many other uMMORPG developers use it for their real world hosting scenarios. I will always fix all known HLAPI Pro bugs before releasing a new version.

    Support:

    My time is limited and best spent on the code. Use the UNET documentation to learn how it works. Report bugs to Unity if they happen in the original HLAPI too. Reply to this thread for questions, suggestions, bugs related to HLAPI Pro, etc.

    Unity / UNET Team:

    I have huge respect for your work and I am UNET's biggest fan. Almost all of my suggestions during the last two years went unanswered, the majority of my bug reports were ignored or labeled as 'by design' or as my own fault. It's disappointing that I was treated so poorly.
    • If you want to include HLAPI Pro bugfixes in the official HLAPI code, please contact me first.
    • If HLAPI Pro becomes a huge success, I'd be more than happy to work towards making this the official HLAPI code.
    Patrons:
    Special thanks to everyone who supports HLAPI Pro development on Patreon.
    @doctorpangloss @yellinglionstudio
     
    Last edited: May 19, 2018
  2. iileychen

    iileychen

    Joined:
    Oct 13, 2015
    Posts:
    57
    If it contains a better designed and stable and documented LLAPI to use, i would like to pay.
    Also, Websocket support is needed.
     
    nonom likes this.
  3. voltage

    voltage

    Joined:
    Nov 11, 2011
    Posts:
    488
    I would buy a NetworkTransform fix alone. 5 bucks USD sounds fair, maybe 10 if it's smooth as butter. I want it to be virtually identical to the current one, only it works properly. My only request really.
     
  4. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    I don't intend to modify the LLAPI, only the HLAPI. The LLAPI is closed source anyway.

    Thanks for your feedback.
     
  5. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    840
    I'd definitely be interested in paying for that. Especially if I can see the stress test results, which would reassure me that this would be a good and dependable option for my projects

    PS: if you do manage to successfully communicate with the rare unicorns working on UNET, can you please try to drag them back here in the forums and make them read/comment on this thread?: http://forum.unity3d.com/threads/official-multiplayer-improvements.390823/
     
  6. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Good to hear.

    That reminds me, I also have a lot of SyncListStruct examples (including hooks) that would probably be interesting for a lot of people too.

    I am sure they do read the forums every now and then.
     
    Chom1czek likes this.
  7. Quis

    Quis

    Joined:
    May 16, 2014
    Posts:
    14
    Paying for it wouldnt be an issue, although if it was free but based on for example donations/patreon/etc then more people could adapt it to their projects.
     
  8. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Patreon is a good idea too. Will add this option to the first post.
     
  9. Xaon

    Xaon

    Joined:
    Feb 28, 2013
    Posts:
    52
    I'm most interest in premium support or complex examples as those provided by Unity are oversimplied for me and don't cover many typical use-cases. While I am a programmer, programming network stuff is still a bit of magic for me and I would gladly pay to learn some more of this magic ;)
    More features are secondary for me, at least for now.

    P.S.
    NetworkTransform with Rigidbody synchronization works good enough for me (as it does movement interpolation) so I don't quite understand why is this issue mentioned so often.
     
  10. moco2k

    moco2k

    Joined:
    Apr 29, 2015
    Posts:
    275
    There is no question about the current UNET flaws and the urgent need for further improvements. So any kind of initiative is more than welcome.

    I have one concern though. If you go that route with custom HLAPI, in the end, we have both native UNET HLAPI and HLAPI_fixed. That means we have 2 different HLAPIs which are developed independently by different people. This has some potential for confusion and I guess also for problems. For example, what happens if official UNET implements significant updates to the original HLAPI but I am now using HLAPI_fixed and it is not updated as well? Also, support could become an issue.

    Maybe this is a good idea still. But before going this way, I would strongly recommend to first get in direct contact with the UNET devs and try to establish some kind of official collaboration. So that you can directly work together with the UNET team to get your improvements in the official branch. Maybe even something like freelance UNET improvement task force. Then, we would have both a faster improvement progress and official support and maintenance. And above all, a single stable and feature-rich HLAPI that everyone can use, with a single documentation and community. If this way does not work out, okay, then you could still take the custom route.
     
    Last edited: Aug 17, 2016
    Artaani likes this.
  11. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    NetworkTransform is laggy for most people when trying it over the network.

    Valid concerns.

    If enough people end up being interested in this project, then I will contact Unity first of course. For legal reasons, etc.

    I definitely don't want to reinvent the wheel. Phase 1-3 are very important and can be developed without messing with the HLAPI core (NetworkManager etc.) at all. Component based development allows us to replace NetworkTransform, NetworkProximityChecker, etc. easily. Allowing us to replace and create our own components seems to be one goal of UNET anyway.

    Phase 4 would be more difficult, if we decide to replace more critical components like NetworkManager. Don't get me wrong, I really don't want to do that. This would be a nightmare to deal with. Keeping it up to date with the latest official HLAPI code would be hard. Just like anyone else here, I am really only interested in launching my own online game. If UNET would have the same quality like other Unity components, then we wouldn't have this discussion in the first place. Sooner or later, many of us will launch their games though. If critical bugs in core components are found, then they will have to be fixed. Let's hope we don't get to that point though.
     
    Last edited: Aug 13, 2016
  12. jjobby

    jjobby

    Joined:
    Nov 28, 2009
    Posts:
    89
    We are currently developing MMORPG for our client and the lack of update in Unity network is really becoming our main concern. It would be nice if you can get the collaboration from UNET developers. I fully support your plan and I am willing to pay for your effort.
     
  13. larus

    larus

    Unity Technologies

    Joined:
    Oct 12, 2007
    Posts:
    249
    Just to be clear on legal matters: Anyone can freely modify, use, re-distribute (for free or for a fee) or do anything they want with the high level API source we published on bitbucket (as explained in the LICENSE file in the repo). Nothing to worry about there.

    You can work on the project, and do fixes/modifications/improvements, and it's up to you if you want to make them public and available for us to integrate if that is suitable or to keep them private (or publish on asset store with a price tag). It should be easy to switch out components (like NetworkTransform) and exchange with modified or customised versions. Of course if the core is changed too much it will be harder to integrate patches from us but that's up to you.

    We'll have an update soon on our immediate roadmap so you can see what changes are incoming in future releases.
     
  14. iCode

    iCode

    Joined:
    May 25, 2013
    Posts:
    28
    To answer your question I would be willing to pay.

    Do to personal issues i have been away from the Unity scene. I recently started up again with the idea of using UNET. I had no idea it was in such a bad state with no signs of improvement. Although i think it is great you are willing to take on this challenge. It is unfortunate that Unity has let it come to this.
     
  15. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Thanks for the info.

    By the way @larus, it looks like you are trying to create your own GetComponentsInChildren function in NetworkProximityChecker:
    Code (CSharp):
    1.         // called hiding and showing objects on the host
    2.         public override void OnSetLocalVisibility(bool vis)
    3.         {
    4.             SetVis(gameObject, vis);
    5.         }
    6.  
    7.         static void SetVis(GameObject go, bool vis)
    8.         {
    9.             foreach (var r in go.GetComponents<Renderer>())
    10.             {
    11.                 r.enabled = vis;
    12.             }
    13.             for (int i = 0; i < go.transform.childCount; i++)
    14.             {
    15.                 var t = go.transform.GetChild(i);
    16.                 SetVis(t.gameObject, vis);
    17.             }
    18.         }
    Even though it already exists and can be used like this:
    Code (CSharp):
    1.         // called hiding and showing objects on the host
    2.         public override void OnSetLocalVisibility(bool vis)
    3.         {
    4.             foreach (var r in GetComponentsInChildren<Renderer>())
    5.                 r.enabled = vis;
    6.         }
     
    Last edited: Sep 27, 2016
  16. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Update: HLAPI Pro V1.0 released for Unity 2017.1.

    After reading @aabramychev 's post about how no one created a better HLAPI , I figured I should finally do it.
    Please read the top post for all the details. V1.0 already has the fix for the ReadString out of Range bug that happens randomly in every single UNET project.

    Please reply if you tested it, if you like it, if you have questions or suggestions. Please tell me what you want to see in the future. Source code access and license info will come soon.

    Let's fix UNET!
     
    Last edited: Aug 1, 2017
  17. moco2k

    moco2k

    Joined:
    Apr 29, 2015
    Posts:
    275
    Thanks for your efforts! I'm looking forward to test it.

    One suggestion: I would appreciate to have more transparency and information on bandwidth consumptions when using particular HLAPI methods. I know that this is more a matter of documentation, but maybe you decide to add some additional documentation as well (ofc only where it is needed).
     
    Last edited: Aug 1, 2017
    vis2k likes this.
  18. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Which methods in particular? Do you mean 'how many bytes a Rpc uses' or do you mean 'how many bytes a SyncVar uses per second' etc.?
     
  19. moco2k

    moco2k

    Joined:
    Apr 29, 2015
    Posts:
    275
    I think both. It would be helpful to have clear information right from the start, e.g. about header bandwidth consumptions or for magic stuff that is going on under the hood. Especially for the relevant methods such as SyncVars, Commands, RPCs, SyncLists, etc. I have once started a thread about this here. I think such information should be part of the official UNET docs but it's not there yet. So, it might be a beneficial option for additional docs.
     
    Last edited: Aug 1, 2017
  20. Possum1

    Possum1

    Joined:
    May 19, 2017
    Posts:
    3
    Great work. A fix for the readstring bug is huge on its own. Thanks for your work.
     
    vis2k likes this.
  21. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    I see. I will look into the code and see if I can find anything.
     
  22. nonom

    nonom

    Joined:
    May 1, 2014
    Posts:
    12
    Good luck and have fun ;)
     
  23. Vancete

    Vancete

    Joined:
    May 3, 2010
    Posts:
    161
    Thanks for your work! (just posting here for checking it later at home)
     
    vis2k likes this.
  24. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Download will be available again soon, looking into a potential issue first

    Wasn't much work yet, but thanks.

    Please keep me all posted on wether or not it works in your project and remember to rebuild all clients/servers first.
     
    Last edited: Aug 1, 2017
  25. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    961
    I know that post is old. But no he isn't reinventing GetComponentsInChildren. GetComponentsInChildren goes down the whole hierachry. He just checks the direct children of this object.
     
  26. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    And then calls SetVis again on each child. Recursion is used there.
     
    ferretnt likes this.
  27. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    961
    Oh, did not spot that! My apologies.
     
  28. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    961
    Are you planning to add support for the Unet LLAPI timestamp feature? Don't think that's in the HLAPI
     
  29. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Update: new version uploaded, download link is in the first post.
    • ProximityChecker GetComponentsInChildren replacement
    • NetworkManager.Awake HLAPI-Pro welcome message for easy check
    • Temporarily removed my ReadString out of range fix while I am researching another occurence at one of my customer's projects
    Maybe later. I am more interested in bugfixes and code improvements first.
     
    Last edited: Aug 1, 2017
    TwoTen likes this.
  30. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    273
    Link to the source code repository please. :rolleyes:
     
  31. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    373
    Hey, could you implement some authorization system on HLAPI, with a some byte[] array for example client.Connect(ip, port, data); an the server won't connect the player to the server if the credentials are wrong, thanks.
     
  32. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Update: next version was uploaded.
    • [2017-08-01] OnSerialize/OnDeserialize safely
      • Problem: the reader/writer is passed through all components. If one messes up, all other components will get strange errors like ReadString/ReadByte out of range
      • Solution: data length is included for each component, buffer copies are passed to each component so that it's impossible to touch any other component's serialization data.
    • [2017-08-01] OnSerialize/OnDeserialize detailed error logging
      • Problem: errors in OnSerialize/OnDeserialize are never shown. If we are lucky, we see ReadString/ReadByte out of range errors, but sometimes just corrupt data.
      • Solution: errors are caught properly before calling OnSerialize/OnDeserialize and log messages give detailed info about the object's name, component's type, sceneId, expected data size, and known causes.
    Here is a screenshot of the detailed error logging:
    2017-08-01_ondeserialize error log.png
    There will be no more randomness in OnSerialize/OnDeserialize errors. They will be easy to debug.


    Coming soon

    You should do that in your custom NetworkManager, just send the account info along when connecting. I do that in uMMORPG too.
     
    Last edited: Aug 1, 2017
    nxrighthere likes this.
  33. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Progress: repository link added to the first post.
     
  34. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    189
    This is a great idea! If we could fix up the NetworkTransform so people don't have to write their own and get rid of massive per frame memory allocs in the NetworkAnimator then I'd be a happy camper.
     
  35. nhold

    nhold

    Joined:
    May 2, 2017
    Posts:
    29
    Maybe it's a good idea to work with Unity and get these accepted as PRs (Minus the SyncVar limit increase, that can be kept out).

    Unfortunately it seems Unity hasn't merged any PRs for the HLAPI, they should really think about it.

    I also don't think you should change NetworkTransform, just have a NetworkTranformInterpolated.
     
    Last edited: Aug 2, 2017
  36. VergilUa

    VergilUa

    Joined:
    Dec 22, 2014
    Posts:
    295
    Wow, this is HUGE. How did I missed this earlier?

    I hope this would became a part of actual HLAPI library. Not that I wouldn't pay for it, just to keep my sanity.

    If you'd fix that sceneId bug, that would be really, really goood.
     
    Last edited: Aug 2, 2017
  37. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Good morning everyone. Glad to see so much positive feedback here. Next wave of improvements coming soon.

    That would be nice yes.

    Will see about that yes

    SceneId bug is my highest priority. I have a fix that works, except when you build the game with a scene file that you never opened before. Need to figure out how to set and save a sceneid in a scene that isn't open first.
     
    VergilUa likes this.
  38. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Update: added a new Experimental version to the first post:
    • [2017-08-02] assetId uses a prefab's m_localIdentifierInFile instead of a complicated NetworkHash128 workaround
      • Problem: Networkhash128 was used to uniquely identify a prefab based on it's path. This is unnecessarily complicated and uses 128 bytes of data and wasn't resistant to renaming.
      • Solution: Unity's built in m_localIdentifierInFile is a persistent unique id for prefabs, so we might as well use it. It fits into a long value, so we save 64 bytes each time we send it over the network.
    This bothered me for a long time.

    Now that the asset id is a simple struct wrapped around an 'ulong', NetworkAssetId, NetworkSceneId and NetworkInstanceId are just simple wrappers around built in types. Might be possible to make them even simpler too.
     
    Last edited: Aug 2, 2017
    nhold, VergilUa and camta005 like this.
  39. TheWesselGames

    TheWesselGames

    Joined:
    Aug 31, 2016
    Posts:
    29
    Seems really promising. Looking forward to the updates!
     
    vis2k likes this.
  40. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Thanks. Playing around with NetworkTransform right now..
     
  41. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Progress: turns out that NetworkTransform is a giant mess, 2144 lines of code for all the relevant components. It even has package handling hooks in the NetworkServer code.

    That's way beyond fixing, so I decided to rewrite NetworkTransform from scratch.

    Position interpolation is already ultra smooth, here is a uMMORPG test: https://gyazo.com/0519736c3794fa64f5c3dbf2f0287ae1
     
    Last edited: Aug 2, 2017
    mons00n and PartyBoat like this.
  42. PartyBoat

    PartyBoat

    Joined:
    Oct 21, 2012
    Posts:
    47
    That link is one of the most pathetic things I have ever read. I'm sure aabramychev and the rest of the networking team are just doing their best, but Unity Technologies has a valuation of over one BILLION dollars and has over 1500 employees. The fact that they thought it was okay to leave something as important as networking to be written by the community almost makes me want to give up on Unity altogether. It blows my mind how many games released today are multiplayer and yet a company as massive as this treats their networking solution like a second class citizen. Now we rely on a single unpaid member of the community to fix something that a company with plenty of cash and staff to throw around should be working on.

    Anyway, thanks for your effort vis2k! It's very generous of you to do Unity's work for them and release it for free. I will always be a big supporter of anyone who decides to create an open source project to help the community. Best of luck!
     
    Last edited: Aug 3, 2017
    nickonmir, PeterB, Thanathos and 2 others like this.
  43. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    Thanks. I guess it's like he said, HLAPI was just an example on how to use LLAPI. The LLAPI is UNET. And that works impressively well and is far from easy to do.

    HLAPI is just one way to do a high level architecture, there's value in not being tied to any. I think this architecture with SyncVars/Commands/Rpcs is really good though, it just needs some cleaning up.
     
    nickonmir, angusmf and TwoTen like this.
  44. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    961
    Alex is the LLAPI guy. And how do you know they have 1500+ employees? I am not saying you are wrong. Just wondering about your sources? Only one I can find is Wikipedia. But there is no source on that number. So it could be made up.
     
  45. PartyBoat

    PartyBoat

    Joined:
    Oct 21, 2012
    Posts:
    47
    You're right about Alex, I'll edit my post.

    Actually the valuation is even higher than I had seen previously. This is the most recent interview I could find (May 2017). About halfway through the Q and A John Riccitiello says:

    "We’re hiring 10 or 15 people a week, so yeah, that number changes. But every company like ours has some attrition. We’re at a low attrition number. We’ll probably finish the year with 1,500 to 1,600. We’re not hiring like we were two years ago, building up the company to do all the stuff we’re able to do now. But we’re still growing. Revenue is growing even faster."​

    The article is actually pretty interesting if you want a sense of Unity as a company and where they are heading.

    Fair enough, I mostly felt like ranting a little bit. Although I maintain there is no reason why Unity couldn't create a fully-featured, non-example, robust high level architecture for the most popular use cases, or at least they could improve the old one. It would be a huge feature win for the engine.

    I don't want to derail your thread though so I'll leave it at that. :p
    Can't wait to see your work progress!
     
    Last edited: Aug 2, 2017
    VergilUa and TwoTen like this.
  46. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    189
    How frequently are you sending position updates and via what QoS?
     
  47. robochase

    robochase

    Joined:
    Mar 1, 2014
    Posts:
    210
    if i could chime in about the HLAPI and a possible direction of this library - what I would really like out of the HLAPI is more dependency-injection-esque patterns. for example, the HLAPI already has ways for you to supply your own NetworkConnection class to use instead of the built in one. more stuff like this would be very helpful.

    the thing about the HLAPI that really sucks is that if you don't like one part of it, you have to change the source code, generate DLLs, and shove them into your unity application folder. this is tedious to maintain, especially if you have multiple HLAPI unity projects on your computer. ideally, it should be a cinch to inject changes into the HLAPI at the project level and not the system level so that all your different projects can play nicely with each other. it would also make upgrading unity versions a lot less painful
     
    TwoTen and camta005 like this.
  48. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    It uses the DefaultUnreliable channel like the original one. There's another issue with default channels that I want to work on too. So right now the code always assumes that DefaultUnreliable is channel 0 in your NetworkManager. But if you reorder them it might wind up using Reliable, StateUpdate or whatever other channel there is in the first slot. Unless I am missing something. Will have to do more research there soon.

    Frequency can be selected by the user via 'Send Interval' in seconds. So a smaller send interval means more updates. The video uses 0.1s. I just tested 0.5s and that works just as well.
    upload_2017-8-3_10-28-40.png

    The original NetworkTransform sets the position in OnDeserialize immeditately without any interpolation, so any interval should look smoother than before.

    Would be best if we can put the Scripts in the editor directly. Right now this is not possible because parts of NetworkTransport are internal. I mentioned this to Alex.
     
    VergilUa likes this.
  49. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    961
    I suggest using the Unreliable Ordered. Otherwise this might happend:
    We get an update. And move them there.
    We then get an older update. So we move them back!?
    We then get an even newer update. So we move them really far ahead
     
    vis2k and nxrighthere like this.
  50. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    2,275
    True. My current interpolation method keeps a history with timestamp anyway, so packets arriving out of order won't be a problem.