Question Receiving a ndjson with UnityWebRequest

So, I'm parsing a ndjson stream, which means that it lives as long as my app is listening and whenever an evet occurs it appears here.

I'm actually doing it but I don't like it at all:

Code (csharp):

1.  
2.         private async Awaitable streamReceive()
3.         {
4.             UnityWebRequest www = UnityWebRequest.Get("https://somestream");
5.             var operation = www.SendWebRequest();
6.             while (!operation.isDone)
7.             {
8.                 await Awaitable.NextFrameAsync();
9.                 string json = www.downloadHandler.text;
10.                 var jsonLines = json.Split(new[] { '\r', '\n' });
11.                 foreach (var line in jsonLines) {
12.                  // bleh
13.                }
14.            }
15.  

The issue is that the download handler does not send me the new content received, but the whole content and it makes it quite difficult to maintain. Also the content of downloadhandler.text grows indefinitely which can be a problem in a tight resources environment.

Is there a way to get the download handler to send me the new content only and not the whole content? I'm not sure I can write my own download handler since it's written in C and I don't know what it does exactly.

Maybe I can override this?

Code (csharp):

1.  
2.         [UsedByNativeCode]
3.         protected virtual bool ReceiveData(byte[] data, int dataLength)
4.         {
5.             return true;
6.         }
7.  

Or is there an alternative to UnityWebRequest that would let me handle a ndjson stream more easily?

 

Sure you can. Have a look at this page. The last class mentioned "DownloadHandlerScript" is the base class for your own download handlers. It has 3 methods you should provide as the baseclass dosen't do anything on its own. Since you expect a probably endless stream the content length is probably not set in the response. So overriding ReceiveData is probably the only thing you will need. Note that HTTP is based on TCP. So data may not arrive in the exact snippets it may have been "send". TCP is an endless stream. So you have to buffer partial data yourself.

Of course, depending on the platform, you could handle the request manually with an ordinary TcpClient. Though it means you have to do the http stuff yourself. This is probably not worth it, especially when we talk about "httpS". So the custom download handler is probably the best solution.

 

Thanks, I'm looking into the standard csharp HttpClient right now. I've made it work for regular requests and I think it should handle ndjson better.

https://www.tpeczek.com/2020/10/consuming-json-objects-stream-ndjson.html