Question Parallel download (multi thread)

Hi there, hello!
I have about 6k asset bundles on my remote server and I need to download them on the user's device. The first necessary portion of them, about 400 needed to be downloaded as fast as possible so the user can start playing.
So I implemented a simple linear download queue by HTTP and it worked fine, but it was way slower than the available internet speed. Then I decided to try downloading in several parallel threads, turned out 12 threads were just perfect for my files and I managed to reach the maximum speed available. Now the problem that I have now is if a user has a poor internet connection, these threads are starting to "fight" each other, resulting in errors in most of them, preventing the user from getting the content.
Now a can try and adapt the threads count to each case dynamically, but I expect this is going to be a nightmare to implement and debug, and at this point, it starts to look like inventing a bicycle, so I'm now trying to find a working solution. Maybe there already exists some toolset to manage this problem, or maybe some way to stream data more efficiently than by HTTP? Thank you!

 

Maybe share code for the download system?
Do you use the job system? That could maybe help to also have better support depending on the CPU than a set amount of threads

 

There is not much code there, and it's not CPU threads. I just create 12 UnityWebRequest instances, and on complete callback of each, I take a new URL from the queue and create a new instance until the list is empty.

 

Post the code plz.

 

Maybe use the job system for thread generation across CPU cores and verify validity at the end of each job and if it's not valid restart that download job untill it works?
And again, post code. Even if it's small, it's helpful

 

Thanks for the replies! Sorry for repeating myself, but it has nothing to do with CPU threads, I probably should've used the word "thread" at all, since it's all in the main thread. Yes, there is a retry system in case of errors, and each file has 5 attempts to download. That is exactly the problem since with each retry I just burthen the channel with more requests and get more connection-related errors. Sorry, I'm unable to post the code at the moment since it's 5 different classes with a bunch of other features in them, and it will take quite a while to make them short and presentable. And then again, there is nothing interesting there, my question is not about code but about a concept, a problem, and a potential solution.

 

Reduce the amount of paralel downloads if you get errors maybe?

 

That is a good idea, thanks! I'll try that if I won't find any working libraries out there.