Unity Multiplayer Timeout connection problem with Mirror & Steamworks

Hello everybody, I've just found out about this forum after couple of days going back and forth with a connection problem. When I started creating my game, everything was working good, 4 people could connect, see eachother's animations etc. After building the game and having a couple of mirror objects around like doors, oncollision enter, etc, not even 2 players can join anymore. The problem is that, they can join the lobby but after the host is pressing play, he joins instantly the game but others are waiting in the lobby for a few seconds than they connect but can't see anybody, also the console of the host says they just disconnected with no error in it. I've also tried hosting the server on a different PC and join it via the unity client and it somehow worked when the host on the other pc pressed play, but if I'm trying to join with 2 PC both from Steam, they are not connecting... Also figured out it might be a timeout error since the other players take too long to join but can't find how to make the timeout longer so they atleast wait for all the server side items to load. Anybody has this issue? I've been spinning my head for the past couple of days and been working to the game for the last months, it's so sad that it's not working anymore.

 

Try the lobby example in the Mirror folder for comparison