Through experimentation, I suspect the game needs to do a few things over the network.
First it uploads the gamesave to the client. During this time the game is unplayable for both host and client. How long this takes depends on your upload and your clients download. Usually 10-20 seconds for my game. Once finished, all items appear in the game and both the client and host should be smooth and able to move around.
Secondly, the game requires what seems like a baseline amount of network traffic to update your clients location and movement. This includes the actions and interactions of the client such as the Q key, F key.
The network volume required to acheive this i think is set by the Network Speed option in your game settings (or a specific editable file in the game folder, look into this if you want to experiment but it shouldnt be necessary)
I think the higher the speed option you select, the more frequently and thus the higher the bandwidth you need. The higher the setting, the smoother/more responsive the interactions are for your client (were talking whether its half a second or a quarter ofna second).... Until you saturate your actual host internet upload network speed, or client download speed.
The moment you saturate your internet connection, the game falls behind and cant catch up so you end up getting 10-120 second lag on the Q and F keys.
Upload speed would have been my first question but it sounds like you have a good connection speed. Do your friends also have a good downlod speed? And are there any other applications using up your internet bandwidth?
Secondly are your network settings in game selected to the maximum speed? I think its called "ultra"? Assuming your internet speed is fully available, it should be enough. I think we were using about 1-2Mbps for a single client connection, and i think it caps the maximum speed to a particular value based on the in game setting selected.
Thirdly, check your network activity in Task Manager. Are you uploading at a reasonable speed that is within your network bandwidth. I have not tried more than a single client connection, so I dont know what it should be, but I'm guessing the game should use maybe half of your 12Mbps connection on the maximum network speed option. If you see that your 12Mbps upload network connection is maxed out, then check to see if something else is using your network connection.
Latency of your network connection is also relevant but also shouldnt have more than a fraction of a second difference on the gameplay.
I play as a client to a host who has a 1.2Mbps upload and we usually sit just below the 1Mbps mark. When we set a custom network speed above the 1.2Mbps connection speed, we ended up saturating the network and getting the lag again.
We also note that when the host is browsing the internet or we are in a video phone call, it uses up the last bit of the hosts upload bandwith and the lag returns.
TL:DR - if you choose ultra in your game network settings, make sure you have enough internet upload bandwidth on the host available to support maybe 1Mbps per player?