Ready NetworkLobbyPlayers in NetworkLobbyManager

I've gone over the Lobby Manager tutorial and I essentially have a working lobby where the game will countdown and start after all players have clicked 'Ready,' this is great but I want to alter it so I can stop the countdown timer when players deactivate their ready buttons (it works like a toggle like in most games).

To issue a ready call I do this:

Code (CSharp):

1. public void ToggleReady()
2.         {
3.             if (readyToBegin)
4.             {
5.                 readyButtonText.color = Color.red;
6.                 SendNotReadyToBeginMessage();
7.             }
8.             else
9.             {
10.                 readyButtonText.color = Color.green;
11.                 SendReadyToBeginMessage();
12.             }
13.         }

The problem is this, once a player activates ready the countdown starts but if they attempt to then deactivate ready I get:

I'm really not sure what object its referring to being null. At no point do I destroy the player label or anything that might contain a reference back to this. The lobby manager is pretty well documented but its unclear how these other components interact with one another and unfortunately the sample LobbyManager tutorial doesn't have a 'stop countdown' feature.