Moving world based on GPS

I have a car with GPS in it, that sends latitude, longitude, and elevation data to my computer via UDP. I want to use this information to have objects persist at various real world locations, and have the car drive around them like an obstacle course. Only problem is that I want the user to be able to view all of this through a hololens, so I won't be able to change the camera's position, but rather the environment around it relative to what the camera's motion should be. I'm running into some problems, however, as the object I've placed is not always moving appropriately (It may move correctly when I move forward, but will go in the opposite direction it's supposed to when I go backwards. Before I post any code, I was curious how people would approach this problem. Any thoughts?