Very important question

If you leave the agent to train for a year and give him access to the Internet, will he take over the world?

 

Nope. You could never define a "take over the world reward function" correctly.

Only an evolutionary approach could write something so complicated and complex, but unfortunately then we have no real control over what it evolves into (maybe it will become the best Elvis Impersonator in Vegas instead, or any other random thing that it randomly decides is good).

 

I think if you give him rewards every time he does something good. In the end, he considers that people are superfluous on this planet.