Eval intermediate posttrained models through the trajectory

[hrdkbhatnagar]

Instead of evaluating a final_model instead have a current_best_model during the run. Have a separate watcher job to look for changes to this directory and evaluate each new model as it is copied over.

Very helpful for tracking progress through a run, instead of tracking it from agent's traces

[rank-and-file]

Also track the code changes of the agent just before each training run.

To do this, we write a small script which git commits everything but not large files (model files etc.).
Git commit message can be "new training run {timestamp}".