Loss values increase significantl after slowly decreasing. What is this issue that is causing this?

[ProGamerGov]

The training Log: https://gist.github.com/ProGamerGov/21f7bf21105bbea0010bab69a2761386

I am trying to finetune a model with two categories. 

Category 1 has 3177 training images and 353 validation images.

Category 2 has 2948 training images and 328 validation images. 

My "create_imagenet.sh" and my "make_imagenet_mean.sh" scripts can be found here: https://gist.github.com/ProGamerGov/16eddea12aee2b49e1fce9eb506a9648

My goal is trying to train the network to be better at artistic style transfer, and thus I am using images from Ostagram and Deepart.io. I would appreciate any insight into what I am doing wrong, or if the training is working properly.