Calling Solver.solve does not display intermediate steps at the given display step
cnnfor...@gmail.com
Issue summary
When running Solver.solve() in Spyder the console does not display anything, no test accuracy, no model definition, nothing. Actually nothing is ever displayed, only when there is an error, I receive the output log, with everything I have told Caffe to do up until then. This also blocks my console, and I have to restart the kernel. I've tried to lookup this issue, but can't find any post about it.
I know that everything else is working as it should, solver.net.forward() shows the loss is getting smaller after calling solver.step(n). Further, a solver snapshot is also made and saved to the specified directory when calling solver.solve(). However, I am unable to inspect my training loss or validation accuracy like this. Of course I could do it all manually, but I would prefer that the display works.
I'm applying a model for transfer learning, and am now checking if everything works using the bvlc_reference_caffenet.
code to call solver:
#load all packages
import os
os.environ["GLOG_minloglevel"] = "1" #removing this line doesn't make a difference
import graphviz
import caffe
import numpy as np
import cv2
import lmdb
import matplotlib.pyplot as plt
from PIL import Image
import skimage.io
from scipy.ndimage import zoom
from skimage.transform import resize
import subprocess
import platform
import copy
import random
#caffe.set_device(0)
caffe.set_mode_cpu()
caffedir = 'C:/Users/*******/Documents/Git/caffe'
os.chdir(caffedir)
###################################################################################
#get solver
solver = caffe.get_solver('models/myfirsttransfer/solver.prototxt')
testnet = solver.test_nets[0]
#get pretrained model
caffenet = caffe.Net('models/bvlc_reference_caffenet/deploy.prototxt',
'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel',
caffe.TEST)
#generate dictionaries of the parameters
params = caffenet.params.keys()
source_params = {pr: (caffenet.params[pr][0].data, caffenet.params[pr][1].data) for pr in params}
target_params = {pr: (solver.net.params[pr][0].data, solver.net.params[pr][1].data) for pr in params}
#copy weights from caffenet to solver
for pr in params:
target_params[pr][1][...] = source_params [pr][1] #bias
target_params[pr][0][...] = source_params [pr][0] #weights
#solve
solver.solve()
Solver.prototxt:
net: "models/myfirsttransfer/train_val.prototxt"
test_iter: 1
test_interval: 200
base_lr: 0.0001
lr_policy: "step"
gamma: 0.1
stepsize: 100
display: 200
max_iter: 10000
momentum: 0.9
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "models/myfirsttransfer/transfer_train"
solver_mode: GPU
Relevant train_val.prototxt lines:
name: "MyfirstTransferNet"
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mirror: true
crop_size: 227
mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
}
data_param {
source: "data/train_data"
batch_size: 64
backend: LMDB
}
}
layer {
name: "data"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mirror: false
crop_size: 227
mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
}
data_param {
#source: "examples/imagenet/ilsvrc12_val_lmdb"
source: "data/val_data"
batch_size: 50
backend: LMDB
}
}
......
layer {
name: "accuracy"
type: "Accuracy"
bottom: "fc9"
bottom: "label"
top: "accuracy"
include {
phase: TEST
}
}
layer {
name: "loss"
type: "SigmoidCrossEntropyLoss"
bottom: "fc9"
bottom: "label"
top: "loss"
}
Steps to reproduce
Tried solutions
System configuration
- Operating system: Windows 10
- Compiler: Anaconda Spyder
- CUDA version (if applicable): 8
- CUDNN version (if applicable): 7.2
- BLAS: open
- Python version (if using pycaffe): 3.5
- MATLAB version (if using matcaffe):
