ADAM not working sometimes

[Shawn Liu]

Hi all,

I've been using Caffe to train some ConvNets recently. I always used ADAM as the default optimization strategy. However, I noticed that sometimes it just doesn't work. The test accuracy doesn't change much, so does the training loss. However, if I switch to ADAGRAD, I can see test accuracy increase.

Any ideas where might go wrong? I used the bvlc_alexnet train_val prototxt. My solvers are below:

test_iter: 377

test_interval: 879

base_lr: 0.005

display: 109

max_iter: 26370

lr_policy: "exp"

gamma: 0.999805505096

weight_decay: 0.0005

snapshot: 879

snapshot_prefix: "snapshot/snapshot"

solver_mode: GPU

net: "train_val_conv_5.prototxt"

solver_type: ADAGRAD

test_iter: 377

test_interval: 879

base_lr: 0.001

momentum: 0.9

momentum2: 0.999

display: 109

max_iter: 26370

lr_policy: "fixed"

snapshot: 879

snapshot_prefix: "snapshot/snapshot"

solver_mode: GPU

net: "train_val_conv_5.prototxt"

solver_type: ADAM

Thanks,

Siyuan

[Shawn Liu]

Any helps?