epoch 0: train loss: 1.375314 train acc 0.265000 test loss 1.171191 test acc 0.585000
epoch 1: train loss: 1.223649 train acc 0.375000 test loss 1.055289 test acc 0.595000
epoch 2: train loss: 1.116125 train acc 0.465000 test loss 0.956632 test acc 0.555000
epoch 3: train loss: 0.997714 train acc 0.530000 test loss 0.870735 test acc 0.605000
epoch 4: train loss: 1.016216 train acc 0.535000 test loss 0.786455 test acc 0.700000
epoch 5: train loss: 0.858704 train acc 0.670000 test loss 0.708278 test acc 0.765000
epoch 6: train loss: 0.910774 train acc 0.600000 test loss 0.690680 test acc 0.780000
epoch 7: train loss: 0.806082 train acc 0.620000 test loss 0.621082 test acc 0.840000
epoch 8: train loss: 0.744155 train acc 0.700000 test loss 0.598704 test acc 0.825000
epoch 9: train loss: 0.676767 train acc 0.720000 test loss 0.556982 test acc 0.885000
epoch 10: train loss: 0.674348 train acc 0.735000 test loss 0.532380 test acc 0.880000
epoch 11: train loss: 0.650707 train acc 0.730000 test loss 0.528952 test acc 0.885000
epoch 12: train loss: 0.563217 train acc 0.775000 test loss 0.461938 test acc 0.905000
epoch 13: train loss: 0.596845 train acc 0.770000 test loss 0.441529 test acc 0.875000
epoch 14: train loss: 0.572269 train acc 0.765000 test loss 0.475156 test acc 0.845000
epoch 15: train loss: 0.588880 train acc 0.755000 test loss 0.444637 test acc 0.865000
epoch 16: train loss: 0.586087 train acc 0.760000 test loss 0.403933 test acc 0.880000
epoch 17: train loss: 0.496034 train acc 0.800000 test loss 0.414725 test acc 0.915000
epoch 18: train loss: 0.472452 train acc 0.820000 test loss 0.402316 test acc 0.910000
epoch 19: train loss: 0.528015 train acc 0.790000 test loss 0.428631 test acc 0.835000
epoch 20: train loss: 0.474193 train acc 0.825000 test loss 0.388237 test acc 0.880000
epoch 21: train loss: 0.517822 train acc 0.765000 test loss 0.345394 test acc 0.905000
epoch 22: train loss: 0.438922 train acc 0.830000 test loss 0.339430 test acc 0.900000
epoch 23: train loss: 0.467753 train acc 0.800000 test loss 0.332779 test acc 0.910000
epoch 24: train loss: 0.422628 train acc 0.800000 test loss 0.319525 test acc 0.880000
epoch 25: train loss: 0.382614 train acc 0.875000 test loss 0.311163 test acc 0.865000
epoch 26: train loss: 0.403976 train acc 0.820000 test loss 0.374219 test acc 0.825000
epoch 27: train loss: 0.369395 train acc 0.860000 test loss 0.303532 test acc 0.905000
epoch 28: train loss: 0.401725 train acc 0.835000 test loss 0.310684 test acc 0.870000
epoch 29: train loss: 0.342000 train acc 0.895000 test loss 0.303024 test acc 0.890000
Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16
Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16
Disabling C code for Elemwise{Cast{float32}} due to unsupported float16
ERROR (theano.gof.opt): Optimization failure due to: local_gpu_elemwise_careduce
ERROR (theano.gof.opt): node: GpuCAReduceCuda{add}{0}(GpuElemwise{sqr,no_inplace}.0)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/fabian/deeplearning_venv/local/lib/python2.7/site-packages/theano/gof/opt.py", line 2036, in process_node
remove=remove)
File "/home/fabian/deeplearning_venv/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/home/fabian/deeplearning_venv/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 518, in replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/fabian/deeplearning_venv/local/lib/python2.7/site-packages/theano/gof/fg.py", line 486, in replace
". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
Variable: id 140574082265808 GpuCAReduceCuda{pre=sqr,red=add}{0}.0
Op GpuCAReduceCuda{pre=sqr,red=add}{0}(GpuElemwise{sub,no_inplace}.0)
Value Type: <type 'NoneType'>
Old Value: None
New Value: None
Reason: local_gpu_elemwise_careduce. The type of the replacement must be the same.
Old Graph:
GpuCAReduceCuda{add}{0} [id A] <GpuArrayType<None>(float32, vector)> ''
|GpuElemwise{sqr,no_inplace} [id B] <GpuArrayType<None>(float16, matrix)> ''
|GpuElemwise{sub,no_inplace} [id C] <GpuArrayType<None>(float16, matrix)> ''
|GpuElemwise{add,no_inplace} [id D] <GpuArrayType<None>(float16, matrix)> ''
| |GpuDot22 [id E] <GpuArrayType<None>(float16, matrix)> ''
| | |GpuElemwise{add,no_inplace} [id F] <GpuArrayType<None>(float16, matrix)> ''
| | |W [id G] <GpuArrayType<None>(float16, matrix)>
| |InplaceGpuDimShuffle{x,0} [id H] <GpuArrayType<None>(float16, row)> ''
| |b [id I] <GpuArrayType<None>(float16, vector)>
|GpuElemwise{Cast{float16}}[]<gpuarray> [id J] <GpuArrayType<None>(float16, row)> ''
|GpuElemwise{true_div,no_inplace} [id K] <GpuArrayType<None>(float32, row)> ''
|InplaceGpuDimShuffle{x,0} [id L] <GpuArrayType<None>(float32, row)> ''
|GpuFromHost<None> [id M] <GpuArrayType<None>(float32, (True, True))> ''
New Graph:
GpuCAReduceCuda{pre=sqr,red=add}{0} [id N] <GpuArrayType<None>(float16, vector)> ''
|GpuElemwise{sub,no_inplace} [id C] <GpuArrayType<None>(float16, matrix)> ''
Hint: relax the tolerance by setting tensor.cmp_sloppy=1
or even tensor.cmp_sloppy=2 for less-strict comparison
epoch 0: train loss: nan train acc 0.390000 test loss nan test acc 0.175000
epoch 1: train loss: nan train acc 0.375000 test loss nan test acc 0.215000
epoch 2: train loss: nan train acc 0.360000 test loss nan test acc 0.140000
epoch 3: train loss: nan train acc 0.390000 test loss nan test acc 0.240000
epoch 4: train loss: nan train acc 0.370000 test loss nan test acc 0.215000
epoch 5: train loss: nan train acc 0.390000 test loss nan test acc 0.190000
epoch 6: train loss: nan train acc 0.345000 test loss nan test acc 0.200000
epoch 7: train loss: nan train acc 0.345000 test loss nan test acc 0.230000
epoch 8: train loss: nan train acc 0.430000 test loss nan test acc 0.210000
epoch 9: train loss: nan train acc 0.375000 test loss nan test acc 0.190000
epoch 10: train loss: nan train acc 0.390000 test loss nan test acc 0.190000
epoch 11: train loss: nan train acc 0.340000 test loss nan test acc 0.160000
epoch 12: train loss: nan train acc 0.410000 test loss nan test acc 0.265000
epoch 13: train loss: nan train acc 0.360000 test loss nan test acc 0.225000
epoch 14: train loss: nan train acc 0.445000 test loss nan test acc 0.165000
epoch 15: train loss: nan train acc 0.350000 test loss nan test acc 0.245000
epoch 16: train loss: nan train acc 0.345000 test loss nan test acc 0.225000
epoch 17: train loss: nan train acc 0.410000 test loss nan test acc 0.185000
epoch 18: train loss: nan train acc 0.420000 test loss nan test acc 0.195000
epoch 19: train loss: nan train acc 0.335000 test loss nan test acc 0.185000
epoch 20: train loss: nan train acc 0.360000 test loss nan test acc 0.185000
epoch 21: train loss: nan train acc 0.335000 test loss nan test acc 0.265000
epoch 22: train loss: nan train acc 0.360000 test loss nan test acc 0.220000
epoch 23: train loss: nan train acc 0.345000 test loss nan test acc 0.175000
epoch 24: train loss: nan train acc 0.350000 test loss nan test acc 0.255000
epoch 25: train loss: nan train acc 0.355000 test loss nan test acc 0.235000
epoch 26: train loss: nan train acc 0.350000 test loss nan test acc 0.185000
epoch 27: train loss: nan train acc 0.370000 test loss nan test acc 0.165000
epoch 28: train loss: nan train acc 0.365000 test loss nan test acc 0.255000
epoch 29: train loss: nan train acc 0.420000 test loss nan test acc 0.160000
We fixed problem with float16 since the last release. Update to the dev version of Theano and update libgpuarray array to 0.6.6
--
You received this message because you are subscribed to the Google Groups "lasagne-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/97f7ca33-d5d9-4289-a201-12698b48f53f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
They could a simple trick, scale the cost to 256 to 2048. Then adjust the learning rate in consequence.
updates = lasagne.updates.adam(loss_tr, lasagne.layers.get_all_params(output_layer, trainable=True), 0.0005)
--
You received this message because you are subscribed to the Google Groups "lasagne-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/7041224f-7bbe-4cda-aa0b-176aef3cbcc7%40googlegroups.com.
thank you very much for your replies! Unfortunately, I am already using adamupdates = lasagne.updates.adam(loss_tr, lasagne.layers.get_all_params(output_layer, trainable=True), 0.0005)which is why I am a bit confused about the network still not training properly. I will try to scale the cost and learning rate
thanks for getting back to me. Yeah I may have misunderstood what you said - my bad!Do you think we should add an epsilon to that specific line? It should not influence fp32 networks but may help others run fp16 in the future.
After being able to run the simple fully connected network, I got overly optimistic about running my actual stuff in float16, with the result that neither segmentation nor classification would work (getting nan's again). Therefore I (again) prepared a little standalone script to demonstrate my problem. This will run cifar10 on a very simple network.
--
You received this message because you are subscribed to the Google Groups "lasagne-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/8dde09e8-cc4c-4f8b-bb06-28cb25c14b8b%40googlegroups.com.