Caffe Convolution floating point calculation rounding error at 7th decimal place?

[Peerajak Witoonchart]

I found that Caffe Convolution has little difference w.r.t manual calculation. I show this problem using the python code below. You can see the result manual calculation differs from caffe calculation even though I use float32 to calculate. What is the cause? How to solve this little difference?

[(k, v.data.shape) for k, v in solver.net.blobs.items()]

[('data', (4, 3, 1713, 1713)),
 ('label', (4, 24, 1, 1)),
 ('conv1', (4, 96, 429, 429)),
 ('pool1', (4, 96, 215, 215)),
 ('norm1', (4, 96, 215, 215)),
 ('conv2', (4, 256, 215, 215)),
 ('pool2', (4, 256, 108, 108)),
 ('norm2', (4, 256, 108, 108)),
 ('conv3', (4, 384, 108, 108)),
 ('conv4', (4, 384, 108, 108)),
 ('conv5', (4, 256, 108, 108)),
 ('convsvm', (4, 15, 104, 104)),
 ('loss', (1,))]

[(k, v[0].data.shape) for k, v in solver.net.params.items()]

[('conv1', (96, 3, 11, 11)),
 ('conv2', (256, 48, 5, 5)),
 ('conv3', (384, 256, 3, 3)),
 ('conv4', (384, 192, 3, 3)),
 ('conv5', (256, 192, 3, 3)),
 ('convssvm', (15, 256, 5, 5)),
 ('loss', (96091,))]

n,ci,hi,wi = solver.net.blobs['conv5'].shape#n=4
co,channel,hp,wp = solver.net.params['convssvm'][0].shape
ho = hi-hp+1
wo = wi-wp+1
print ho,wo
convssvm_cal = np.zeros((co,ho,wo),dtype='float32')

#We only calculate at n=0
for ico in range(co):
    for ih in range(ho):
        for iw in range(wo):
            convssvm_cal[ico,ih,iw] = np.sum(np.multiply(solver.net.blobs['conv5'].data[0,:,ih:ih+5,iw:iw+5],\
                                                         solver.net.params['convssvm'][0].data[ico,:,:,:]))
print '======Manual Cal ==============='
print convssvm_cal.shape
print convssvm_cal[0,1:10,1:10]
print '======Caffe================'
print solver.net.blobs['convssvm'].data.shape
print solver.net.blobs['convssvm'].data[0,0,1:10,1:10]

104 104
======Manual Cal ===============
(15, 104, 104)
[[ 214.9699707   183.41452026  184.61450195  178.46702576  177.38441467
   185.26622009  190.00328064  193.26628113  194.03968811]
 [ 176.69282532  163.93658447  173.06672668  169.03997803  166.86943054
   173.13253784  175.34861755  176.12556458  174.84169006]
 [ 162.23005676  157.97476196  171.03063965  169.62014771  168.98962402
   175.99473572  177.61654663  177.20144653  174.01849365]
 [ 148.75254822  145.73674011  158.26408386  156.0193634   154.20718384
   160.84342957  162.40013123  160.46391296  155.53564453]
 [ 144.53422546  140.8062439   151.54417419  148.88557434  144.8578186
   150.57751465  154.30436707  156.7893219   154.43161011]
 [ 146.38804626  141.20291138  150.83670044  148.89775085  144.61340332
   149.44239807  155.67054749  168.53955078  174.42933655]
 [ 153.36924744  148.88755798  158.80445862  156.01245117  150.85470581
   153.25511169  160.25935364  177.60931396  190.25639343]
 [ 158.6247406   157.68592834  169.72207642  166.96284485  159.60473633
   158.69094849  164.29534912  182.657547    199.13529968]
 [ 158.37397766  159.32563782  171.95677185  168.82000732  159.37504578
   155.48500061  159.38833618  180.42996216  202.17167664]]
======Caffe================
(4, 15, 104, 104)
[[ 214.96995544  183.41452026  184.61450195  178.46704102  177.38441467
   185.26622009  190.00328064  193.26628113  194.03968811]
 [ 176.69282532  163.93658447  173.06671143  169.03997803  166.86943054
   173.13253784  175.34863281  176.12556458  174.8416748 ]
 [ 162.23005676  157.97477722  171.03063965  169.62014771  168.98962402
   175.99473572  177.61654663  177.20144653  174.01849365]
 [ 148.75254822  145.73674011  158.26408386  156.0193634   154.20718384
   160.84344482  162.40013123  160.46391296  155.53565979]
 [ 144.53424072  140.80625916  151.54417419  148.8855896   144.8578186
   150.57751465  154.30436707  156.78930664  154.43161011]
 [ 146.38804626  141.20291138  150.83670044  148.89775085  144.61340332
   149.44239807  155.67054749  168.53955078  174.42933655]
 [ 153.36924744  148.88755798  158.80445862  156.01243591  150.85470581
   153.25511169  160.2593689   177.60931396  190.25639343]
 [ 158.6247406   157.68592834  169.72209167  166.96284485  159.60473633
   158.69096375  164.29533386  182.657547    199.13531494]
 [ 158.3739624   159.32565308  171.95677185  168.81997681  159.37503052
   155.48500061  159.38833618  180.42996216  202.17167664]]

[Jan]

Probably rounding errors. As caffe computes the convolution differently from this and there are a lot of floating point operations involved, this does not surprise me. And on the GPU rounding is also handled differently than on the CPU iirc. Apparently this loss of precision does no real harm, the trend is to even go to 16 bit floats (mostly because of the memory requirements for training).

Jan