Strange behaviour of rbmBB with small learning rates

[Максим Кореневский]

Hello!

I've tried rbmBB with default parameters and found it behaves pretty good with respect to reconstruction error.
Then I added the free energy (average per sample) computation after each epoch and observed monotonic decrease of it during learning.

But then I added the free energy computation at start of learning too and I was surprised by the fact that after first epoch free energy increased by a large value and only then it started to decrease but didn't reach its start value even after 50 epochs.

Moreover, when I reduced learning rate 'eta' by two orders of magnitude (from 0.1 to 0.001) I observed that the free energy increases for a several epochs (10-30) and only after that starts to decrease.

It appears to be strange for me, because the free energy is the ultimate objective function to minimize during RBM learning and small antigradient steps must decrease it.
Can anybody explain me this trouble?

Here is a my code of free energy computation:
    FE=0;
    numcasestotal=0;
    for batch = 1:numbatches
        data = batchdata{batch};
        [numcases numdims]=size(data);
        X=data*W + repmat(b,numcases,1);
        FEbatch=-data*c'-sum(log(1+exp(X)),2);
        FE=FE+sum(FEbatch);
        numcasestotal= numcasestotal+numcases;
    end;
    FE=FE/numcasestotal;
It follows an expression (25) from G.Hinton's "A Practical Guide to Training RBMs".

Thanks in advance, Maxim.