Dalia:
brown seed was much larger than its current size when it was green = logical.
so speaks of greater disparity in size of the two = obvious.
second the smaller seed could be from the same tree ... fruits of same tree can be of different size, dropping off means some violence, high winds storm rain storm or birds or animals dropped it, so it remains small...????
i can only suggest cut/saw it neatly in half and see if the inside structure is the same
if only one seeds inside, only Clusiaceae with one seeded fruit is calophyllum
==
i wish you had picked a leaf and photographed that... strong primary vein and almost parallel secondary veins with inconspicuous tertiary veins = calophyllum
=======
or let it dry and see if it begins to look the same as the brown on...
Secondly what other trees were there that could produce roundish green fruits?
==
if you are back in kolkata,
salt lake central park has many calophyllum inophyllum trees to the left of the main walkway keeping the lake to the right
you could check out resemblance to your tree there and the fallen seeds early in the morning will be still there before the sweepers haul them away..