Thank you for your answer. I have ran the cron job but it doesn't make a
difference, the number of files doesn't decrease. It did give me another
idea. I went into the backend to check the Clear Cache page and it shows
the same number of files on the server as SSH shows me, 46997 but Joomla
shows a size of 21.3kb instead of the actual 724MB. Doing a clean cache
from the backend does not decrease the files either, so I guess it is in
line with what the cronjob did.
Looking at the folder, these cache files go back to February 5th 2012 when
I started to use Smart Search. Something doesn't seem right here. Just want
to see if there is anything I can check, before wiping the folder, that may
be of help in preventing this from happening again.