Could not obtain latest ID error

[Tamim]

I have a job where I need a lot of domains, and I'm using CeRTSeacH to paginate through the DB and get the subdomains I need. Are there any guidelines I have to follow doing this? I assume the CeRTSeacH code already follows all the guide-lines and is compliant with all fair-usage policies? Could I run scraping jobs for a few days without causing any issues? 

I tried it last night, and was able to scrape about 12,000 records that I needed but it started giving me this error:

{"err":"conn closed","level":"error","msg":"Could not obtain latest ID","time":"2025-09-14T08:08:08+05:30"}

{"err":"conn closed","level":"error","msg":"Could not obtain latest ID","time":"2025-09-14T08:08:23+05:30"}

It gave me the same error even after scraping "just a few" records today.. 

As far as I can tell, this error appears when we fail to fetch the max ID in the certificates table, I can just omit this query, as it feels quite unnecessary to run everytime, but again should it fail to begin with? Symptom of an underlying issue, because I dont see any records after this starts failing aswell.

[Tamim]

I understand that my db connection was closed, but why? I don't want to strain the DB too much, so I'd love to hear any guidance you have regarding this. I could just write a script to restart the connection and resume the job, but please advise me how to proceed.