CSV export file size limit to 10000 lines

[Ricardo Pinho]

Dear AtoM users, developers and maintainers,

The CSV exports function performed via the user interface generates compressed ZIP packages with the resulting csv text files. If the number of results exceeds the 10000 the results are split in several csv text files.

My question, is there any way we can change this 10000 size limit to a larger number?

Thank you in advance!

Best regards,

--

Ricardo Pinho

[Dan Gillean]

Hi Ricardo, 

I checked with our AtoM developer Radda, who believes this is being set here: 

• https://github.com/artefactual/atom/blob/qa/2.x/lib/job/arInformationObjectCsvExportJob.class.php#L90
  

Keep in mind we have not tested the performance implications of changing this value! 

And note as well that if you wanted, for some reason, to include LESS rows in a single CSV, then there's already a CLI task option for this. I suspect it will not allow you to set a number higher than the one linked above in the code, but the --rows-per-file option can be used to temporarily set the CSV row count smaller. When using it, you specify a directory as the output, rather than a (single) CSV file path. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-import-export/#csv-export-task-options
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/7b381424-2dfb-4c3b-853e-8f2ce8ab4e5bn%40googlegroups.com.

[Ricardo Pinho]

Hi Dan,

Thank you for your reply and providing a solution and explanation.

I'm aware that higher values can cause unstable behavior. But merging several files is an unpleasant task...

With the correct RAM management, high volume register export to file shouldn't be a problem nowadays.

I've tried the CLI export command. Unfortunately it has 2 major problems:

1. The Reference Code is not exported and it is required.

2. It needs to be done by a SysAdmin and not a common AtoM user.

I will give your suggestion a try (on a test VM) and let you know the results.

Cheers,

Ricardo Pinho

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKkvzK0w9XXdy_x5Hf3a2L_6aMKJxU7kRxeRx6QXLvi%3DQ%40mail.gmail.com.

--

Ricardo Pinho