PDF search limit change

[Gábor Kovács]

Hello, is it possible to modify this value to scan the entire document? I mean specifically "truncates PDF text after the first 65,535 bytes". Possible to change this ? 

[Dan Gillean]

Hi Gábor,

You will need to make a local customization to the value column in the property_i18n table in AtoM's MySQL database. The field is currently set as TEXT, and you will want to change its type to MEDIUMTEXT or LONGTEXT. For further context, see: 

• https://groups.google.com/g/ica-atom-users/c/Y9ot1jR4djI/m/cmjil24NBwAJ
• https://groups.google.com/g/ica-atom-users/c/BymeAnJ9P28/m/2YsQqxb4AgAJ

For reference, here are the 4 TEXT types available in MySQL: 

• TINYTEXT: 255 characters - 255 B
• TEXT: 65,535 characters - 64 KB
• MEDIUMTEXT: 16,777,215 - 16 MB
• LONGTEXT: 4,294,967,295 characters - 4 GB

See: 

• https://dev.mysql.com/doc/refman/8.0/en/string-type-syntax.html
  
• https://dev.mysql.com/doc/refman/8.0/en/blob.html
  
• See also: https://dev.mysql.com/doc/refman/8.0/en/alter-table.html

I strongly recommend you backup your data before trying to alter any table settings! You'll need to restart MySQL and PHP-FPM after making any changes, and you would also want to re-extract the text layer after, using this command-line task: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#re-indexing-pdf-text
  

Finally, you would also want to rebuild the search index, so that newly extracted text shows up in search results. 

Good luck! 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Thu, Oct 22, 2020 at 4:24 AM Gábor Kovács <kovacsg...@gmail.com> wrote:

Hello, is it possible to modify this value to scan the entire document? I mean specifically "truncates PDF text after the first 65,535 bytes". Possible to change this ? 

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/81a5bb9c-765a-471b-8a9f-e68d97aeb301n%40googlegroups.com.

[David Juhasz]

Hi Gábor,

There is also a line in the code that needs to be changed to allow more than 65,535 bytes of PDF text to be stored and indexed:

https://github.com/artefactual/atom/blob/stable/2.6.x/lib/model/QubitDigitalObject.php#L3146

Best regards,

David

--

David Juhasz
Senior Developer
Artefactual Systems
he/him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJXH%2Bo4z4wantk1WreyYn5guxWOkuF262Rq2ownDPqNUw%40mail.gmail.com.