Handling large projects & memory issues in OpenRefine

[Thad Guidry]

Recently Java 8 has arrived and offers a few benefits that can directly impact and improve performance of OpenRefine without the development team having to change our code to improve performance. (something that is still on our road map)

Java 8 offers a new feature called String Deduplication that is available inside the JVM itself.  In practice, the performance hit for using it from what I have witnessed is absolutely marginal to none at all.

In order to turn on String Deduplication (and additionally Print the statistics as well in the command console if you desire) you can uncomment and add the following JAVA_OPTIONS to your refine.ini file:

-XX:+UseG1GC -XX:+UseStringDeduplication -XX:+PrintStringDeduplicationStatistics

​Enjoy !

Thad

+ThadGuidry

[Owen Stephens]

Hi Thad,

I found that to get this to work I had to put each option on a separate line in refine.ini:

JAVA_OPTIONS=-XX:+UseG1GC

JAVA_OPTIONS=-XX:+UseStringDeduplication

JAVA_OPTIONS=-XX:+PrintStringDeduplicationStatistics

(not sure if this is expected or not, but when I put these in a single JAVA_OPTIONS line I got errors of the nature /var/folders/50/vz8ngr_13gj1syf7wdwhlvkh0000gn/T/refine.XXXXXXX.OJo5icDJ: line 3: export: `-XX:+PrintStringDeduplicationStatistics': not a valid identifier)

Owen

[Owen Stephens]

That's on OS X 11.3

[Thad Guidry]

Wierd.  Dunno about OS X. :)  Thanks for the update however , Owen !

Thad

+ThadGuidry

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[qi cui]

Not sure if adding the double quote between the value will help. OR use a lazy way to parse the ini file by sed the file and run as a shell to export the variables.