How to create a new project using large dataset?

[Ye Xu]

Hi, everyone, 

I'm brand new user of OpenRefine and had issue of creating project using a dataset about 115Mb with about half million rows of data. The openRefine showed reaching maximum memory (536Mb) in the process of creating the new project. So I scaled up the max memory size and initial memory size in the the configuring file "openrefine.l4j". I've allocated about 4Gb memory to max memory size and 1024 Mb to the initial memory size. However, nothing changed and I got the same "reaching maximum memory 536Mb" warning and it took forever to create the project. Any suggestion about this issue?

[Thad Guidry]

Did you ensure that you used the right syntax for specifying the memory allocation?

-Xmx4GB

or

-Xmx4096MB

The only other thing that might be possibly causing issues is that your particular JVM version cannot be modified by the openrefine.l4j.ini depending on access rights, OS preferences in Control Panel, etc.

You might also try the OpenRefine - Windows kit with embedded Java to avoid issues such as that.

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

On Sun, Jun 20, 2021 at 4:41 PM Ye Xu <xuye...@gmail.com> wrote:

Hi, everyone, 

I'm brand new user of OpenRefine and had issue of creating project using a dataset about 115Mb with about half million rows of data. The openRefine showed reaching maximum memory (536Mb) in the process of creating the new project. So I scaled up the max memory size and initial memory size in the the configuring file "openrefine.l4j". I've allocated about 4Gb memory to max memory size and 1024 Mb to the initial memory size. However, nothing changed and I got the same "reaching maximum memory 536Mb" warning and it took forever to create the project. Any suggestion about this issue?

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/a941792a-6acb-4f6b-8839-de198d5a1513n%40googlegroups.com.