Uploading old projects into OpenRefine

[Meredith Hale]

Hi,

I recently upgraded to OpenRefine 3.1 from OpenRefine 2.7. Before the upgrade, I saved all my files on my desktop. I'm realizing now that I need two of my old project files, but I'm unsure which files should be imported (all in a zip?) and if it is even possible to upload a 2.7 project into a 3.1 version of OpenRefine. The general file structure I'm dealing with is included here:

Thanks for your help!

Meredith

[Antonin Delpeuch (lists)]

Hi Meredith,

It is great that you have backed up your workspace before the upgrade.
In principle, projects created with 2.7 should still be readable by
newer versions. Have you changed your workspace directory manually, or
are you using a different machine?

You can run the new version on your previous workspace (make sure you
keep a copy of it somewhere else). If you prefer, you can also run 2.7,
export your projects to an archive from it, and import that archive from
the newer version.

Let us know if it works!
Antonin

On 7/29/19 6:02 PM, Meredith Hale wrote:
> Hi,
>
> OpenRefineProjectFolders.png <about:invalid#zClosurez>I recently

> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine+...@googlegroups.com
> <mailto:openrefine+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com
> <https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Meredith Hale]

Thanks Antonin. I was attempting to import a project into 3.1 (which didn't work), but realized I just needed to move the project folders over to ~/Library/Application Support/OpenRefine on my Mac. Things are working now!

Meredith

> an email to openr...@googlegroups.com
> <mailto:openrefine+unsub...@googlegroups.com>.

[Antonin Delpeuch (lists)]

Awesome! By the way, I would recommend using 3.2 over 3.1 - we fixed
quite a lot of bugs between these two versions!

Antonin

> > an email to openr...@googlegroups.com <javascript:>
> > <mailto:openrefine+...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com
> <https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/openrefine/800229a8-0292-4d68-a1e8-28ceec50f0b9%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to openrefine+...@googlegroups.com
> <mailto:openrefine+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openrefine/92643553-1d7d-465f-8de1-558ea3a0aa86%40googlegroups.com
> <https://groups.google.com/d/msgid/openrefine/92643553-1d7d-465f-8de1-558ea3a0aa86%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Meredith Hale]

Sadly it looks like the changes I made yesterday in OpenRefine didn't save after closing the application? Is this one of the bugs you've found previously?

Meredith

[Antonin Delpeuch (lists)]

Hi Meredith,

Which version of OpenRefine did you run when you had this issue?
If it was 3.1, it is possible that some change fixed it in 3.2, but I do
not remember working such a symptom being reported before.

Antonin

> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine+...@googlegroups.com
> <mailto:openrefine+...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openrefine/b5eb006f-f160-4004-9696-d160aafcda30%40googlegroups.com
> <https://groups.google.com/d/msgid/openrefine/b5eb006f-f160-4004-9696-d160aafcda30%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Meredith Hale]

Hi Antonin,

It was 3.1. I'm currently trying to install 3.2.

Meredith

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/a6b8a0b5-37ca-16e2-f37a-106213943ef5%40antonin.delpeuch.eu.

--

"To know when one's self is interested is the first condition of interesting other people."
-Walter Pater

www.meredithhaleart.blogspot.com
www.meredithhale.com

[Jevon, Graham]

Hi Meredith, Antonin

 

This might be unrelated/irrelevant, but I previously found that prior to downloading java 64bit, my workspace directory was pointing to appdata/local/ but after downloading JDK the directory was pointing to a new directory in appdata/roaming/. Until I realised this, I thought that I had lost saved work. Once I realised this, I just copied the data across.

 

Best wishes

 

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/CALf43Go%3D1QP8ugNJFRW4K1MHhX9hNjRQg7cEuKuzcuvBGuw8uw%40mail.gmail.com.

 
******************************************************************************************************************

Experience the British Library online at www.bl.uk

The British Library’s latest Annual Report and Accounts : www.bl.uk/aboutus/annrep/index.html

Help the British Library conserve the world's knowledge. Adopt a Book. www.bl.uk/adoptabook

The Library's St Pancras site is WiFi - enabled

*****************************************************************************************************************

The information contained in this e-mail is confidential and may be legally privileged. It is intended for the addressee(s) only. If you are not the intended recipient, please delete this e-mail and notify the postm...@bl.uk : The contents of this e-mail must not be disclosed or copied without the sender's consent.

The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the British Library. The British Library does not take any responsibility for the views of the author.

*****************************************************************************************************************

Think before you print

[Bradley Ferguson]

I haven't tested it, but it is my understanding that you need to Ctrl-C on the OpenRefine console to get it to flush work to disk. If you just close the console, you aren't guaranteed to have your latest changes saved.

[Meredith Hale]

I still have not been able to make changes and save those changes. This is apparent because when I export the project and then reload it, the project has much fewer edits. Any help people can provide on how to find where your changes are being saved on a Mac would be much appreciated.

Meredith

[Meredith Hale]

I also cannot extract the actions via JSON (as another safety). The extract button doesn't go anywhere on my interface when you click it.

Meredith

On Monday, July 29, 2019 at 2:04:44 PM UTC-4, Meredith Hale wrote:

[Owen Stephens]

Hi Meredith,

Is this affecting all your projects?

If it is a specific project that is effected, are you able to export it and share it here?

If it is all projects, can you try starting OpenRefine in a slightly different way to usual:

Find the OpenRefine app/icon in Finder

Ctrl+Click on the icon and select "Show Package Contents" from the context menu that displays

This should open a new Finder menu showing a folder called "Contents" - navigate into this folder then into the "MacOS" folder

Ctrl+Click on "JavaAppLauncher"

Choose 'Open With' from menu, and select "Terminal"

This should run OpenRefine with a terminal window showing text (similar to how OpenRefine runs on Windows). This terminal window will show information that may help work out what the problem is. If you try opening a project, making a change, then see if there is any information in the terminal window about errors etc. If you paste any information from the terminal window here it may help us work out what is going wrong.

Thanks

Owen

[Meredith Hale]

Hi Owen,

This was not affecting all of my projects, just one with an unruly number of columns. I reverted to OpenRefine 2.7 and am now able to get OpenRefine to save changes for this large project. In 3.2 there seems to be a max number of columns the system can handle. After that the final column is simply given the text "ERROR: TOO MANY COLUMNS" and all other columns are deleted when you export the data. I'll submit this to the OpenRefine GitHub, but it seems odd that an older version of the program can handle more data than the new version. Let me know if you have any suggestions. Sorry for the delay in responding back.

Meredith

[Luc Gauvreau]

Bonjour,

Same situation after upgrading 3.3 to 3.5.2

My current directory is 

• C:\Documents and Settings\(user id)\Local Settings\Application Data\OpenRefine
• AND all my +100 projects still there.

BUT...

• Directory of the new version installed 3.5.2 is:  C:\Users\Luc\AppData\Roaming\OpenRefine... There is just few projects, and old ones?

How can I change the Directory?

Merci,

Luc

[Thad Guidry]

Hi Luc,

1. backup your current directory for OpenRefine (just in case).

2. Yes with newer versions of OpenRefine we now ask the OS what the user folder is and based on it's response we then set the default OpenRefine workspace folder to be located within that.  (NOTE:  Things changed with newer versions of Windows versus older versions of Windows and this might be affecting you... for details of Windows User Profile changes, see:  https://answers.microsoft.com/en-us/windows/forum/all/where-can-i-find-the-documents-and-settings-file/18894fab-81bc-4c38-93df-f2d4a0b3592b
3. you can change or update the default OpenRefine workspace folder to point to the folder path that you want using the commands or preferences as noted in our documentation here:  https://docs.openrefine.org/manual/running#starting-with-modifications  <-- ensure to read that entire section, since you can either use the /w parameter, or you can set the refine.data_dir= in the refine.ini file.

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

[Luc Gauvreau]

Merci Thadg,

I resolve the problem. I have access to all my projects.

There is an other problem: The History of all fonctions I wrote with GREL and Regex are disappeared, lost???

For me, it's very a problem. I'm a amateur historian, and all these fonctions are all I know to use. Whitout this History, I'm very, very lost myself.

Do you know, where his this History? 

I make a complete copy of my workspace before.

I try to run my 3.3 version: it works, but I lost all my History there too.

(I'll open an other conversation with this problem).

Merci for your help,

Luc

[Thad Guidry]

Hi Luc,

We do not have a conversion script to import your Starred or History of functions which are stored per project, and we know this was an issue during version 3.3 and likely prior and has not been resolved or made "bulletproof" as tfmorris had said at one point in bug Loss of expression history and language at restart · Issue #2543 · OpenRefine/OpenRefine (github.com).
So we have seen others report this bug as you have since then in bug Backup the history and starred commands · Issue #3512 · OpenRefine/OpenRefine (github.com)

So subsequently Starred expressions and History expressions might not carry over into your projects when doing an upgrade from 3.3 to 3.5+ and pointing to the same original workspace as you did initially.

I would love to see this fixed properly, but I don't think anyone had the time yet.  BUT, You could help us by providing your backed up old workspace folder and sending to me privately via email as a .zip or put it up on Google Drive or Dropbox, etc. and send to my email address and we'll see what we can do.

 

However, regardless of the above, those old expressions are still in your old metadata.json (and metadata.old.json) files.

You might be able to individually go into each project folder in your old workspace (backup) and then open those metadata.json files within a text editor like Notepad++ and scroll all the way over to the near end and see the "preferences" key and entries and then copy/paste the GREL out for the keys of "scripting.starred-expressions" and "scripting.expressions".

As reference, here's a snippet of the json string built in our code that creates some of the metadata.json file structure that you will need to look for as explained in previous sentence:

String jsonData = "{\"projectIDs\":[]\n" +
	",\"preferences\":{\"entries\":{\"scripting.starred-expressions\":" + starred +
	",\"scripting.expressions\":" + expressions + "}}}";

Then after copying out the relevant expressions you like or want to keep, you can then apply them as needed in the projects you need, and hopefully things are OK going forward.

Let me also ask one of our other developers...

Antonin - do you have any other advice or suggestions here?

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/ce9ae067-ca75-48a4-91a2-d4daa6b497e6n%40googlegroups.com.

[Antonin Delpeuch (lists)]

Thanks Thad for the extensive workaround!

I am curious as to what might have caused this, and do not have anything else to propose, except perhaps looking for a "workspace.old.json" file in the workspace directory.

This could be caused by a deserialization error when reading workspace.json, so I am hopeful that if we added error reporting with Sentry we could potentially understand better what could have caused this.

Antonin

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/CAChbWaNLBvorsU_bkFP8nM9FCNMnhBtLVzNZD6kq9%3DWbQF%2BD-g%40mail.gmail.com.

[Luc Gauvreau]

Bonjour Thad,

Thanks for your help.

I notices that before with 3.3 my workspace  location was:

C:\Documents and Settings\(user id)\Local Settings\Application Data\OpenRefine

Now, with 3.5.2, it's:

C:\Users\(user id)\AppData\Local\OpenRefine

(I see all my projects: no problem with this change).

Is it related to my lost History and Favorite?

I send you my backup with Drive (4G).

---- 

I look in the json file: I'll try to extract fonctions. I used a lot of éâôèç... But it looks like that:

value.replace(/(@[A-Z][a-z��������]* [A-Z][A-Z]*@)/i,\"@$1%\")",

I try some encoding, but I could'not find the good one. Any suggestion?

Merci!

Luc

[Thad Guidry]

Hi Luc,

That char � is a unicode replacement char and when UTF-8 is not used initially for encoding. (which is the bug that we fixed for preferences and Starred/History and now should work correctly in later versions of OpenRefine which you now have with 3.5.2)

Also, you might try starting OpenRefine 3.5.2 with the following JAVA OPTION added to the refine.ini file:

-Dfile.encoding=UTF-8

Other than that, I am out of ideas.

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine/029e923c-e687-4460-9258-63b5c6f8e034n%40googlegroups.com.