Upgrading to a specific version of Arches - 4.4.1 (not the latest 4.4.2 version)

[Shewring, Tom]

Hi,

 

We have an Arches deployment at 4.1.1, I wish to upgrade this environment to 4.4.1 (not the latest 4.4.2 release).

 

1)      Can you advise on the specific command to do this since ‘pip install arches --upgrade --no-binary :all:’ will install 4.4.2

2)      Are there any other steps that I will need to take when upgrading from 4.1.1 to 4.4.1

 

Many thanks

 

Tom Shewring

GIS Systems Developer

IMT Development Centre

Historic England

We are the public body that helps people care for, enjoy and celebrate England's spectacular historic environment, from beaches and battlefields to parks and pie shops.
Follow us:  Facebook  |  Twitter  |  Instagram     Sign up to our newsletter     

This e-mail (and any attachments) is confidential and may contain personal views which are not the views of Historic England unless specifically stated. If you have received it in error, please delete it from your system and notify the sender immediately. Do not use, copy or disclose the information in any way nor act in reliance on it. Any information sent to Historic England may become publicly available. We respect your privacy and the use of your information. Please read our full privacy policy for more information.

[Alexei Peters]

Hi Tom,

The pip command to install a specific version should be this `pip install myPackage=={version num}

So for Arches that should be:

‘pip install arches==4.4.1 --upgrade --no-binary :all:’

Cheers,

Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

--
-- To post, send email to arches...@googlegroups.com. To unsubscribe, send email to archesprojec...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups "Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archesprojec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/4236dfe5e1b54308b93b8ba2d6e7c9cc%40SVMEX02.english-heritage.org.uk.

[Adam Cox]

Hi Tom, one other thought, sometimes I like to just create a brand new virtual environment and install the new version in there, leaving the existing on as is. Then you can switch which environment is used by your app in your apache config (if that's how you're serving it), etc, and you can switch back if something goes wrong with the pip installation process (which, frankly, is unlikely).

As for other actions to take, it's possible you will need to run migrations in your project. I'd recommend looking back through release notes for the incremental versions between 4.1.1 and 4.4.1, which can be found on this forum.

Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/CA%2BZLqy8oUEOsJoaPMWpip-ac3woMbTMCFxzRhg-PAdE6qoCA-g%40mail.gmail.com.

[Cyrus Hiatt]

Hi Tom - 

One of the challenges for upgrading from 4.1.1 will be upgrading your project(s). This is because at version 4.2 we switched from Bower to Yarn for js dependencies because bower was getting deprecated. As Adam has mentioned, each version of Arches has the upgrade process outlined in the release notes: https://github.com/archesproject/arches/tree/master/releases.  You could go from version to version following the upgrade instructions for each version. Alternatively, you could try the process below which is a bit shorter. I test the upgrade from version to version with each release, so it should go smoothly. I haven't tested the shortcut process below, but I think it should be fine. In either case, there are quite a few steps going from 4.1 to 4.4, so I would recommend creating a database backup and web server snapshot (if possible) before getting started.

1. upgrade to version 4.3.1:

pip install arches==4.3.1 --upgrade --no-binary :all:
pip uninstall pycryptodome, rdflib-jsonld
pip install pycryptodome django-oauth-toolkit==1.1.2 PyLD[requests]==1.0.3 pyprind==2.11.2

python manage.py migrate
python manage.py es delete_indexes
python manage.py es setup_indexes
python manage.py es index_database

2. follow the project upgrade steps in the 4.2.0 release notes starting with the yarn install:

https://github.com/archesproject/arches/blob/master/releases/4.2.0.md#installing-yarn

3. Then follow the project upgrade steps for version 4.3.1:

https://github.com/archesproject/arches/blob/master/releases/4.3.1.md#upgrading-an-arches-project

4. Upgrade to version 4.4.1:

pip install arches==4.4.1 --upgrade --no-binary :all:
python manage.py migrate
python manage.py es delete_indexes
python manage.py es setup_indexes
python manage.py es index_database

5. Then complete the project upgrades for 4.4.1:

 

https://github.com/archesproject/arches/blob/master/releases/4.4.1.md#upgrading-an-arches-project

Hope that helps and that your upgrade goes smoothly. Let me know if you have any questions.

- Cyrus

To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/CAGYBTavsKPzuTQcZyQR7RGhRV0UFH7FKRrGu_zTJG1QaqHuYAg%40mail.gmail.com.

[Shewring, Tom]

Hi,

 

Thank you for you your replies. I took a snapshot of the server before I started the upgrade process. I will roll back to the snapshot and follow the process Cyrus has detailed. Before I do so, would you be able to comment on the process I followed and the error I encountered -

1)      Upgrade Yarn and Node.js to 1.13.0 and 10.15.2 respectively (this brings the versions of this software up to the same level as the our other working 4.4.1 environments)

2)      python -m pip install --upgrade pip

3)      pip install arches==4.4.1 --upgrade --no-binary :all: (this completed successfully, upgrading directly from 4.1.1)

4)      python manage.py migrate – (errors encountered) – is this due to a lack of memory on the server? – (see final error massage - django.db.utils.DatabaseError: out of memory for query result)

 

Operations to perform:

  Apply all migrations: admin, auth, contenttypes, guardian, models, oauth2_provider, sessions

Running migrations:

  Applying models.2293_recent_edits... OK

  Applying models.2974_provisionaledits_in_editlog... OK

  Applying models.3098_custom_map_markers... OK

  Applying models.3199_graphmodel_color... OK

  Applying models.3201_remove_node_and_nodetype_branches... OK

  Applying models.3210_card_components... OK

  Applying models.3314_graphmodel_jsonldcontext... OK

  Applying models.3201_replace_node_and_nodetype_branches... OK

  Applying models.3201_second_removal_of_node_nodetype_branch... OK

  Applying models.deserialize_provisional_edits...Traceback (most recent call last):

  File "manage.py", line 29, in <module>

    execute_from_command_line(sys.argv)

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\__init__.py", line 364, in execute_from_command_line

    utility.execute()

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\__init__.py", line 356, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\base.py", line 283, in run_from_argv

    self.execute(*args, **cmd_options)

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\base.py", line 330, in execute

    output = self.handle(*args, **options)

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\commands\migrate.py", line 204, in handle

    fake_initial=fake_initial,

  File "c:\obap\env4.1.1\lib\site-packages\django\db\migrations\executor.py", line 115, in migrate

    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)

  File "c:\obap\env4.1.1\lib\site-packages\django\db\migrations\executor.py", line 145, in _migrate_all_forwards

    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)

  File "c:\obap\env4.1.1\lib\site-packages\django\db\migrations\executor.py", line 244, in apply_migration

    state = migration.apply(state, schema_editor)

  File "c:\obap\env4.1.1\lib\site-packages\django\db\migrations\migration.py", line 129, in apply

    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)

  File "c:\obap\env4.1.1\lib\site-packages\django\db\migrations\operations\special.py", line 193, in database_forwards

    self.code(from_state.apps, schema_editor)

  File "c:\obap\env4.1.1\lib\site-packages\arches\app\models\migrations\deserialize_provisional_edits.py", line 13, in forwards_func

    for tile in tiles:

  File "c:\obap\env4.1.1\lib\site-packages\django\db\models\query.py", line 250, in __iter__

    self._fetch_all()

  File "c:\obap\env4.1.1\lib\site-packages\django\db\models\query.py", line 1118, in _fetch_all

    self._result_cache = list(self._iterable_class(self))

  File "c:\obap\env4.1.1\lib\site-packages\django\db\models\query.py", line 53, in __iter__

    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)

  File "c:\obap\env4.1.1\lib\site-packages\django\db\models\sql\compiler.py", line 899, in execute_sql

    raise original_exception

django.db.utils.DatabaseError: out of memory for query result

 

Thank you

 

Tom Shewring

GIS Systems Developer

IMT Development Centre

07748 182134

 

IMT Development and Programmes:  Together, we enable the celebration of England’s historic environment by empowering our customers and the public through technology and innovation.

[Alexei Peters]

Tom,

Did you set "DEBUG=False" in your settings.py file?

If you don't Django will keep a record of all you sql statements called.  That may be causing your out of memory error.

-Alexei

Director of Web Development - Farallon Geographics, Inc. - 971.227.3173

[Shewring, Tom]

Hi Alexei,

 

Yes I set "DEBUG=False" in settings.py file but python manage.py migrate still failed with a memory error.

 

Thanks

 

Tom Shewring

GIS Systems Developer

IMT Development Centre

07748 182134

 

IMT Development and Programmes:  Together, we enable the celebration of England’s historic environment by empowering our customers and the public through technology and innovation.

 

[Shewring, Tom]

Hi Cyrus,

 

Unfortunately when I tested the shortcut process detailed below I ran into errors in section 1 upgrade to version 4.3.1:

 

After running pip install arches==4.3.1 --upgrade --no-binary :all:  (success)

Trying to uninstall pycryptodome failed

 

pip uninstall pycryptodome, rdflib-jsonld

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

ERROR: Invalid requirement: 'pycryptodome,'

 

Installing pycryptodome works

pip install pycryptodome django-oauth-toolkit==1.1.2 PyLD[requests]==1.0.3 pyprind==2.11.2 (success)

 

but then py migrate fails

 

python manage.py migrate

Traceback (most recent call last):

  File "manage.py", line 29, in <module>

    execute_from_command_line(sys.argv)

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\__init__.py", line 364, in execute_from_command_line

    utility.execute()

  File "c:\obap\env4.1.1\lib\site-packages\django\core\management\__init__.py", line 338, in execute

    django.setup()

  File "c:\obap\env4.1.1\lib\site-packages\django\__init__.py", line 27, in setup

    apps.populate(settings.INSTALLED_APPS)

  File "c:\obap\env4.1.1\lib\site-packages\django\apps\registry.py", line 85, in populate

    app_config = AppConfig.create(entry)

  File "c:\obap\env4.1.1\lib\site-packages\django\apps\config.py", line 94, in create

    module = import_module(entry)

  File "c:\python27\Lib\importlib\__init__.py", line 37, in import_module

    __import__(name)

ImportError: No module named revproxy

 

 

I am thinking that it will be better to create a new clean 4.4.1 environment and recreate the application from there. Could you advise on the best process to follow if I take this approach.

 

Tom Shewring

GIS Systems Developer

IMT Development Centre

07748 182134

 

IMT Development and Programmes:  Together, we enable the celebration of England’s historic environment by empowering our customers and the public through technology and innovation.

 

From: Cyrus Hiatt [mailto:cyrus...@gmail.com]

Sent: 04 October 2019 01:28
To: Adam Cox
Cc: Alexei Peters; Shewring, Tom; Arches Project