What's the best way to keep Requirements up-to-date?

[Alex]

Hi all, 

Would love to hear your views on this. I run a Python app on GAE. I have quite a few packages in my requirements.txt file, and I made the choice to not version them. Which I understand means the latest version of each requirement at the time of the project build will be used. 

• Is that a crazy approach? I feel like I would not otherwise make the effort of tracking packages updates and changing versions numbers. 

It of course raises the issue of possible breaking change in a package update - and that happened to me already. 

• I'm looking for best practices on how to mitigate that. 

I'm using three different projects for my app: one is a dev environment, and I have a staging and production project. 

I would ideally like to keep requirements non-versioned in dev, but to freeze a version for staging and prod. So that I don't get into the situation where I may have validated the app in staging, then build to Prod with a significant package change. 

• What approach would you suggest? I'm looking for something I can automate - as I'm already using CloudBuild automation to build and deploy in staging and prod. 

thank you

[Elliott (Cloud Platform Support)]

Hello Alex,

From your description, you’ve decided not to version your requirements.txt in Python. This is not a crazy approach but there are alternatives. I’m assuming you’re using App Engine Flex. I was able to find a document here that describes that you do not need to specify a version but indicates why it would work in some cases. The following information that I found highly suggests using a virtual environment in Python.

It reads:

Specifying a version is not a requirement though it does help a lot in the future. Some versions of packages will not work well with other packages and their respective versions. It is hard to predict how changes in the future will affect these interrelationships. This is why it is very beneficial to create a snapshot in time (in your requirements.txt) showing which version interrelationships do work.

There is an approach for best practices in this thread regarding your concern of breaking change in a package update and explains virtual environments:

it's a good idea to get into the habit of using virtual environments to avoid dependency issues, especially when developing your own stuff. Anaconda offers a simple solution with the conda create command, and virtualenv works great with virtualenvwrapper for a lighter-weight solution. Another solution, pipenv, is quite popular.

There is further explanation:

A tip - you should aim to be using a virtual environment for each individual project that you'll be working on. This creates a 'bubble' for you to work within and to install specific package versions in, without it affecting your other projects. It will save you a lot of headaches in the future as your packages and versions will be kept project specific. 

There is a way in this thread on how to check if the packages in your requirements.txt are up to date. You may choose to act on them as described.

$ pip list --outdated

You have described how you are working with three projects to implement your non-versioning scenario on Google Cloud Platform but I am unclear about your question. Can you tell me more about how you are validating the application and building to production so I may research?

[Alex]

I'm on GAE standard but I don't think that makes a difference. The thread you share is interesting but goes into the direction of having versioned packages in the requirements file for even your dev environment. 

Trying to be more specific on the workflow I imagine, I would describe the following: 

1- continue to build and deploy my Dev project without versioned requirements

2- when I have a stabilized Dev project build, be able to gather the actual package version it was built with. How could I do that even manually?

3- build staging with a versioned requirements file - versions being those gathered at step 2

4- perform full QA on staging

5- build prod with a versioned requirements file - versions being those gathered at step 2 and those staging QA was performed with

So a key point seems to be about how can I perform step 2? 

[Andrew Cooper]

Managing dependencies can be tricky! You might want to check Pipenv as suggested above as it acts as a package and virtual environment management tool. It also uses a [Pipfile][1] and a [Pipfile.lock][2] rather than a requirments.txt file. The advantage of a Pipfile is it will be more accommodating semantically regarding how you declare your versions. For example, `pipenv install requests~=1.2` will install version 1.2 and any minor update, but not 2.0. You can also specify a hash to the 

```

$ pipenv install "requests>=1.4"   # install a version equal or larger than

$ pipenv install "requests<=2.13"  # install a version equal or lower

$ pipenv install "requests>2.19"   # install 2.19.1 but not 2.19.0

```

You can specify `pipenv install -r requirements.txt` to import a requirements file and create the Pipfile. If you want to upgrade all packages, you can just run `pipenv update`.

[1]: https://github.com/pypa/pipfile

[2]: https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock

[Alex]

Very interesting thank you!

[Andrew Cooper]

You're welcome. If you have any other questions let us know,