Python source files missing from sandstorm-files.list

[Troy Farrell]

Hello Sandstormers!

I'm packaging a few Python apps for Sandstorm and I've noticed a problem with my custom builds of Python not running.  I can't say exactly when this started.

According to Python's PEP 3147, Python expects to find the source file (example.py) before it uses the cached bytecode file (__pycache__/example.cpython-3.9.pyc).  I suspect that Sandstorm's union filesystem implementation is not adding the source files to the list of used files because (my guess is that) they are stat()ed instead of read().   Because the source files are not in sandstorm-files.list, they are not included in the Sandstorm package.  Because they are not in the Sandstorm package, according to the flowchart, Python does not find them and fails to find the cached bytecode files that are in the Sandstorm package.

https://www.python.org/dev/peps/pep-3147/#flow-chart
https://stackoverflow.com/a/53945327

I've written a short Python script which adds the source files to the sandstorm-files.lists files.  It seems to fix the problem.  I'd like to hear opinions on the best way to fix this.  Do add some special logic to the union filesystem to add the Python source files to sandstorm-files.list?  Do we add a post-processing step?  I'm open to suggestions as this affects three of my current projects.

Thanks,

Troy

For the sake of making this error easier to find in the future, here is the most common error message Python gives when the source files are missing.  You can see this in the app console:

Could not find platform independent libraries <prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/opt/app/berkut-django-app'
  program name = '/opt/app-venv/bin/python3.9'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/opt/app-venv/bin/python3.9'
  sys.base_prefix = '/opt/python3.9'
  sys.base_exec_prefix = '/opt/python3.9'
  sys.platlibdir = 'lib'
  sys.executable = '/opt/app-venv/bin/python3.9'
  sys.prefix = '/opt/python3.9'
  sys.exec_prefix = '/opt/python3.9'
  sys.path = [
    '/opt/app/berkut-django-app',
    '/opt/python3.9/lib/python39.zip',
    '/opt/python3.9/lib/python3.9',
    '/opt/python3.9/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: no codec search functions registered: can't find encoding

Current thread 0x00007f4d5c39e700 (most recent call first):
<no Python frame>

[Jacob Weisz]

Standard recommendation is to use the alwaysInclude feature in the pkgdef if there are files a package needs that don't always get included in testing. It may be your preference to alwaysInclude the project directory entirely if you know it doesn't have too much unnecessary clutter.

--

  Jacob Weisz

  goo...@jacobweisz.com

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/5c9bdda2-5893-456b-8fe4-9b94041117d0n%40googlegroups.com.

[Troy Farrell]

On Tuesday, August 3, 2021 at 8:58:09 PM UTC+6 Jacob Weisz wrote:

Standard recommendation is to use the alwaysInclude feature in the pkgdef if there are files a package needs that don't always get included in testing. It may be your preference to alwaysInclude the project directory entirely if you know it doesn't have too much unnecessary clutter.

I'd forgotten alwaysInclude.  Thanks for mentioning that.  I will note, however, that when this problem first appeared, it was because /usr/local/lib/python3.9/encodings/__init__.py was missing.  That was installed when a custom build of Python 3.9 was created in setup.sh.  alwaysInclude will help with missing app files, but it seems that most Python apps are likely to need whatever workaround we devise.