UTF-8 name of a staticfiles test is breaking Django installation

[Marcin Nowak]

Hi all,

Here is a file which breaks installation on fresh Debian system:

 

https://github.com/django/django/blob/master/tests/staticfiles_tests/apps/test/static/test/%E2%8A%97.txt

I'm using Buildout and virtualenv with Python 2.7.9.

The full traceback is:

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/home/project/env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 2271, in main

    **kw

  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup

    dist.run_commands()

  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run

    self.easy_install(spec, not self.no_deps)

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 645, in easy_install

    return self.install_item(None, spec, tmpdir, deps, True)

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item

    dists = self.install_eggs(spec, download, tmpdir)

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 845, in install_eggs

    unpack_archive(dist_filename, tmpdir, self.unpack_progress)

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/archive_util.py", line 52, in unpack_archive

    driver(filename, extract_dir, progress_filter)

  File "/home/webcar/env/lib/python2.7/site-packages/setuptools/archive_util.py", line 148, in unpack_tarfile

    prelim_dst = os.path.join(extract_dir, *name.split('/'))

  File "/home/webcar/env/lib/python2.7/posixpath.py", line 80, in join

    path += '/' + b

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)

An error occurred when trying to install Django 1.8.14. Look above this message for any errors that were output by easy_install.

The path to a file which fails is:

/tmp/easy_install-A3XRR9 Django-1.8.14/tests/staticfiles_tests/apps/test/static/test/⊗.txt

Is this file really needed in Django dist?

Marcin

 

[Marcin Nowak]

On Tuesday, August 2, 2016 at 12:40:27 PM UTC+2, Marcin Nowak wrote:

I'm using Buildout and virtualenv with Python 2.7.9.

and setuptools 25.1.2 

M.

[Marcin Nowak]

I'm using Buildout and virtualenv with Python 2.7.9.

and setuptools 25.1.2 

It works with setuptools 17.0, but displays warning:

Getting distribution for 'Django==1.8.14'.

'tests/staticfiles_tests/apps/test/static/test/⊗.txt' not ANSI_X3.4-1968 encodable -- skipping

warning: no previously-included files matching '__pycache__' found under directory '*'

warning: no previously-included files matching '*.py[co]' found under directory '*'

'tests/staticfiles_tests/apps/test/static/test/⊗.txt' not ANSI_X3.4-1968 encodable -- skipping

Got Django 1.8.14.

 

M.

[Tim Graham]

A similar report was https://code.djangoproject.com/ticket/24761. I'm not sure if this is a bug in those install tools or if it's something we should try to fix in Django. You can use `git blame tests/staticfiles_tests/apps/test/static/test/⊗.txt` to find the commit where the file was introduced to learn the reason for it.

[Berker Peksağ]

On Tue, Aug 2, 2016 at 6:14 PM, Tim Graham <timog...@gmail.com> wrote:
> A similar report was https://code.djangoproject.com/ticket/24761. I'm not
> sure if this is a bug in those install tools or if it's something we should
> try to fix in Django. You can use `git blame
> tests/staticfiles_tests/apps/test/static/test/⊗.txt` to find the commit
> where the file was introduced to learn the reason for it.

It looks like this is a bug in setuptools:
https://github.com/pypa/setuptools/issues/709

--Berker

[Aymeric Augustin]

Hello,

Django used to create this file dynamically while running tests, but that got in the way of test parallelization. To avoid race conditions, the git checkout needs to be treated as readonly when running tests in parallel.

That’s why I switched to committing it to the git repository when I implemented test parallelization. I assumed all relevant systems for Django would support utf-8 by now… I was too optimistic!

Can we consider “update setuptools” to be an adequate resolution for this issue?

--
Aymeric.

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAF4280%2BXA9%3DKDAE7CKrgCjY4ttq%2BkxG5VdXZk2p6KdUn5k8VWQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[René Fleschenberg]

Hi everyone,

 

> Can we consider “update setuptools” to be an adequate resolution for this

> issue?

 

I realize this is probably not the most substantial argument, but I want to mention it anyway: Considering Django's popularity, requiring a recent version of setuptools might be a positive side-effect on the larger Python ecosystem. The prevalence of old setuptools versions can be rather annoying. For example: https://hynek.me/articles/conditional-python-dependencies/

 

--

René Fleschenberg