[Django] #31716: django-admin runserver mostly does not work on Windows
Django
---------------------------------------------+------------------------
Reporter: Christian Ullrich | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
I frequently encounter the problem that running "django-admin runserver"
on Windows fails:
{{{
<some-venv>\scripts\python.exe: can't open file '<some-venv>\test\Scripts
\django-admin': [Errno 2] No such file or directory
}}}
The command works if run with --noreload.
There are probably other conditions that must be met for it to fail, or
work. I *think* it makes a difference, for example, if Django was
installed with pip or setuptools. The .exe "console scripts " are
different between the two; one type has a bit of readable Python at the
end, the other does not.
What I do know is that the problem is either caused or enabled by the
generated console scripts (the .exe wrappers in venv/Scripts) stripping
the .exe extension from their own sys.argv[0] when passing that to Python
to run. If I sabotage this by changing the regex in the file itself (where
present), or by adding the extension again in utils/autoreload.py before
running the command line, it works fine.
To be honest, I have no idea whether this is a bug in Django, or pip, or
setuptools, or distutils, or something else entirely. I have spent the
last two days trying to figure out what exactly goes wrong, and where,
with little success. I'm reporting it here because Django is the only
place I could find where it is possible to kludge the code into working.
I'll be happy to take my business elsewhere if someone can point the way.
--
Ticket URL: <https://code.djangoproject.com/ticket/31716>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Uncategorized | Version: 3.0
Severity: Normal | Resolution: worksforme
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* status: new => closed
* resolution: => worksforme
Comment:
I can't reproduce an error here. venv (and virtualenv and ...) and pip
(and ...) all seem to work correctly. Unless you can provide a reproduce I
don't think there's anything we can say.
> <some-venv>\scripts\python.exe: can't open file '<some-
venv>\test\Scripts\django-admin': [Errno 2] No such file or directory
I would expect both `python` and `django-admin` to be in `<some-
venv>\Scripts`.
The `.exe.` generally isn't needed.
If your venv is active you can run `django-admin` directly, e.g. `>
django-admin help`
All I can think is to recommend checking that you're on up to date
versions and such, but day to day this works fine normally... Sorry I
can't say more.
You may need to see TicketClosingReasons/UseSupportChannels.
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:1>
Django
Reporter: Christian Ullrich | Owner: nobody
Type: Uncategorized | Status: new
Component: Uncategorized | Version: 3.0
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Christian Ullrich):
* status: closed => new
* resolution: worksforme =>
Comment:
Here you go.
{{{
C:\Daten>py -m venv test
C:\Daten>test\Scripts\activate
(test) C:\Daten>python -c "import sys; print(sys.version)"
3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit
(AMD64)]
(test) C:\Daten>python -m pip install -U pip setuptools Django
Looking in indexes: https://nexus/repository/PyPI-mirror/simple
Collecting pip
Downloading https://nexus/repository/PyPI-
mirror/packages/pip/20.1.1.post1/pip-20.1.1.post1-py2.py3-none-any.whl
(1.6MB)
|████████████████████████████████| 1.6MB 656kB/s
Collecting setuptools
Downloading https://nexus/repository/PyPI-
mirror/packages/setuptools/47.3.1/setuptools-47.3.1-py3-none-any.whl
(582kB)
|████████████████████████████████| 583kB 726kB/s
Collecting Django
Downloading https://nexus/repository/PyPI-
mirror/packages/django/3.0.7/Django-3.0.7-py3-none-any.whl (7.5MB)
|████████████████████████████████| 7.5MB 726kB/s
Collecting asgiref~=3.2 (from Django)
Downloading https://nexus/repository/PyPI-
mirror/packages/asgiref/3.2.9/asgiref-3.2.9-py3-none-any.whl
Collecting pytz (from Django)
Downloading https://nexus/repository/PyPI-
mirror/packages/pytz/2020.1/pytz-2020.1-py2.py3-none-any.whl (510kB)
|████████████████████████████████| 512kB 726kB/s
Collecting sqlparse>=0.2.2 (from Django)
Downloading https://nexus/repository/PyPI-
mirror/packages/sqlparse/0.3.1/sqlparse-0.3.1-py2.py3-none-any.whl (40kB)
|████████████████████████████████| 40kB 871kB/s
Installing collected packages: pip, setuptools, asgiref, pytz, sqlparse,
Django
Found existing installation: pip 19.2.3
Uninstalling pip-19.2.3:
Successfully uninstalled pip-19.2.3
Found existing installation: setuptools 41.2.0
Uninstalling setuptools-41.2.0:
Successfully uninstalled setuptools-41.2.0
Successfully installed Django-3.0.7 asgiref-3.2.9 pip-20.1.1.post1
pytz-2020.1 setuptools-47.3.1 sqlparse-0.3.1
(test) C:\Daten\dtest>pip list
Package Version
---------- ------------
asgiref 3.2.9
Django 3.0.7
pip 20.1.1.post1
pytz 2020.1
setuptools 47.3.1
sqlparse 0.3.1
}}}
pip 20.1.1.post1 is 20.1.1 minus the information disclosure. For our
purposes it's the same.
{{{
(test) C:\Daten>openssl.exe sha256 test\Scripts\django-admin.exe
SHA256(test\Scripts\django-admin.exe)=
24a24fefaaa27e069474985353cceacc9cb4128a35f50e99af487768d4255c90
}}}
No idea if they are identical between different installations.
{{{
(test) C:\Daten>django-admin startproject dtest
(test) C:\Daten>set PYTHONPATH=C:\Daten\dtest
(test) C:\Daten>set DJANGO_SETTINGS_MODULE=dtest.settings
(test) C:\Daten>django-admin
Type 'django-admin help <subcommand>' for help on a specific subcommand.
Available subcommands:
[django]
check
compilemessages
createcachetable
dbshell
diffsettings
dumpdata
flush
inspectdb
loaddata
makemessages
makemigrations
migrate
runserver
sendtestemail
shell
showmigrations
sqlflush
sqlmigrate
sqlsequencereset
squashmigrations
startapp
startproject
test
testserver
Note that only Django core commands are listed as settings are not
properly configured (error: No module named 'dtest').
(test) C:\Daten>cd dtest
(test) C:\Daten\dtest>django-admin runserver
C:\Program Files\Python38\python.exe: can't open file
'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory
}}}
Note the missing extension.
{{{
(test) C:\Daten\dtest>django-admin runserver --noreload
Performing system checks...
System check identified no issues (0 silenced).
You have 17 unapplied migration(s). Your project may not work properly
until you apply the migrations for app(s): admin, auth, contenttypes,
sessions.
Run 'python manage.py migrate' to apply them.
June 17, 2020 - 08:58:46
Django version 3.0.7, using settings 'dtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
^C
}}}
Works with --noreload.
Now let's see what it is actually trying to execute.
{{{
(test) C:\Daten\dtest>vi ..\test\Lib\site-
packages\django\utils\autoreload.py
}}}
Before line 230 (the subprocess.run call in restart_with_reloader()),
insert "print(args)".
{{{
(test) C:\Daten\dtest>django-admin runserver
['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts
\\django-admin', 'runserver']
C:\Program Files\Python38\python.exe: can't open file
'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory
}}}
This cannot possibly work.
{{{
(test) C:\Daten\dtest>dir ..\test\scripts
Volume in drive C is Windows
Volume Serial Number is 8EC9-7F66
Directory of C:\Daten\test\scripts
2020-06-17 08:54 <DIR> .
2020-06-17 08:54 <DIR> ..
2020-06-17 08:53 2.273 activate
2020-06-17 08:53 950 activate.bat
2020-06-17 08:53 18.454 Activate.ps1
2020-06-17 08:53 368 deactivate.bat
2020-06-17 08:54 106.388 django-admin.exe
2020-06-17 08:54 142 django-admin.py
2020-06-17 08:54 106.355 easy_install-3.8.exe
2020-06-17 08:54 106.355 easy_install.exe
2020-06-17 08:54 106.346 pip.exe
2020-06-17 08:54 106.346 pip3.8.exe
2020-06-17 08:54 106.346 pip3.exe
2020-06-17 08:53 532.040 python.exe
2020-06-17 08:53 531.016 pythonw.exe
2020-06-17 08:54 106.341 sqlformat.exe
2020-06-17 08:54 <DIR> __pycache__
14 File(s) 1.829.720 bytes
3 Dir(s) [...] bytes free
(test) C:\Daten\dtest>vi -b ..\test\scripts\django-admin.exe
}}}
Edit the Python block at the end to not match and remove .exe anymore. I
replaced "\.exe" with "\.rxe" in the regex.
{{{
(test) C:\Daten\dtest>django-admin runserver
['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts
\\django-admin.exe', 'runserver']
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
You have 17 unapplied migration(s). Your project may not work properly
until you apply the migrations for app(s): admin, auth, contenttypes,
sessions.
Run 'python manage.py migrate' to apply them.
June 17, 2020 - 09:07:47
Django version 3.0.7, using settings 'dtest.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
^C
}}}
With the extension left alone, this works now, too.
The .exe. generally isn't needed.
Not if it is the command name. Here it is an argument on python's command
line, though, and python does not guess about missing extensions.
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:2>
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* keywords: => Windows
* type: Uncategorized => Bug
* component: Uncategorized => Core (Management commands)
* stage: Unreviewed => Accepted
Comment:
OK, good yes, I see it. Thanks for the follow-up.
Looks like the auto-reloaded should indeed map back to an actually
existing file.
Fancy making a PR for that?
(Wondering if Windows would let us drop the exe extensions totally as we
did in POSIX-land...)
In the meantime, `manage.py` works as expected.
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:3>
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Christian Ullrich):
The best solution that I can come up with is adding this in the else
branch of get_child_arguments():
{{{
if sys.platform == 'win32':
for i in range(1, len(args)):
if args[i].lower().endswith("scripts\\django-admin"):
# setuptools style; setup.py develop or pip install -e
if os.path.exists(args[i] + "-script.py"):
args[i] = args[i] + "-script.py"
# pip style
elif os.path.exists(args[i] + ".exe"):
args[i] = args[i] + ".exe"
}}}
I think this has about as much chance of being accepted as would solving
the problem properly by dropping all Windows support, but if you think I
should make the attempt, please let me know.
The two branches exist because there are (at least) two types of these
.exe "scripts". If generated by pip install, they have ~100 KiB, contain a
small block of Python at the end, and can be run directly by the
interpreter ("python.exe django-admin.exe" works in that Python finds the
embedded script). However, if they are generated by setuptools, they do
not have the Python block, are only ~70 KiB, and can ''not'' be run as
scripts. Instead, there is the "...-script.py" companion script whose name
they derive from their own and that they run via sys.executable .
Then there is the case of "python -m django runserver", about whose
existence I just found out in get_child_arguments(), and whether that may
need special treatment, too.
Not to mention that the processes end up stacked five deep. Yes, it is a
dev situation so it doesn't matter much, but a better fix would still be
to reinvent autoreloading entirely, again, and ''somehow'' get by without
the monitoring process and $RUN_MAIN. (Getting rid of the bouncing around
between the venv and system pythons would be nice, too, but that's another
department.)
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:4>
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tom Forbes):
* cc: Tom Forbes (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:5>
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Carlton Gibson):
> I think this has about as much chance of being accepted as would solving
the problem properly by dropping all Windows support...
😀 We can't do that I'm afraid. (I'm just warming to it as it happens...)
I see Tom has cc-d himself. Let's see if he has any thoughts... ;)
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:6>
Django
Reporter: Christian Ullrich | Owner: nobody
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tom Forbes):
Replying to [comment:6 Carlton Gibson]:
> > I think this has about as much chance of being accepted as would
solving the problem properly by dropping all Windows support...
>
> 😀 We can't do that I'm afraid. (I'm just warming to it as it
happens...)
>
> I see Tom has cc-d himself. Let's see if he has any thoughts... ;)
My autoreloader-sense was tingling! I wonder if this has always been the
case - the previous implementation was pretty much the same, but I see
that Werkzeug has a partial workaround in their reloader:
https://github.com/pallets/werkzeug/blob/fc999a6262c847ad185ea3ffe0cc6f2e915a867a/src/werkzeug/_reloader.py#L69-L93
I think we can take inspiration (read: shamelessly copy parts) their
implementation as it seems to make sense, but I'd be interested to know
why they don't handle `-script.py` companions.
Lets be a bit generic here rather than hard-code anything to do with
`django-admin.py`. If `sys.argv[0]` doesn't exist we check if `-script.py`
exists, falling back to looking for `.exe` on Windows only. If we cannot
find it we explode with a reasonable error message (what that is I cannot
say!).
I'll work on a patch, and i would appreciate it if someone on Windows
could test it? :D
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:7>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: assigned
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tom Forbes):
* owner: nobody => Tom Forbes
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:8>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: assigned
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tom Forbes):
I think this might fix it:
https://github.com/orf/django/commit/cb200453065ef0dec7ed97cb09eab02bc99342c1
Christian, any chance you could test with my branch? `pip install
https://github.com/orf/django/archive/31716-got-99-problems-but-exe-aint-
one.zip`
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:9>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: assigned
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Christian Ullrich):
Yep, works; both with pip and setuptools .exes.
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:10>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: assigned
Component: Core (Management | Version: 3.0
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tom Forbes):
* has_patch: 0 => 1
Comment:
Thanks Christian!
PR: https://github.com/django/django/pull/13077
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:11>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: assigned
commands) |
Severity: Normal | Resolution:
Keywords: Windows | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Carlton Gibson):
* version: 3.0 => 3.1
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:12>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Component: Core (Management | Version: 3.1
commands) |
Keywords: Windows | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"8a902b7ee622ada258d15fb122092c1f02b82698" 8a902b7]:
{{{
#!CommitTicketReference repository=""
revision="8a902b7ee622ada258d15fb122092c1f02b82698"
Fixed #31716 -- Fixed detection of console scripts in autoreloader on
Windows.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:13>
Django
Reporter: Christian Ullrich | Owner: Tom
| Forbes
Type: Bug | Status: closed
Component: Core (Management | Version: 3.1
commands) |
Severity: Normal | Resolution: fixed
Keywords: Windows | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"ac7f7eab0fad18fffc465bc4fdc092bcb2de2d80" ac7f7ea]:
{{{
#!CommitTicketReference repository=""
revision="ac7f7eab0fad18fffc465bc4fdc092bcb2de2d80"
[3.1.x] Fixed #31716 -- Fixed detection of console scripts in autoreloader
on Windows.
Backport of 8a902b7ee622ada258d15fb122092c1f02b82698 from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/31716#comment:14>