[Django] #33253: Collectstatic fails when using ManifestStaticFilesStorage and specific Javascript file
Django
Javascript file
-------------------------------------+-------------------------------------
Reporter: Hervé Le | Owner: nobody
Roy |
Type: Bug | Status: new
Component: | Version: 4.0
contrib.staticfiles | Keywords:
Severity: Normal | ManifestStaticFilesStorage
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
{{{
python manage.py collectstatic --no-input
Post-processing 'rest_framework/docs/js/highlight.pack.js' failed!
}}}
'''Steps to reproduce the issue'''
Start a new project
{{{django-admin startproject mysite}}}
Install the following packages
{{{
Django==4.0b1
djangorestframework~=3.12
}}}
Add {{{rest_framework}}} to installed apps
Update {{{settings.py}}} to use ManifestStaticFilesStorage
{{{
STATIC_ROOT = BASE_DIR / 'static'
STATICFILES_STORAGE =
'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
}}}
Run collecstatic
{{{
python manage.py collectstatic --no-input
Post-processing 'rest_framework/docs/js/highlight.pack.js' failed!
Traceback (most recent call last):
File "/home/Devs/Sandbox/django4/mysite/manage.py", line 22, in <module>
main()
File "/home/Devs/Sandbox/django4/mysite/manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/core/management/__init__.py", line 425, in
execute_from_command_line
utility.execute()
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/core/management/__init__.py", line 419, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/core/management/base.py", line 373, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/core/management/base.py", line 417, in execute
output = self.handle(*args, **options)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 187, in handle
collected = self.collect()
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/management/commands/collectstatic.py",
line 134, in collect
raise processed
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/storage.py", line 321, in
_post_process
content = pattern.sub(converter, content)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/storage.py", line 207, in converter
hashed_url = self._url(
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/storage.py", line 144, in _url
hashed_name = hashed_name_func(*args)
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/storage.py", line 371, in _stored_name
cache_name = self.clean_name(self.hashed_name(name))
File "/home/Devs/Sandbox/django4/venv/lib/python3.9/site-
packages/django/contrib/staticfiles/storage.py", line 106, in hashed_name
raise ValueError("The file '%s' could not be found with %r." %
(filename, self))
ValueError: The file 'rest_framework/docs/js/,constant:' could not be
found with <django.contrib.staticfiles.storage.ManifestStaticFilesStorage
object at 0x7f3aa1b7ebe0>.
}}}
It used to work with previous versions of ManifestStaticFilesStorage.
Maybe a regression with the recent additions of ES Module support and
Javascript source map support ?
--
Ticket URL: <https://code.djangoproject.com/ticket/33253>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Javascript file
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.staticfiles | Version: 4.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
Tentatively marking as accepted, and indeed relates to ESM support via
ticket #32319 (commit 91e21836f667c784a8a63ab1f18d81f553e679cb)
Looks like it's this regex:
{{{
r"""(?P<matched>import\s+(?s:(?P<imports>.*?))\s*from\s*["'](?P<url>.*?)["'])"""
}}}
which is being used like so:
{{{
(Pdb) matchobj
<re.Match object; span=(15745, 125465), match='import from
as",c:[e.ASM,e.QSM]},{cN:"class",bK:">
(Pdb) matchobj.groupdict().keys()
dict_keys(['matched', 'imports', 'url'])
(Pdb) len(matches['matched'])
109720
(Pdb) matches['url']
',constant:'
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:1>
Django
Javascript file
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* severity: Normal => Release blocker
Comment:
Better to be safe than sorry, worst that can happen is that these states
are reverted.
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:2>
Django
Javascript file
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: gilmrjc (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:3>
Django
Javascript file
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Hervé Le Roy):
The following regex restricts characters for identifiers:
{{{
<matched>import\s+(?:(?P<imports>[\*\{\}\,\w
]*?))\s*from\s*[\"'](?P<url>.*?)[\"'])
}}}
It works with several import syntaxes: [https://regex101.com/r/oH8NY5/1
Regex101]
It doesn't fail with the example provided in the ticket
(highlight.pack.js).
Probably needs further testing and feedback.
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:4>
Django
Javascript file
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Hervé Le Roy):
For both import and export:
{{{
r"""(?P<matched>import\s+(?:(?P<imports>[\*\{\}\,\w
]*?))\s*from\s*[\"'](?P<url>.*?)[\"'])"""
r"""(?P<matched>export\s+(?:(?P<exports>[\*\{\}\,\w
]*?))\s*from\s*[\"'](?P<url>.*?)[\"'])"""
}}}
I have also:
- removed the 's' in (?s: (Regex101 was reporting it as incorrect in
Python regex syntax, and, anyway, Javascript import statement can be
multi-line)
- escaped double-quotes
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:5>
Django
Javascript file
-------------------------------------+-------------------------------------
| Gibson
Type: Bug | Status: assigned
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Carlton Gibson
* status: new => assigned
* has_patch: 0 => 1
Comment:
I've opened a [https://github.com/django/django/pull/15058 draft PR] for
this that tightens the regex to ≈valid identifiers plus `{},\s ` due to
appearing in the `import`/`export`.
I'm sure it can be improved. Comments welcome there.
> removed the 's' in (?s: (Regex101 was reporting it as incorrect in
Python regex syntax, and, anyway, Javascript import statement can be
multi-line
`(?s)` is [https://docs.python.org/3.10/library/re.html#re.DOTALL
`re.DOTALL`] allowing `.` to match newlines, which matters for the multi-
line case.
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:6>
Django
Javascript file
-------------------------------------+-------------------------------------
| Gibson
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"ba9ced3e9a643a05bc521f0a2e6d02e3569de374" ba9ced3e]:
{{{
#!CommitTicketReference repository=""
revision="ba9ced3e9a643a05bc521f0a2e6d02e3569de374"
Fixed #33253 -- Reverted "Fixed #32319 -- Added ES module support to
ManifestStaticFilesStorage."
This reverts commit 91e21836f667c784a8a63ab1f18d81f553e679cb.
`export` and `import` directives have several syntax variants and not
all of them were properly covered.
Thanks Hervé Le Roy for the report.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:7>
Django
Javascript file
-------------------------------------+-------------------------------------
| Gibson
Type: Bug | Status: closed
Component: contrib.staticfiles | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: | Triage Stage: Accepted
ManifestStaticFilesStorage |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"b7b3bbc8356b8365f3ab8ed9c8d863b47c18a3d4" b7b3bbc8]:
{{{
#!CommitTicketReference repository=""
revision="b7b3bbc8356b8365f3ab8ed9c8d863b47c18a3d4"
[4.0.x] Fixed #33253 -- Reverted "Fixed #32319 -- Added ES module support
to ManifestStaticFilesStorage."
This reverts commit 91e21836f667c784a8a63ab1f18d81f553e679cb.
`export` and `import` directives have several syntax variants and not
all of them were properly covered.
Thanks Hervé Le Roy for the report.
Backport of ba9ced3e9a643a05bc521f0a2e6d02e3569de374 from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33253#comment:8>