[Django] #33002: Make DebugLexer.tokenize() more similar to Lexer.tokenize()
Django
-------------------------------------+-------------------------------------
Reporter: Chris | Owner: Chris Jerdonek
Jerdonek |
Type: | Status: assigned
Cleanup/optimization |
Component: Template | Version: dev
system |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Currently,
[https://github.com/django/django/blob/8208381ba6a3d1613bb746617062ccf1a6a28591/django/template/base.py#L396-L418
DebugLexer.tokenize()] looks somewhat dissimilar to
[https://github.com/django/django/blob/8208381ba6a3d1613bb746617062ccf1a6a28591/django/template/base.py#L352-L364
Lexer.tokenize()], even though they do more or less the same thing. For
example, the `Lexer` class's implementation contains just one call to
`self.create_token()`, whereas `DebugLexer` contains three calls to it.
This ticket is to make `DebugLexer.tokenize()` look visibly much more the
same as `Lexer.tokenize()`, and in particular simpler. One advantage is
that it will be more obvious on inspection that `DebugLexer` acts the same
as `Lexer`. Another advantage is that it will be easier to keep the two
implementations in step when applying optimizations to `Lexer`. It will be
more maintainable and less likely to introduce bugs if the implementations
for the two lexers don't diverge too much.
The idea will become more clear once the PR is posted.
--
Ticket URL: <https://code.djangoproject.com/ticket/33002>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
PR: https://github.com/django/django/pull/14753
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:1>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:2>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:3>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:4>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:5>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"b2be7e12cce8e62eeee512fbea4a32503d048120" b2be7e1]:
{{{
#!CommitTicketReference repository=""
revision="b2be7e12cce8e62eeee512fbea4a32503d048120"
Refs #33002 -- Made template_tests.tests.py's tests test both Lexer and
DebugLexer.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:6>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Component: Template system | Version: dev
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"f0776a558c4bee8524aa698c8dbe36790918152b" f0776a55]:
{{{
#!CommitTicketReference repository=""
revision="f0776a558c4bee8524aa698c8dbe36790918152b"
Fixed #33002 -- Made DebugLexer.tokenize() more closely parallel
Lexer.tokenize().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:10>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Component: Template system | Version: dev
Severity: Normal | Resolution:
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"6fedd868e1d50421204ea0c05cc8d2200dcb3cb0" 6fedd868]:
{{{
#!CommitTicketReference repository=""
revision="6fedd868e1d50421204ea0c05cc8d2200dcb3cb0"
Refs #33002 -- Renamed variable from bit to token_string in
Lexer.tokenize().
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:9>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"65ed96fa394f3d9d548c6d9d8144bbb51c4c68ff" 65ed96fa]:
{{{
#!CommitTicketReference repository=""
revision="65ed96fa394f3d9d548c6d9d8144bbb51c4c68ff"
Refs #33002 -- Added DebugLexer/Lexer.tokenize() test for trailing text
token.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:7>
Django
-------------------------------------+-------------------------------------
Type: | Jerdonek
Cleanup/optimization | Status: assigned
Component: Template system | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"6242c22a2f062f1b55f8a612b75063966937bc5d" 6242c22]:
{{{
#!CommitTicketReference repository=""
revision="6242c22a2f062f1b55f8a612b75063966937bc5d"
Refs #33002 -- Optimized Lexer.tokenize() by skipping computing lineno
when not needed.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/33002#comment:8>