[Django] #36368: `makemessages` command runs `write_po_file()` multiple times for the same potfile

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Type: Bug
Status: new | Component: Core
| (Management commands)
Version: 5.2 | Severity: Normal
Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
This is both a Bug and a Cleanup/optimization issue.

In some cases, the `makemessages` management command will run
`write_po_file()` multiple times per the same potfile and locale.

1. It happens because inside `handle()`, the `write_po_file()` method is
called in a for loop for every `potfile`:
https://github.com/django/django/blob/main/django/core/management/commands/makemessages.py#L459

2. `potfiles` is a `list`, and that makes it possible for it to have
duplicates, which is exactly the case when the bug described here happens.
Inside `build_potfiles()`, used for `potfiles` inside `handle()`, inside a
for loop for every path in `self.locale_paths`, the returned list is
appended.
https://github.com/django/django/blob/main/django/core/management/commands/makemessages.py#L513

3. The `self.locale_paths` is also a list, and it also can have
duplicates, because e.g. when the `<project root>/locale` folder is
present, it is being appended twice:
https://github.com/django/django/blob/main/django/core/management/commands/makemessages.py#L384

4. Later, it is possible that inside the `find_files()`, the same locale
path is inserted to `self.locale_paths` once again, which makes it
possible to have 2 duplicates or 3 in total of the same locale path.

I propose the solution to change `self.locale_paths` from `list` to `set`,
and do the same for `potfiles` inside `handle()`, after doing that the
problems seems to be fixed. This also shouldn't break anything, because it
looks like this is how it is supposed to work, meaning ony one occurence
of each value.

If the issue is accepted I would be happy to be assigned to it and make a
PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/36368>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------

Reporter: Michał Pokusa | Owner: (none)
Type: Bug | Status: new
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:

Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Michał Pokusa):

* easy: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:1>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: (none)

Type: Bug | Status: closed

Component: Core (Management | Version: 5.2
commands) |

Severity: Normal | Resolution: needsinfo

Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* resolution: => needsinfo
* status: new => closed

Comment:

Hello Michał Pokusa, thank you for your ticket! I think I followed your
explanation, though I don't see how `self.locale_paths` could have
duplicates. If you could show in a Django sample project or in a unit test
how this scenario can be reached, I'll be happy to accept this ticket.

I'll close as `needsinfo` for now but please reopen when you can provide
further details (i.e. a real case scenario for having duplicates in the
path lists).
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:2>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: (none)
Type: Bug | Status: closed
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution: needsinfo
Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michał Pokusa):

Thank you for looking into it.

I prepared a very minimal project that will show the issue:
https://github.com/michalpokusa/django-test-project/tree/ticket-36368

Everything is default as from the `startproject` except the optional
`LOCALE_PATHS` setting and a `debug_makemessages` command, which is copy-
pasted `makemessages` command with some prints for clarity.

The bug is present also in `makemessages`, but it is not clearly visible,
that is why I added this command, but it is not necessary to reproduce.

Without setting `LOCALE_PATHS` in `settings`:
{{{
(.venv) [2025-05-06 00:44:31] michalpokusa@dev:~/projects/django-ticket-
xyz$ python manage.py debug_makemessages --locale en
settings.LOCALE_PATHS: []
locale_paths inside handle(): ['/home/michalpokusa/projects/django-
ticket-xyz/locale']
self.locale_paths at the end of find_files():
['/home/michalpokusa/projects/django-ticket-xyz/locale',
'/home/michalpokusa/projects/django-ticket-xyz/locale']
potfiles returned from build_potfiles(): ['/home/michalpokusa/projects
/django-ticket-xyz/locale/django.pot', '/home/michalpokusa/projects
/django-ticket-xyz/locale/django.pot']
processing locale en
calling write_po_file
calling write_po_file
}}}

With `LOCALE_PATHS` set to `[BASE_DIR / "locale"]`:
{{{
(.venv) [2025-05-06 00:48:44] michalpokusa@dev:~/projects/django-ticket-
xyz$ python manage.py debug_makemessages --locale en
settings.LOCALE_PATHS: [PosixPath('/home/michalpokusa/projects/django-
ticket-xyz/locale')]
locale_paths inside handle(): [PosixPath('/home/michalpokusa/projects
/django-ticket-xyz/locale'), '/home/michalpokusa/projects/django-ticket-
xyz/locale']
self.locale_paths at the end of find_files():
['/home/michalpokusa/projects/django-ticket-xyz/locale',
PosixPath('/home/michalpokusa/projects/django-ticket-xyz/locale'),
'/home/michalpokusa/projects/django-ticket-xyz/locale']
potfiles returned from build_potfiles(): ['/home/michalpokusa/projects
/django-ticket-xyz/locale/django.pot', '/home/michalpokusa/projects
/django-ticket-xyz/locale/django.pot', '/home/michalpokusa/projects
/django-ticket-xyz/locale/django.pot']
processing locale en
calling write_po_file
calling write_po_file
calling write_po_file
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:3>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: (none)

Type: Bug | Status: new

Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:

Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michał Pokusa):

* resolution: needsinfo =>
* status: closed => new

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:4>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: (none)
Type: Bug | Status: new
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Senthil Kumar):

Hey, I am new to the community and I would love to attempt to troubleshoot
and propose a solution. May i pick this up? Thanks
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:5>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: (none)
Type: Bug | Status: new
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michał Pokusa):

Replying to [comment:5 Senthil Kumar]:
> Hey, I'm new to the community and would love to try troubleshooting and
proposing a solution. May I take this on? Thanks

Hi, thank you for taking interest in this issue, although I think there is
nothing more to troubleshoot here, as both the direct cause and the
solution are determined.
Right now I am waiting with PR for the issue to be accepted and
information whether anything else should be included in it.
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:6>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------

Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned

Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage:
write_po_file | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 1 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Jacob Walls):

* owner: (none) => Michał Pokusa
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:7>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:

Keywords: makemessages | Triage Stage: Accepted
write_po_file |

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Natalia Bidart):

* cc: Claude Paroz (added)
* easy: 1 => 0
* stage: Unreviewed => Accepted

Comment:

Thank you Michał Pokusa for the test project. I have used that as starting
point, and I can confirm that there are two places where the files list to
be processed could have duplicates included. One is where you said:

{{{#!python
if self.settings_available:
self.locale_paths.extend(settings.LOCALE_PATHS)
}}}

But also here, and dupes can be seen when running for example
`i18n.test_extraction.BasicExtractorTests.test_valid_locale`
(`locale_paths` end up containing, for example,
`['/tmp/django__fn2ga0m/i18n_zmigwa3d/commands/locale',
'/tmp/django__fn2ga0m/i18n_zmigwa3d/commands/locale']`):
{{{#!python
elif dirname == "locale":
dirnames.remove(dirname)
self.locale_paths.insert(
0, os.path.join(os.path.abspath(dirpath), dirname)
)
}}}

I'm accepting with the goal that we handle duplicates in a way that does
not alter the order of the locale folders being processed. We need to
respect the order described in
[https://docs.djangoproject.com/en/5.2/topics/i18n/translation/#how-
django-discovers-translations these docs].
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:8>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michał Pokusa):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:9>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Claude Paroz):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:10>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Michał Pokusa):

I have prepared a PR with fix with respect to the order described in docs
and added appropriate tests.

Is there anything else that needs to be done on my part or should I now
wait for the review on Github?
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:11>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Replying to [comment:11 Michał Pokusa]:

> I have prepared a PR with fix with respect to the order described in
docs and added appropriate tests.

Thank you!

> Is there anything else that needs to be done on my part or should I now
wait for the review on Github?

If you consider the PR ready to be reviewed again, you should ideally
update this ticket's flag so the PR is listed as "needing review" in the
[https://code.djangoproject.com/query?status=!closed&needs_better_patch=0&needs_tests=0&needs_docs=0&has_patch=1&stage=Accepted&desc=1&order=changetime
Django Development Dashboard].
In this specific case, it means that you should unset the "needs test"
flag which was set by a previous reviewer. More docs here:
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code
/submitting-patches/#patch-review-checklist
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:12>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michał Pokusa):

* needs_tests: 1 => 0

Comment:

I have added tests that check for duplicate locale paths and
`write_po_file` calls.
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:13>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by kay):

I came across this bug just this week!

My fix (for the portion that compares `dirname`) would have looked almost
identical, except I named my variable `locale_dir_path` instead of
`locale_dir` to be extra verbose about its contents (vs. `dirname`'s) and
opted to only `insert` matches `not` already in `self.locale_paths` anyway
rather than removing and (re)inserting them.

Good thinking re: the potential for `LOCALE_PATHS` containing `Path`s
rather than strings, Michał, even if that part may have needed to go in a
separate ticket as slightly different (if related) concern.

Is there anything regular users can do to speed up contributions by
others?
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:14>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0

Needs tests: 1 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* needs_tests: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:15>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:
Keywords: makemessages | Triage Stage: Accepted
write_po_file |
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Michał Pokusa):

* needs_tests: 1 => 0

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:16>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa
Type: Bug | Status: assigned
Component: Core (Management | Version: 5.2
commands) |
Severity: Normal | Resolution:

Keywords: makemessages | Triage Stage: Ready for
write_po_file | checkin

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce):

* stage: Accepted => Ready for checkin

--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:17>

[Django]

#36368: `makemessages` command runs `write_po_file()` multiple times for the same
potfile
-------------------------------------+-------------------------------------
Reporter: Michał Pokusa | Owner: Michał
| Pokusa

Type: Bug | Status: closed

Component: Core (Management | Version: 5.2
commands) |

Severity: Normal | Resolution: fixed

Keywords: makemessages | Triage Stage: Ready for
write_po_file | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Sarah Boyce <42296566+sarahboyce@…>):

* resolution: => fixed
* status: assigned => closed

Comment:

In [changeset:"2c99fbcf95f07eaff0bf9673af7c82d07353df8f" 2c99fbc]:
{{{#!CommitTicketReference repository=""
revision="2c99fbcf95f07eaff0bf9673af7c82d07353df8f"
Fixed #36368 -- Prevented duplicate locale paths and write_po_file calls
in makemessages.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36368#comment:18>