[Proposal] Preserve TJ text arrays when regenerating PDF page content instead of flatting to Tj
Amit Pathak
Hi PDFium Team,
I have been working with PDFium's page content editing and regeneration functionality and would like to propose an enhancement related to the TJ text-showing operator.
PDFium supports parsing and rendering both Tj and TJ. However, when page contents are regenerated after editing, text originally represented using TJ arrays is serialized using Tj, effectively flattening the positioning adjustments present in the TJ array.
I have implemented support for generating TJ arrays during page content regeneration, preserving relevant glyph positioning and spacing adjustments where appropriate.
I would be interested in contributing this enhancement upstream. Before preparing a focused CL with tests, I would appreciate feedback on whether this functionality aligns with PDFium's design goals and whether there is a preferred approach for implementing it in the page content generation path.
Thanks
Amit Kumar Pathak
Lei Zhang
Are you using the latest PDFium? How is your proposal different from
this commit? [1]
[1] https://pdfium.googlesource.com/pdfium/+/05e4858256a8b18e3c3a550d1b8b1a293f5232a2
> You received this message because you are subscribed to the Google Groups "pdfium" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pdfium+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/pdfium/d97eeaad-38db-4fbc-a161-9bdd09897712n%40googlegroups.com.
Amit Pathak
HI Lei,
Thanks for pointing me to this change.
I compared it with my implementation and realized that my PDFium branch was based on an older revision that predates this change.
There is one difference in the approach, though. My implementation tracks whether the original text object was represented using TJ (HasTJ()) and reconstructs it during content regeneration (RebuildTJStream()). Text originally represented using Tj continues to be generated as Tj.
So the behavior is essentially:
Original Tj regenerated as Tj
Original TJ regenerated as TJ
From my understanding, the referenced upstream change generates TJ based on the character positioning/kerning information and does not attempt to preserve the original Tj vs TJ operator choice.
My original goal was therefore slightly different: preserve the original text-showing operator representation during page content regeneration, in addition to retaining the positioning information for TJ.
Would preserving the original Tj/TJ representation be useful from PDFium's perspective or is canonicalizing regenerated text to TJ intentional and by Design??
Thanks
Amit Kumar Pathak
Lei Zhang
more, it seemed easier to just use TJ everywhere. Do you see any
significant advantages to preserving Tj?
Amit Pathak
Potential advantages of preserving Tj where possible are:
1. Minimal serialization changes: If a text object was originally "Tj" & doesn't require kernings, regenerating it as "Tj" avoids an unnecessary representation change.
2. Cleaner round-trip editing: Localized edits produce smaller and more focused content stream diffs instead of normalizing every text-showing operator to "TJ".
3. Slightly smaller content streams: "Tj" is a more compact representation for simple contiguous text. While compression reduces the impact, avoiding unnecessary "TJ" arrays still eliminates some serialization overhead.
4. "Representation fidelity: Since both "Tj" & "TJ" are valid PDF operators, preserving the original operator where possible retains more of the original content stream structure instead of canonicalizing it.
That said, I completely understand the trade-off. If PDFium intentionally prefers a single canonical serialization path for simplicity and maintainability, then using "TJ" everywhere is a perfectly reasonable design choice.
Lei Zhang
"dirty" bit set to indicate they need to be rewritten. Those content
streams are completely rewritten with no attempt to preserve the
original content stream. Given this behavior, trying to preserve the
original Tj operators for reasons 1, 2 and 4 is a lost cause.
That leaves reason 3. If you feel strongly about it and it is not too
much work, then feel free to submit a Changelist for that.
Amit Pathak
Amit Pathak
Hi Lei
The CL is ready locally and all relevant unit tests are passing (10/10). However, I am currently unable to upload it to Gerrit because Chromium's Gerrit ReAuth is requiring a physical FIDO security key, which I don't currently have.
please suggest how you'd like me to proceed?
Lei Zhang
requirements. Maybe https://groups.google.com/g/repo-discuss can help?
geisserml
I think it unlikely that such requirements were imposed, as that would be impedimental to many contributors.
Amit Pathak
Thanks. That's helpful to know.
In my case, git cl creds-check successfully configured the LUCI OAuth helper & git credential-luci login completes successfully. However, git credential-luci reauth returns NO_AVAILABLE_CHALLENGES, while git cl upload still returns 401 Unauthorized.
I am not sure how to proceed from here. Do you have any suggestions on what I should try next?
geisserml
Amit Pathak
That would be really helpful, thank you! If you could share the steps &output from a fresh authentication/upload attempt, I can compare them with my setup & hopefully identify where mine is going wrong.
geisserml
```bash
$ mkdir ~/tmp/pdfium_cl_test
$ cd ~/tmp/pdfium_cl_test/
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools
Cloning into 'depot_tools'...
remote: Sending approximately 71.47 MiB ...
remote: Counting objects: 42, done
remote: Finding sources: 100% (42/42)
remote: Total 70520 (delta 48718), reused 70506 (delta 48718)
Receiving objects: 100% (70520/70520), 70.90 MiB | 14.53 MiB/s, done.
Resolving deltas: 100% (48718/48718), done.
$ export PATH="${PWD}/depot_tools:${PATH}"
$ gclient config --custom-var checkout_configuration=minimal --unmanaged https://pdfium.googlesource.com/pdfium
gclient sync --no-history --shallow
$ gclient sync --no-history --shallow
Syncing projects: 100% (41/41), done.
Running hooks: 58% ( 7/12) test_fonts
________ running 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --extract --bucket chromium-fonts -s third_party/test_fonts/test_fonts.tar.gz.sha1' in '/home/manuel/tmp/pdfium_cl_test/pdfium'
0> Downloading third_party/test_fonts/test_fonts.tar.gz@cd96fc55dc243f6c6f4cb63ad117cad6cd48dceb...
0> Extracting 33 entries from third_party/test_fonts/test_fonts.tar.gz to third_party/test_fonts/test_fonts
Downloading 1 files took 50.171533 second(s)
Hook 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --extract --bucket chromium-fonts -s third_party/test_fonts/test_fonts.tar.gz.sha1' took 51.66 secs
Running hooks: 100% (12/12), done.
$ cd pdfium/
$ git config --global depot-tools.useNewAuthStack 1
$ git cl creds-check
This tool will help check your Gerrit authentication.
(Report any issues to https://issues.chromium.org/issues/new?component=1456702&template=2076315)
You don't have a cookie file configured in Git (good).
Checking for SSO helper...
Looks like we are running inside a Gerrit repository,
so we will check your Git configuration for it.
Your global Git email is: geis...@gmail.com
Configuring Gerrit auth for /home/manuel/tmp/pdfium_cl_test/pdfium
Repo remote is https://pdfium.googlesource.com/pdfium
Since you don't have a different local email, we'll set up auth in your global config
alongside your globally configured email.
>>> In your global Git config, we set credential.https://pdfium.googlesource.com.helper='', replacing any existing values
>>> In your global Git config, we appended credential.https://pdfium.googlesource.com.helper='luci' to existing values
>>> In your global Git config, we set credential.https://pdfium.googlesource.com.useHttpPath='yes', replacing any existing values
>>> In your global Git config, we cleared all values for url.sso://pdfium/.insteadOf
>>> In your local Git config, we cleared all values for url.https://pdfium.googlesource.com/pdfium.insteadOf
>>> In your local Git config, we cleared all values for url.sso://pdfium/.insteadOf
We have configured Git to use an OAuth helper.
The OAuth helper requires its own login.
!!! If you haven't yet, run `git credential-luci login` using the same email as Git.
(If you have already done this, you don't need to do it again.)
(However, if you changed your email, you should do this again
to ensure you're using the right account.)
Successfully finished auth configuration check.
However, there are some manual actions that are suggested
(you don't have to re-run this command afterward):
- If you haven't yet, run `git credential-luci login` using the same email as Git.
$ git credential-luci logout
$ git credential-luci login
Getting a refresh token with following OAuth scopes:
* https://www.googleapis.com/auth/accounts.reauth
* https://www.googleapis.com/auth/gerritcodereview
* https://www.googleapis.com/auth/userinfo.email
Visit this link to complete the login flow in the browser. Do not share it with anyone!
https://ci.chromium.org/cli/login/<REDACTED>
When asked, use this confirmation code (it refreshes with time):
<REDACTED>
─────────────────────────
Waiting for the login flow to complete in the browser. Done!
Warning: Login successful, but ReAuth failed: renew RAPT: GetRAPT: reauth rejected with "NO_AVAILABLE_CHALLENGES"
If you're asked for ReAuth later, run `git-credential-luci reauth` to retry.
For context, see: https://chromium.googlesource.com/chromium/src.git/+/HEAD/docs/gerrit_reauth.md
$ git checkout -b test_cl -t origin/main
branch 'test_cl' set up to track 'origin/main'.
Switched to a new branch 'test_cl'
# Then I wrote the TEST file and committed the change using a git GUI
$ git cl creds-check
This tool will help check your Gerrit authentication.
(Report any issues to https://issues.chromium.org/issues/new?component=1456702&template=2076315)
You don't have a cookie file configured in Git (good).
Checking for SSO helper...
Looks like we are running inside a Gerrit repository,
so we will check your Git configuration for it.
Your global Git email is: geis...@gmail.com
Configuring Gerrit auth for /home/manuel/tmp/pdfium_cl_test/pdfium
Repo remote is https://pdfium.googlesource.com/pdfium
Since you don't have a different local email, we'll set up auth in your global config
alongside your globally configured email.
>>> In your global Git config, we set credential.https://pdfium.googlesource.com.helper='', replacing any existing values
>>> In your global Git config, we appended credential.https://pdfium.googlesource.com.helper='luci' to existing values
>>> In your global Git config, we set credential.https://pdfium.googlesource.com.useHttpPath='yes', replacing any existing values
>>> In your global Git config, we cleared all values for url.sso://pdfium/.insteadOf
>>> In your local Git config, we cleared all values for url.https://pdfium.googlesource.com/pdfium.insteadOf
>>> In your local Git config, we cleared all values for url.sso://pdfium/.insteadOf
We have configured Git to use an OAuth helper.
The OAuth helper requires its own login.
!!! If you haven't yet, run `git credential-luci login` using the same email as Git.
(If you have already done this, you don't need to do it again.)
(However, if you changed your email, you should do this again
to ensure you're using the right account.)
Successfully finished auth configuration check.
However, there are some manual actions that are suggested
(you don't have to re-run this command afterward):
- If you haven't yet, run `git credential-luci login` using the same email as Git.
$ git cl upload
Found change with 1 commit...
Running presubmit upload checks on branch test_cl ...
checking owners took a long time: 2.8s
** 0 Presubmit Messages, 0 Presubmit Warnings, 0 Presubmit ERRORS **
Presubmit checks took 4.5s to calculate.
presubmit checks passed.
Waiting for editor...
saving CL description to /home/manuel/tmp/pdfium_cl_test/depot_tools/.git_cl_description_backup
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 407 bytes | 407.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: refs: 1, new: 1, done
remote:
remote: SUCCESS
remote:
remote: https://pdfium-review.googlesource.com/c/pdfium/+/153730 Test CL [NEW]
remote:
To https://pdfium.googlesource.com/pdfium
* [new reference] 3215c74da9b641e12bf2d8519633f2402cd91078 -> refs/for/refs/heads/main%m=Initial_upload
geisserml
https://chromium.googlesource.com/chromium/src/+/3e5b5423b5caa3fb969e995666834e30c67b0e87/docs/gerrit_guide.md#everyone_to-get-access-to-the-chromium-gerrit-instance
https://chromium.googlesource.com/chromium/src/+/3e5b5423b5caa3fb969e995666834e30c67b0e87/docs/contributing.md#creating-a-change
Amit Pathak
Thank you! That was very helpful. I was able to track down the issue, sign the CLA & successfully uploaded my first PDFium CL.
If you have a chance, I'd appreciate it if you could take a look and review it.
https://pdfium-review.googlesource.com/c/pdfium/+/153750
geisserml
Unfortunately I'm not in a position to review anything, unless it happens to be Python (I merely use PDFium's public API via ctypes FFI, and know practically nothing about C++), but I take it PDFium team will take care of reviewing your CL :)
However, I'd be intrigued if you could tell what was the culprit here? Might be helpful for anyone else running into similar problems...
Amit Pathak
Thanks again,It turned out to be two separate issues.
First, I hadn't created a Gerrit account/signed the Google Individual CLA, so the upload was rejected with a 403 ("No Contributor Agreement on file").
Earlier, I was also hitting a 401 during git cl upload, which I initially thought was a Gerrit authentication problem but I had accidentally deleted depot_tools, so git cl wasn't working correctly. After restoring/updating depot_tools, creating my Gerrit account & signing the CLA, the upload succeeded.
Hopefully this helps anyone who runs into something similar!