Bug in RBTools 1.0.4 with git?
109 views
Skip to first unread message
EM W
Aug 28, 2020, 5:29:52 PM8/28/20
to Review Board Community
I am currently using RB 3.0.18. I've configured our instance, which runs in Docker, with a number of Git repos.
The configuration uses ssh; so entries look like this:
ssh://gitoliteuser@gitserver/mygitrepo.git
We link to our own internal gitweb instance, and entries for the raw url file mask look like this:
http://gitserver/gitweb/mygitrepo.git;a=blob_plain;f=<filename>;h=<revision>
Initial access to the repo, and linking reviews for new code works just fine:
rbt post -d a23476f834ef86d8f9549f3c35b258422911421fe
...
...
Review request #5 posted.
However, if I edit a file that already exists in my Git branch, commit, and then try to submit code for review, I get an Error 207 (File not found):
rbt post -d a08f68f834ef86d8f9549fa3eb258422911421fe
>>> RBTools 1.0.3
>>> Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0]
>>> Running on Linux-5.4.0-0.bpo.4-amd64-x86_64-with-debian-10.0
>>> Home = /home/myuser
>>> Current directory = /home/myuser/src/mygitrepo
>>> Command line: rbt post -d a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Running: tf vc help
>>> Checking for a Git repository...
>>> Running: git rev-parse --git-dir
>>> Running: git config core.bare
>>> Running: git rev-parse --show-toplevel
>>> Running: git symbolic-ref -q HEAD
>>> Running: git config --get branch.master.merge
>>> Running: git config --get branch.master.remote
>>> Running: git config --get remote.origin.url
>>> Repository info: Path: git:mygitrepo, Base path: , Supports changesets: False
>>> Making HTTP GET request to http://reviewboard.oursite.com/api/
>>> Running: git rev-parse a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Running: git rev-parse a08f68f834ef86d8f9549fa3eb258422911421fe^
>>> Running: git config --get branch.master.merge
>>> Running: git config --get branch.master.remote
>>> Running: git branch --remotes
>>> Running: git rev-list e66800919f5d3dfba8a6416ad393794485aa2b9c --not --remotes=origin
>>> Running: git version
>>> Running: git -c core.quotepath=false -c diff.noprefix=false diff --no-color --full-index --ignore-submodules -M --no-ext-diff e66800919f5d3dfba8a6416ad393794485aa2b9c..a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Generated diff size: 310 bytes
>>> Making HTTP GET request to http://reviewboard.oursite.com/api/validation/diffs/
>>> Cached response for HTTP GET http://reviewboard.oursite.com/api/validation/diffs/ expired and was modified
>>> Making HTTP POST request to http://reviewboard.oursite.com/api/validation/diffs/
>>> Got API Error 207 (HTTP code 400): The file was not found in the repository.
>>> Error data: {u'stat': u'fail', u'file': u'test.foo', u'err': {u'msg': u'The file was not found in the repository.', u'code': 207}, u'revision': u'b646fadbca709a770dfcca95cadaeb50aac92627'}
Traceback (most recent call last):
File "/usr/local/bin/rbt", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/main.py", line 120, in main
command.run_from_argv([RB_MAIN, command_name] + args)
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/__init__.py", line 725, in run_from_argv
exit_code = self.main(*args) or 0
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/post.py", line 874, in main
(msg_prefix, e))
rbtools.commands.CommandError: Error validating diff
test.foo: The file was not found in the repository. (HTTP 400, API Error 207)
>>> RBTools 1.0.3
>>> Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0]
>>> Running on Linux-5.4.0-0.bpo.4-amd64-x86_64-with-debian-10.0
>>> Home = /home/myuser
>>> Current directory = /home/myuser/src/mygitrepo
>>> Command line: rbt post -d a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Running: tf vc help
>>> Checking for a Git repository...
>>> Running: git rev-parse --git-dir
>>> Running: git config core.bare
>>> Running: git rev-parse --show-toplevel
>>> Running: git symbolic-ref -q HEAD
>>> Running: git config --get branch.master.merge
>>> Running: git config --get branch.master.remote
>>> Running: git config --get remote.origin.url
>>> Repository info: Path: git:mygitrepo, Base path: , Supports changesets: False
>>> Making HTTP GET request to http://reviewboard.oursite.com/api/
>>> Running: git rev-parse a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Running: git rev-parse a08f68f834ef86d8f9549fa3eb258422911421fe^
>>> Running: git config --get branch.master.merge
>>> Running: git config --get branch.master.remote
>>> Running: git branch --remotes
>>> Running: git rev-list e66800919f5d3dfba8a6416ad393794485aa2b9c --not --remotes=origin
>>> Running: git version
>>> Running: git -c core.quotepath=false -c diff.noprefix=false diff --no-color --full-index --ignore-submodules -M --no-ext-diff e66800919f5d3dfba8a6416ad393794485aa2b9c..a08f68f834ef86d8f9549fa3eb258422911421fe
>>> Generated diff size: 310 bytes
>>> Making HTTP GET request to http://reviewboard.oursite.com/api/validation/diffs/
>>> Cached response for HTTP GET http://reviewboard.oursite.com/api/validation/diffs/ expired and was modified
>>> Making HTTP POST request to http://reviewboard.oursite.com/api/validation/diffs/
>>> Got API Error 207 (HTTP code 400): The file was not found in the repository.
>>> Error data: {u'stat': u'fail', u'file': u'test.foo', u'err': {u'msg': u'The file was not found in the repository.', u'code': 207}, u'revision': u'b646fadbca709a770dfcca95cadaeb50aac92627'}
Traceback (most recent call last):
File "/usr/local/bin/rbt", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/main.py", line 120, in main
command.run_from_argv([RB_MAIN, command_name] + args)
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/__init__.py", line 725, in run_from_argv
exit_code = self.main(*args) or 0
File "/usr/local/lib/python2.7/dist-packages/rbtools/commands/post.py", line 874, in main
(msg_prefix, e))
rbtools.commands.CommandError: Error validating diff
test.foo: The file was not found in the repository. (HTTP 400, API Error 207)
Note the very first git diff command:
>>> Running: git -c core.quotepath=false -c diff.noprefix=false diff --no-color --full-index --ignore-submodules -M --no-ext-diff e66800919f5d3dfba8a6416ad393794485aa2b9c..a08f68f834ef86d8f9549fa3eb258422911421fe
This command has the actual, correct git hashes.
Note the API error:
>>> Got API Error 207 (HTTP code 400): The file was not found in the repository.
>>> Error data: {u'stat': u'fail', u'file': u'test.foo', u'err': {u'msg': u'The file was not found in the repository.', u'code': 207}, u'revision': u'b646fadbca709a770dfcca95cadaeb50aac92627'}
>>> Error data: {u'stat': u'fail', u'file': u'test.foo', u'err': {u'msg': u'The file was not found in the repository.', u'code': 207}, u'revision': u'b646fadbca709a770dfcca95cadaeb50aac92627'}
This has an incorrect hash, one that doesn't seem to exist in the remote repo, or my local working copy at all.
I put some debugging code in rbtools/api/request.py in "def make_request(self, request)" on the client to see if I could figure out what was going on, and I can confirm that the rbt client is in fact sending the wrong hash:
print("BODY: ", request.url, " ", body, " ", headers)
I get the following output:
rbt post a08f68f834ef86d8f9549fa3eb258422911421fe
(u'BODY: ', 'http://reviewboard.oursite.com/api/', u' ', None, u' ', {u'Content-Length': u'0'})
(u'BODY: ', 'http://reviewboard.oursite.com/api/validation/diffs/', u' ', None, u' ', {u'Content-Length': u'0'})
(u'BODY: ', 'http://reviewboard.oursite.com/api/validation/diffs/', u' ', '--===============6823395440781781946==\r\nContent-Disposition: form-data; name="base_commit_id"\r\n\r\ne66800919f5d3dfba8a6416ad393794485aa2b9c\r\n--===============6823395440781781946==\r\nContent-Disposition: form-data; name="repository"\r\n\r\nmygitrepo\r\n--===============6823395440781781946==\r\nContent-Disposition: form-data; name="path"; filename="diff"\r\nContent-Type: application/octet-stream\r\n\r\ndiff --git a/test.foo b/test.foo\nindex b646fadbca709a770dfcca95cadaeb50aac92627..0c3894442990dcaf42f8099cc3c0d9a731332174 100644\n--- a/test.foo\n+++ b/test.foo\n@@ -2,3 +2,5 @@\n exec /bin/bash\n # oh no I forgot another thing\n exec /bin/anotherthing\n+# boy do I love forgetting things\n+exec /bin/forgettingthings\n\r\n--===============6823395440781781946==--\r\n\r\n', u' ', {u'Content-Length': '741', u'Content-Type': u'multipart/form-data; boundary================6823395440781781946=='})
ERROR: Error validating diff
(u'BODY: ', 'http://reviewboard.oursite.com/api/', u' ', None, u' ', {u'Content-Length': u'0'})
(u'BODY: ', 'http://reviewboard.oursite.com/api/validation/diffs/', u' ', None, u' ', {u'Content-Length': u'0'})
(u'BODY: ', 'http://reviewboard.oursite.com/api/validation/diffs/', u' ', '--===============6823395440781781946==\r\nContent-Disposition: form-data; name="base_commit_id"\r\n\r\ne66800919f5d3dfba8a6416ad393794485aa2b9c\r\n--===============6823395440781781946==\r\nContent-Disposition: form-data; name="repository"\r\n\r\nmygitrepo\r\n--===============6823395440781781946==\r\nContent-Disposition: form-data; name="path"; filename="diff"\r\nContent-Type: application/octet-stream\r\n\r\ndiff --git a/test.foo b/test.foo\nindex b646fadbca709a770dfcca95cadaeb50aac92627..0c3894442990dcaf42f8099cc3c0d9a731332174 100644\n--- a/test.foo\n+++ b/test.foo\n@@ -2,3 +2,5 @@\n exec /bin/bash\n # oh no I forgot another thing\n exec /bin/anotherthing\n+# boy do I love forgetting things\n+exec /bin/forgettingthings\n\r\n--===============6823395440781781946==--\r\n\r\n', u' ', {u'Content-Length': '741', u'Content-Type': u'multipart/form-data; boundary================6823395440781781946=='})
ERROR: Error validating diff
There's that git hash that doesn't exist:
diff --git a/test.foo b/test.foo\nindex
b646fadbca709a770dfcca95cadaeb50aac92627..0c3894442990dcaf42f8099cc3c0d9a731332174
100644\n--- a/test.foo\n+++ b/test.foo\n@@ -2,3 +2,5
Where is the rbt client getting this hash? Should I have my RB repostories configured differently?
Thanks,
Ed
EM W
Aug 29, 2020, 3:41:01 PM8/29/20
to Review Board Community
Some additional information on the git hash: this is apparently the output of the --full-index argument, which git documentation defines as the full
pre- and post-image blob object names on the "index"
line when generating patch format output.
Naturally, when you're generating a diff from your local working repo (that you haven't committed yet), this won't exist on the remote git server, hence the 404. But this process works just fine on new files, which also don't exist in the remote.
Christian Hammond
Aug 30, 2020, 2:40:10 AM8/30/20
to revie...@googlegroups.com
Hi Ed,
Thanks for all the details! Always helpful when debugging.
Those IDs may not be what you'd expect, since generally as a Git user you only need to work with commit IDs, but they're actually the true identifier for a file's contents. Git has two IDs that matter when it comes to Review Board: commit IDs, and blob IDs (it also has others, like tree IDs, but we don't need them). When we want to look up a file's contents, we almost always have to use blob IDs (it depends on the service providing the API wrapping Git, but GitWeb itself does require these).
The blob IDs are ultimately what gets associated with the file on Review Board, since that's what represents those file contents. So when we fail to find it, we show that ID as part of the missing file information.
You should be able to verify that ID locally by doing a `git show <blob ID>`. That should show the raw file contents of the file.
If you run `git cat-file -t <blob ID>`, it should also say "blob".
So given that, it sounds like what's happening is that the parent commit (and its associated blobs) that the diff is based on doesn't exist upstream, or can't be located by GitWeb. There are some situations where this can happen (since Git is pretty loose with workflows and there's rarely a "correct" way of modeling things, our heuristics won't always work). A common one is that your central repository backed by GitWeb has a name other than "origin", and you're pushing changes to your own fork called "origin". We assume "origin" is the upstream to post against, and that it has any parent commits that are represented in the generated parent diff.
If "origin" does *not* point to the server backed by GitWeb, then you'll need to make sure you're specifying the nearest tracking branch that your branch is based off of that GitWeb has access to. This can be done with --tracking-branch=<remote>/<branch>, or by setting the TRACKING_BRANCH setting in .reviewboardrc.
Christian
--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/04ab008f-3102-4942-97ed-f951dc001704n%40googlegroups.com.
EM W
Aug 30, 2020, 8:12:20 PM8/30/20
to Review Board Community
here's my .reviewboardrc file:
cat .reviewboardrc
REVIEWBOARD_URL = "https://reviewboard.oursite.com"
REPOSITORY = "mygitrepo"
REPOSITORY_TYPE = "git"
BRANCH = "master"
LAND_DEST_BRANCH = "master"
TRACKING_BRANCH = "master"
REVIEWBOARD_URL = "https://reviewboard.oursite.com"
REPOSITORY = "mygitrepo"
REPOSITORY_TYPE = "git"
BRANCH = "master"
LAND_DEST_BRANCH = "master"
TRACKING_BRANCH = "master"
Here's the output of "git branch -a":
git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/pipeline
remotes/origin/centraldb
remotes/origin/master
remotes/origin/robot
remotes/origin/refactor
* master
remotes/origin/HEAD -> origin/master
remotes/origin/pipeline
remotes/origin/centraldb
remotes/origin/master
remotes/origin/robot
remotes/origin/refactor
Here is my .git/config file (local):
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = git:mygitrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = git:mygitrepo
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
As you can see, I added the "TRACKING_BRANCH", and I tried using "master", and "remotes/origin/master". However, this did not resolve the situation. Any ideas?
>> So given that, it sounds like what's happening is that the parent commit
(and its associated blobs) that the diff is based on doesn't exist
upstream, or can't be located by GitWeb.
I logged into our Git server, and cloned the repo locally, and looked for the previous commit that I was building on, and can confirm that the hash for that commit exists - or maybe I'm misunderstanding here?
EM W
Sep 10, 2020, 8:58:56 PM9/10/20
to Review Board Community
Per suggestion, I installed RB Gateway on my Git server in order to work around the issues I've been seeing.
Once I started RB Gateway and pointed RB at it, I was able to run "rbt post <hash" successfully. I am also able to download the diff of the file.
However, I could not view the diff inline on the "Diff" tab, and I got the following error:
testrb.sh
Revision 85b7582313a702cf32cac98ec2132d7bd7f47cbe
New Change
There was an error displaying this diff.
'URLError' object has no attribute 'code'
This may be a a temporary outage or an issue with the format of your diff.
Please try again, and if you still have trouble,
contact support.
Details
Revision 85b7582313a702cf32cac98ec2132d7bd7f47cbe
New Change
There was an error displaying this diff.
'URLError' object has no attribute 'code'
This may be a a temporary outage or an issue with the format of your diff.
Please try again, and if you still have trouble,
contact support.
Details
Traceback (most recent call last):
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/views.py", line 345, in get
response = renderer.render_to_response(request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 58, in render_to_response
return HttpResponse(self.render_to_string(request))
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 76, in render_to_string
large_data=True)
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 75, in <lambda>
lambda: self.render_to_string_uncached(request),
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 89, in render_to_string_uncached
request=request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/diffutils.py", line 774, in populate_diff_chunks
chunks = list(generator.get_chunks())
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 786, in get_chunks
for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key):
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 107, in get_chunks
large_data=True)
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 106, in <lambda>
lambda: list(self.get_chunks_uncached()),
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 792, in get_chunks_uncached
self.encoding_list)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/diffutils.py", line 230, in get_original_file
request=request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 424, in get_file
large_data=True)[0]
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 423, in <lambda>
request)],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 641, in _get_file_uncached
base_commit_id=base_commit_id)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/hostingsvcs/rbgateway.py", line 224, in get_file
if e.code == 404:
AttributeError: 'URLError' object has no attribute 'code'
response = renderer.render_to_response(request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 58, in render_to_response
return HttpResponse(self.render_to_string(request))
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 76, in render_to_string
large_data=True)
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 75, in <lambda>
lambda: self.render_to_string_uncached(request),
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/renderers.py", line 89, in render_to_string_uncached
request=request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/diffutils.py", line 774, in populate_diff_chunks
chunks = list(generator.get_chunks())
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 786, in get_chunks
for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key):
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 107, in get_chunks
large_data=True)
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 106, in <lambda>
lambda: list(self.get_chunks_uncached()),
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/chunk_generator.py", line 792, in get_chunks_uncached
self.encoding_list)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/diffviewer/diffutils.py", line 230, in get_original_file
request=request)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 424, in get_file
large_data=True)[0]
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 299, in cache_memoize
compress_large_data))
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 253, in cache_memoize_iter
items = items_or_callable()
File "/opt/venv/local/lib/python2.7/site-packages/djblets/cache/backend.py", line 296, in <lambda>
lambda: [lookup_callable()],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 423, in <lambda>
request)],
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/scmtools/models.py", line 641, in _get_file_uncached
base_commit_id=base_commit_id)
File "/opt/venv/local/lib/python2.7/site-packages/reviewboard/hostingsvcs/rbgateway.py", line 224, in get_file
if e.code == 404:
AttributeError: 'URLError' object has no attribute 'code'
For anyone else that runs into this, I get the same sort of errors anywhere I'm running RB, due to our self-signed certs. Adding the SSL_CERT_FILE
environment variable which pointed to the correct cert chain resolved this issue.
Reply all
Reply to author
Forward
0 new messages