problems encountered using git lfs push on a very large repository

[euphxenos]

One of our projects has a very large repository as a result of developers checking in a number of large binaries.  I'm looking at using lfs on Gerrit 2.14 to see if that will help.  I took a mirror of this repository and used git-lfs-migrate to convert it.  At the end of the conversion, my converted local copy of the repository has a much smaller objects directory, and the bulk of the repository is now under the new lfs subdirectory.  After pushing refs/heads and refs/tags to a new repository on my Gerrit server, I did a "git lfs push --all origin" and I see that the directory I specified for the fs backend in lfs.config starts to fill up with data.  It gets a bit more than 60% of the way through, then I get a bunch of errors that look like this:

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/fc9913317b68a40842badadea3039dfbfbb584f999a66ee947a094180cd78b9e

Check that you have proper access to the repository

Which seems odd, since it already transferred most of my converted repo's lfs subdirectory to the server.  If there were genuine authentication/authorization issues, I would have expected to hit that at the beginning, not more than halfway through the push.  If I run my "git lfs push --all origin" again, it continues without error, but it does not transfer the entire contents of my converted repo's lfs subdirectory.  Any successive runs after that have no effect.  If I try this process again from scratch (easily repeated, since I'm doing this in a virtual machine which I can revert between tests), the first and second "git lfs push --all origin" always fail in the same place and continue to the same place.  What would cause it to fail like this?  The authentication/authorization error sure makes this sound like a permissions mistake.  I've given myself create reference, create annotated tag, push (force), push merge commit, forge author, and forge committer on refs/* -- was there another permission I needed for this?

I notice that the directory on the Gerrit server that acts as the lfs backend and the lfs/objects subdirectory of my converted repository appear to have the same contents (though the server's copy is incomplete so far), with the data sorted into buckets two levels deep based on the first four characters of the filenames.  Can I simply copy the contents of that directory over to the lfs backend directory on my gerrit server and skip the git lfs push, or is there some additional accounting taking place when it gets pushed in through the Gerrit server's git-lfs plugin?  Is there a better way to workaround this?  In addition to getting the initial data into the repository, I'm also concerned about what I'll need to do to recover going forward if a user has a git lfs push fail partway through their push.

thanks,

--Andrew

[David Pursehouse]

On Fri, Jun 2, 2017 at 3:41 PM euphxenos <euph...@gmail.com> wrote:

One of our projects has a very large repository as a result of developers checking in a number of large binaries.  I'm looking at using lfs on Gerrit 2.14 to see if that will help.  I took a mirror of this repository and used git-lfs-migrate to convert it.  At the end of the conversion, my converted local copy of the repository has a much smaller objects directory, and the bulk of the repository is now under the new lfs subdirectory.  After pushing refs/heads and refs/tags to a new repository on my Gerrit server, I did a "git lfs push --all origin" and I see that the directory I specified for the fs backend in lfs.config starts to fill up with data.  It gets a bit more than 60% of the way through, then I get a bunch of errors that look like this:

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/fc9913317b68a40842badadea3039dfbfbb584f999a66ee947a094180cd78b9e

Check that you have proper access to the repository

Can you try it with tracing and see if that provides any useful additional information?

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git lfs push --all origin

 

Also, are you pushing over ssh or http?

Which seems odd, since it already transferred most of my converted repo's lfs subdirectory to the server.  If there were genuine authentication/authorization issues, I would have expected to hit that at the beginning, not more than halfway through the push.  If I run my "git lfs push --all origin" again, it continues without error, but it does not transfer the entire contents of my converted repo's lfs subdirectory.  Any successive runs after that have no effect.  If I try this process again from scratch (easily repeated, since I'm doing this in a virtual machine which I can revert between tests), the first and second "git lfs push --all origin" always fail in the same place and continue to the same place.  What would cause it to fail like this?  The authentication/authorization error sure makes this sound like a permissions mistake.  I've given myself create reference, create annotated tag, push (force), push merge commit, forge author, and forge committer on refs/* -- was there another permission I needed for this?

I notice that the directory on the Gerrit server that acts as the lfs backend and the lfs/objects subdirectory of my converted repository appear to have the same contents (though the server's copy is incomplete so far), with the data sorted into buckets two levels deep based on the first four characters of the filenames.  Can I simply copy the contents of that directory over to the lfs backend directory on my gerrit server and skip the git lfs push, or is there some additional accounting taking place when it gets pushed in through the Gerrit server's git-lfs plugin?  Is there a better way to workaround this?  In addition to getting the initial data into the repository, I'm also concerned about what I'll need to do to recover going forward if a user has a git lfs push fail partway through their push.

thanks,

--Andrew

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jacek Centkowski]

Hi,

It would be also good to get more details about the operation:

- what is the binaries size that are being pushed

- git LFS client version

- how much time it goes before unrecoverable failure happens

- can you also go ahead and check if there are entries in the gerrit/logs/error_log

Regards

Jacek

[euphxenos]

On Friday, June 2, 2017 at 12:36:43 AM UTC-7, David Pursehouse wrote:

On Fri, Jun 2, 2017 at 3:41 PM euphxenos <euph...@gmail.com> wrote:

One of our projects has a very large repository as a result of developers checking in a number of large binaries.  I'm looking at using lfs on Gerrit 2.14 to see if that will help.  I took a mirror of this repository and used git-lfs-migrate to convert it.  At the end of the conversion, my converted local copy of the repository has a much smaller objects directory, and the bulk of the repository is now under the new lfs subdirectory.  After pushing refs/heads and refs/tags to a new repository on my Gerrit server, I did a "git lfs push --all origin" and I see that the directory I specified for the fs backend in lfs.config starts to fill up with data.  It gets a bit more than 60% of the way through, then I get a bunch of errors that look like this:

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/fc9913317b68a40842badadea3039dfbfbb584f999a66ee947a094180cd78b9e

Check that you have proper access to the repository

Can you try it with tracing and see if that provides any useful additional information?

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git lfs push --all origin 

Here's the bit from right before the failures start to happen.  The last two files that it sends (d4938df8319769abda0b2748c26acf81f07781a6be825d1accbe4c1381dd7676 and 281e6715ea9dbc56d4cf46ea33aa078f1345f80c42552f1d20853ff20fa7773f) made it to the server.

trace git-lfs: HTTP: 200

trace git-lfs: HTTP: PUT https://redacted.redacted.com:8080/plugins/lfs/content/default/d4938df8319769abda0b2748c26acf81f07781a6be825d1accbe4c1381dd7676

> PUT /plugins/lfs/content/default/d4938df8319769abda0b2748c26acf81f07781a6be825d1accbe4c1381dd7676 HTTP/1.1

> Host: redacted.redacted.com:8080

> Authorization: f/Ddz9FzErmZ3BqVcK/hxMnDwhewcH9blKyLgqdYcvqms2ZgsPL/+0Rv49RqHSqG+J+NXg2zYiSPrCzHOyAE+7CINwh+XtIKPicZHD+wkHUtPk5fxm9IyVTJYJWh+9CcLAxFB5M9lPj0bXhJZaJ0k7e2p1Dav3wTPBX7or4OK10=

> Content-Length: 16984

> Content-Type: application/octet-stream

> User-Agent: git-lfs/2.1.1 (GitHub; linux amd64; go 1.8.1; git 3314e28)

> 

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: application/vnd.git-lfs+json

< Date: Fri, 02 Jun 2017 17:42:32 GMT

< 

trace git-lfs: HTTP: 200

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: application/vnd.git-lfs+json

< Date: Fri, 02 Jun 2017 17:42:32 GMT

< 

trace git-lfs: HTTP: PUT https://redacted.redacted.com:8080/plugins/lfs/content/default/b15c4545d3bd847bb8f77bb8535704fe82e90b1e8ee38585668c38cb0da70ebb

> PUT /plugins/lfs/content/default/b15c4545d3bd847bb8f77bb8535704fe82e90b1e8ee38585668c38cb0da70ebb HTTP/1.1

> Host: redacted.redacted.com:8080

> Authorization: mAXZyW9sw+50gyWeyd+itGwvcmijfwC27BxowfUweQyCzxWX6mx08PvuN3MRfROoQZbDK/SEFH91smok1PITxUBJtKhblGKiXV6JFYDhauSkInGTJq//XRMArX75+S4PyFDzS1I0DF3uZ7s+qnjDGIRZzFpgPd8OQcLo1Qz/uCs=

> Content-Length: 16808

> Content-Type: application/octet-stream

> User-Agent: git-lfs/2.1.1 (GitHub; linux amd64; go 1.8.1; git 3314e28)

> 

trace git-lfs: HTTP: PUT https://redacted.redacted.com:8080/plugins/lfs/content/default/281e6715ea9dbc56d4cf46ea33aa078f1345f80c42552f1d20853ff20fa7773f

> PUT /plugins/lfs/content/default/281e6715ea9dbc56d4cf46ea33aa078f1345f80c42552f1d20853ff20fa7773f HTTP/1.1

> Host: redacted.redacted.com:8080

> Authorization: NGTpL3rB6ITgL4Lc5ZjbH3QPSWk8mQEmpCRwBjekD1FkecSjgFFTXMC4ieINhsPAbMqrxKG6SKJ8jGN7hOZFqN0L6qsV2qYbtwc69NaoHn6uZIvZ7ESzC1RBiMFGhiWx7aESjaYcJWYAZQukvp+kaPPL6CfTOAhQw/yRFmIVfpE=

> Content-Length: 9960

> Content-Type: application/octet-stream

> User-Agent: git-lfs/2.1.1 (GitHub; linux amd64; go 1.8.1; git 3314e28)

> 

trace git-lfs: HTTP: 200

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: application/vnd.git-lfs+json

< Date: Fri, 02 Jun 2017 17:42:32 GMT

< 

trace git-lfs: HTTP: 200

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: application/vnd.git-lfs+json

< Date: Fri, 02 Jun 2017 17:42:32 GMT

< 

Git LFS: (3758 of 3831 files) 62.81 GB / 77.76 GB                              

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/fc9913317b68a40842badadea3039dfbfbb584f999a66ee947a094180cd78b9e

Check that you have proper access to the repository

LFS: Put https://redacted.redacted.com:8080/plugins/lfs/content/default/2ba61b2bc7fb1f23d1f88526d3f60d4649e37181b569230a654ddefb78edfc76: write tcp 10.22.8.19:50782->10.22.8.19:8080: write: broken pipe

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/0ac96ea22be5fbf2de13bc138e8f62f77d58a59fd770a745167506dd292b4aa3

Check that you have proper access to the repository

Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/389d96a39f458c2a2821cc2b5230e309d4b45e63c389636247e160aa0574963f

Check that you have proper access to the repository

 

Also, are you pushing over ssh or http?

I'm pushing over http.  I'll give ssh a try as well.

thanks,

--Andrew

[euphxenos]

On Friday, June 2, 2017 at 1:19:20 AM UTC-7, Jacek Centkowski wrote:

Hi,

It would be also good to get more details about the operation:

- what is the binaries size that are being pushed

In total, I'm pushing 100 gigs of binaries.  The binary file it fails on is 1.1 gigs in size (1162715887 bytes).  It pushes larger files to the server before it fails on this one.  63 gigs were pushed in total before the failure.

 

- git LFS client version

git lfs version reports:

git-lfs/2.1.1 (GitHub; linux amd64; go 1.8.1; git 3314e28)

 

- how much time it goes before unrecoverable failure happens

55 minutes go by between starting the git lfs push and the failure.  It was successfully pushing files right up until the failure.

 

- can you also go ahead and check if there are entries in the gerrit/logs/error_log

Nothing was logged to error_log during my push.  The last messages to appear in error_log were the normal startup messages from starting Gerrit.

thanks,

--Andrew

[euphxenos]

If I try pushing over ssh, it fails faster.  "git lfs push --all origin" pushes nine files to the server (totalling about 2 gigs), then hangs (or at least it hadn't done anything else after two hours, before I killed it).  Turning on debugging, I see this at the end -- it looks like it gets errors and retries multiple objects before it hangs, again with some sort of authentication/authorization error.  What could cause it to hit that kind of error after successfully uploading other files?

thanks,

--Andrew

< HTTP/1.1 401 Unauthorized

< Transfer-Encoding: chunked

< Date: Sat, 03 Jun 2017 10:34:02 GMT

< 

trace git-lfs: tq: retrying object d9097971c5a7692a43753ca2a229e8ec1c3f75c185906fd140ee4b9b971f4fa0: Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/d9097971c5a7692a43753ca2a229e8ec1c3f75c185906fd140ee4b9b971f4fa0

Check that you have proper access to the repository

trace git-lfs: HTTP: PUT https://redacted.redacted.com:8080/plugins/lfs/content/default/fc0a69d302a99f7d6565503d3ffaec972b4f9b87a2e46bcd3d52a1e79afb976f

trace git-lfs: tq: enqueue retry #1 for "d9097971c5a7692a43753ca2a229e8ec1c3f75c185906fd140ee4b9b971f4fa0" (size: 285)

> PUT /plugins/lfs/content/default/fc0a69d302a99f7d6565503d3ffaec972b4f9b87a2e46bcd3d52a1e79afb976f HTTP/1.1

> Host: redacted.redacted.com:8080

> Authorization: Sk8+9hVe5zR3vx8uwIHOo+UCtUrw3yIhW5RSEsQZ433sP4VD+y8NNn6wGXBSZlptyW52du/5tyVjbGzlnchljppWx4Hyv3GW4BMItS9dKCOYH8sPBcTK5xxESfUWiW0G3drOfRXJnx6bdj0wn9/ozIi9PIcKq8gqw5SlWRAEE8U=

> Content-Length: 264

> Content-Type: application/octet-stream

> User-Agent: git-lfs/2.1.1 (GitHub; linux amd64; go 1.8.1; git 3314e28)

> 

trace git-lfs: HTTP: 401

< HTTP/1.1 401 Unauthorized

< Transfer-Encoding: chunked

< Date: Sat, 03 Jun 2017 10:34:02 GMT

< 

trace git-lfs: tq: retrying object fc0a69d302a99f7d6565503d3ffaec972b4f9b87a2e46bcd3d52a1e79afb976f: Authentication required: Authorization error: https://redacted.redacted.com:8080/plugins/lfs/content/default/fc0a69d302a99f7d6565503d3ffaec972b4f9b87a2e46bcd3d52a1e79afb976f

Check that you have proper access to the repository

trace git-lfs: tq: enqueue retry #1 for "fc0a69d302a99f7d6565503d3ffaec972b4f9b87a2e46bcd3d52a1e79afb976f" (size: 264)

Git LFS: (8 of 201 files) 1.90 GB / 2.28 GB                                    trace git-lfs: HTTP: 200

< HTTP/1.1 200 OK

< Content-Length: 0

< Content-Type: application/vnd.git-lfs+json

< Date: Sat, 03 Jun 2017 10:33:29 GMT

< 

Git LFS: (9 of 201 files) 1.90 GB / 2.28 GB                                    

[Jacek Centkowski]

Hi,

Last but not least what is the version of Gerrit and plugin (as seen on plugins list)?

You can try to play with authorisation token expiration time (default 10s) and set it for push time to some big value (e.g. 10min?), but it could be rather curing symptoms not real issue. Will look into it in the following days to reproduce/confirm.

Regards,

Jacek 

[euphxenos]

On Saturday, June 3, 2017 at 6:46:28 AM UTC-7, Jacek Centkowski wrote:

Hi,

Last but not least what is the version of Gerrit and plugin (as seen on plugins list)?

This is on Gerrit version 2.14, and the lfs plugin shows up as in a plugin ls as version e73548f:

lfs                            e73548f    ENABLED  lfs.jar

 

You can try to play with authorisation token expiration time (default 10s) and set it for push time to some big value (e.g. 10min?), but it could be rather curing symptoms not real issue. Will look into it in the following days to reproduce/confirm.

I already tried setting a large timeout for credential.helper on the client side.  I believe that's off by default and defaults to 15 minutes when you turn it on (I tried setting it to 3 hours).  Are you referring to a different setting?  If so, which one?

I was able to push the files by specifying the individual object IDs for each one on the command line (git lfs push --object-id origin <object ID>).  That's obviously a bit of a kludge, but it worked.  If you need more information or have anything else you'd like me to try, just let me know.

thanks,

--Andrew

[Jacek Centkowski]

Thanks for versions. What I have meant was Gerrit LFS client setting and link points to plugin documentation.

Regards
Jacek

You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/o-Ypjv8bM6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

[Jacek Centkowski]

Plugin documentation :) sorry autocorrect changed it to client in the first part of the sentence.

[euphxenos]

I tried it again with lfs.expirationSeconds set to 600 (ten minutes) in gerrit.config, but it failed in the same way as before.


--Andrew

[Yingchun Li]

Hi, 
     Did you resolved this problem? I have encounter almost same issue, 

When I pushed the binary, the who process ask me input user/password

three times? the server return ok, but when push the real content, it fails

with HTTP 401:

   5:54:30.529576 trace git-lfs: HTTP: POST http://gerrit-user@gerrit-server/a/lfs-test.git/info/lfs/objects/batch

15:54:30.595074 trace git-lfs: HTTP: 200
15:54:30.595180 trace git-lfs: HTTP: {"objects":[{"oid":"e3d74a19b69b0508a995fd83f020cd21dc49db1fa26ff20a93eb2a64ab1e2094","size":41205760,"actions":{"upload":{"expires_at":"2022-09-17T07:54:40.590Z","expires_in":10,"href":"http://gerrit-server/plugins/lfs/content/default/e3d74a19b69b0508a995fd83f020cd21dc49db1fa26ff20a93eb2a64ab1e2094","header":{"Authorization":"jDxcij0lq0xW4sND0VxG7SAwQ6fxRvqaEbOV9Bv81JtZmThp7CftYgAASaSCUce2HFoSDLZIDg9UgdbfAC7ETPUafFAiht4QGlwYXG6gBa8UeSx8P/iA2kte/moUJfee65l3ws1BuWvq+86xYTV/M0j+zzJO+TxIHWq1sW+Eur4="}}}}]}
15:54:30.595356 trace git-lfs: tq: starting transfer adapter "basic"
15:54:30.595907 trace git-lfs: HTTP: PUT http://gerrit-server/plugins/lfs/content/default/e3d74a19b69b0508a995fd83f020cd21dc49db1fa26ff20a93eb2a64ab1e2094
15:54:30.623759 trace git-lfs: HTTP: 401

   The gerrit-server is 3.4.3, I use the nginx http server, the git-lfs client is 2.13.0(the latest git lfs 3.20 also cannot work).