git clone failing with some large repositories

[Keithel]

When cloning any of the larger repositories in gitorious, I've been
getting failures from git, I've been getting failures like:

[kyzivat@myhost kyzivat] git clone http://git.gitorious.org/webkit/webkit.git
Initialized empty Git repository in /tmp/kyzivat/webkit/.git/
error: Unable to get pack file
http://git.gitorious.org/webkit/webkit.git/objects/pack/pack-c82803068e1906127bb264768945a5d40e59692a.pack
transfer closed with 458944958 bytes remaining to read
error: Unable to find 24a6f0d94544cf308967bbbcaa815ead81def704 under
http://git.gitorious.org/webkit/webkit.git
Cannot obtain needed blob 24a6f0d94544cf308967bbbcaa815ead81def704
while processing commit c7a73cd47acbddc29eb3f5aace30a452e986d4a3.
fatal: Fetch failed.

This error happens with at least 3 large gitorious repositories, and
either accessed via http or git protocol, with and without proxy (I
tested this at home on a different system last night)
The error is reported slightly differently when repository is cloned
with git protocol, but the same chunk reports failure.
Repositories tested:
git clone http://git.gitorious.org/webkit/webkit.git
git clone git://gitorious.org/webkit/webkit.git
git clone http://git.gitorious.org/xbmc/xbmc.git

Looking through the webkit-help archives, I see someone posted
yesterday that they were having a similar problem to mine with the
git.webkit.org repository -- does that one point to gitorious?
If not, then my thought of it being an issue with gitorious is less
credible.

https://lists.webkit.org/pipermail/webkit-help/2009-November/000375.html

Can anyone shed some light onto the issue and provide any help cloning
the qt webkit repository?

[Artem Zolochevskiy]

On 4 ноя, 17:10, Keithel <kam...@gmail.com> wrote:
> When cloning any of the larger repositories in gitorious, I've been
> getting failures from git

+ 1

remote: Counting objects: 4506, done.
remote: Compressing objects: 100% (2950/2950), done.
fatal: The remote end hung up unexpectedlyMiB | 20 KiB/s
fatal: early EOF
fatal: index-pack failed

[Keithel]

Johan mentioned this morning that some of the git daemons were stuck
in a bad state this morning,
however him fixing that issue did not fix my issues with accessing
large repositories with git-over-http. (git http://...)

I'd say give a try again.

I have another reply I'm going to make in a minute to tell people how
to tunnel the git protocol itself over HTTP CONNECT proxies that
support connections to servers over port 9418 which may help you.

On Nov 4, 2:44 pm, Artem Zolochevskiy <artem.zolochevs...@gmail.com>
wrote:

[Keithel]

Unfortunately, I was unable to get git clone http://git.gitorious.org/webkit/webkit.git
to work properly,
however I was able to get git fetching with the git protocol behind
our firewall.
It requires your http proxy to be configured to allow HTTP CONNECT to
port 9418 (the standard git protocol port), so it may not work for
everyone.

I adapted these instructions from:
http://www.emilsit.net/blog/archives/how-to-use-the-git-protocol-through-a-http-connect-proxy/
and changed them to use corkscrew instead of socat - as I already had
corkscrew handy due to configuring SSH to use it (
http://blog.gitorious.org/2009/10/20/stuck-behind-a-firewall/ ).

Here's how:

Edit a new file ~/bin/gitproxy and put this content in it:
#!/bin/sh

# Use corkscrew to proxy git through an HTTP CONNECT firewall.
# Useful if you are trying to clone git:// from inside a company.
# Requires that the proxy allows CONNECT through port 9418
#
# Save this file as gitproxy somewhere in your path (e.g. ~/bin)
# and then run:
# chmod +x gitproxy
# git config --global core.gitproxy gitproxy
#
# Adapted from http://tinyurl.com/8xvpny which uses socat instead of
corkscrew.

_proxy=<your proxy ip here>
_proxyport=<your proxy port here>
exec corkscrew $_proxy $_proxyport $1 $2

Make sure that ~/bin is in your path, and that gitproxy is executable
(per comments in file).
Configure git to use this new proxy command via:
git config --global core.gitproxy gitproxy

and now, if your HTTP CONNECT proxy is configured to allow proxy
connections to port 9418, then you should be able to access git
repositories using the git:// protocol.