HTTP Error: 413 (FULL head)

[Alfred Söderdahl]

Me and my team members get this error when we try to make any action (check incoming, pull, detect outgoing or push) to our central Mercurial repository.

Log output:

% hg --repository C:\HG\WBT2 incoming --quiet --bundle c:\users\ajo\appdata\local\temp\thg.fq96ba\http...@hg.agero.net_hg_hg_WBT2_wnasdi.hg --debug http://aso:***@hg.agero.net/hg/hg/WBT2

using http://hg.agero.net/hg/hg/WBT2

http auth: user aso, password ********

sending capabilities command

http auth: user aso, password ********

comparing with http://aso:***@hg.agero.net/hg/hg/WBT2

query 1; heads

sending batch command

http auth: user aso, password ********

searching for changes

taking initial sample

query 2; still undecided: 390, sample size is: 200

sending known command

HTTP Error: 413 (FULL head)

[command returned code 255 Fri Mar 08 08:20:05 2013]

Our temporary work around is to pull something from another team members repository, which seems to fix our own repository, then we can work towards our central repository again. But the error comes back after a while again.

/Alfred

[Sebastian Sdorra]

Hi,

Which version of scm-manager do you use? Do you use the standalone version or the webapp version?

Sebastian

2013/3/8 Alfred Söderdahl <bist...@gmail.com>

/Alfred

--
You received this message because you are subscribed to the Google Groups "scmmanager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scmmanager+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alfred Söderdahl]

Version is 1.20. I'm not sure about standalone/webapp because I'm not responsible for the installation. Maybe we should upgrade to latest before any further investigation?

/Alfred

Den fredagen den 8:e mars 2013 kl. 08:43:10 UTC+1 skrev Sebastian Sdorra:

Hi,

Which version of scm-manager do you use? Do you use the standalone version or the webapp version?

Sebastian

2013/3/8 Alfred Söderdahl <bist...@gmail.com>

Me and my team members get this error when we try to make any action (check incoming, pull, detect outgoing or push) to our central Mercurial repository.

Log output:

% hg --repository C:\HG\WBT2 incoming --quiet --bundle c:\users\ajo\appdata\local\temp\thg.fq96ba\http__aso@hg.agero.net_hg_hg_WBT2_wnasdi.hg --debug http://aso:***@hg.agero.net/hg/hg/WBT2

using http://hg.agero.net/hg/hg/WBT2

http auth: user aso, password ********

sending capabilities command

http auth: user aso, password ********

comparing with http://aso:***@hg.agero.net/hg/hg/WBT2

query 1; heads

sending batch command

http auth: user aso, password ********

searching for changes

taking initial sample

query 2; still undecided: 390, sample size is: 200

sending known command

HTTP Error: 413 (FULL head)

[command returned code 255 Fri Mar 08 08:20:05 2013]

[Sebastian Sdorra]

The problem is the header buffer of the application server, this is because of mercurial uses very big headers. You have to increase the size of the header buffer and this application server specific. In case you are using the standalone version, you have to edit the server-config.xml and increase the requestHeaderSize value. 

replace:

<Set name="requestHeaderSize">16384</Set>

with:

<Set name="requestHeaderSize">32768</Set>

If you are using the webapp with tomcat or glassfish, please have a look at https://bitbucket.org/sdorra/scm-manager/wiki/applicationserver

Sebastian

[Alfred Söderdahl]

Thanks! I'll try your solution.

/Alfred

[Alfred Söderdahl]

Ok, just found out we're running behind IIS using Helicon Zoo. Next level in complexity;) Now I have to figure out if IIS or Helicon Zoo sets the header limit.

/Alfred

[Alfred Söderdahl]

Solved all issues by installing standalone scmmanager with requestHeaderSize set to 32768 and using IIS as a proxy only. Thanks Sebastian!

/Alfred 

[DÁvidám Horváth]

Hi Sebastian,

We are facing a similar problem when cloning or pulling some of our repos.

I see that our requestHeaderSize is already set to 32768.

Should we further increase this value, or should we start looking for problems elsewhere?

Best regards,

David Horvath

[DÁvidám Horváth]

Hi all, we finally went down on the road of doubling requestHeaderSize. Twice :D

So I can say 131072 works for it as well.

Some relevant things to read:

https://bugzilla.mozilla.org/show_bug.cgi?id=898638

https://www.mercurial-scm.org/wiki/BigRepositories

https://bitbucket.org/sdorra/scm-manager/wiki/applicationserver
https://serverfault.com/questions/136249/how-do-we-increase-the-maximum-allowed-http-get-query-length-in-jetty

Basically - as far as I understand the issue now - hg pull (without any -r parameter) collects all the local heads (closed and open as well) and sends it to the remote server. By default it works via HTTP GET and basically the heads just don't fit into the request. According to my collegue the web server should allow more get requests but our configuration (jetty) apparently did not. We don't see why. So we must fit into one query for now.

In the long term I am about to test 10000 or even 100000 heads and see how it scales. If nothing works, we might create some service which mock-merges the closed heads to a mock_closed branch thus reducing topological heads... I don't know yet.

I hope this helps someone - even future me.