scm:Internal error during upload-pack

[Guillaume Riflet]

Hi all,

I have this production server with scm-manager. One of the repos is causing the server to become unstable and to crash,

This happens when the users try to clone a large repo. They get this error in their git client:

"""

fatal: The remote end hung up unexpectedly error: RPC failed; result=22, HTTP code = 502

"""

If I look at the log on the server I get:

> cat /scm/var/log/scm-server.err | grep -vP --regexp='at \w+'

"""

2014-12-03 14:04:54.348:WARN:/scm:Internal error during upload-pack

org.eclipse.jetty.io.EofException

Caused by:

java.io.IOException: Broken pipe

"""

The jmap -heap of the process yields:

"""

Attaching to process ID 26983, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.65-b04

using thread-local object allocation.

Parallel GC with 2 thread(s)

Heap Configuration:

   MinHeapFreeRatio = 0

   MaxHeapFreeRatio = 100

   MaxHeapSize      = 1038090240 (990.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 85983232 (82.0MB)

   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:

PS Young Generation

Eden Space:

   capacity = 102236160 (97.5MB)

   used     = 6895704 (6.576255798339844MB)

   free     = 95340456 (90.92374420166016MB)

   6.744877741887019% used

From Space:

   capacity = 36175872 (34.5MB)

   used     = 16516224 (15.7510986328125MB)

   free     = 19659648 (18.7489013671875MB)

   45.65535835597826% used

To Space:

   capacity = 38797312 (37.0MB)

   used     = 0 (0.0MB)

   free     = 38797312 (37.0MB)

   0.0% used

PS Old Generation

   capacity = 510132224 (486.5MB)

   used     = 398761384 (380.28849029541016MB)

   free     = 111370840 (106.21150970458984MB)

   78.16824055404115% used

PS Perm Generation

   capacity = 36700160 (35.0MB)

   used     = 36322672 (34.63999938964844MB)

   free     = 377488 (0.3600006103515625MB)

   98.97142682756696% used

11290 interned Strings occupying 946040 bytes.

"""

My java version is

"""

java version "1.7.0_65"

Java(TM) SE Runtime Environment (build 1.7.0_65-b17)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

"""

I don't know how to check the version of jsvc or the version of the scm-manager.

Can you help me out?

Thanks,

Guillaume Riflet

[Sebastian Sdorra]

Hi,

You can see the version of SCM-Manager at the bottom right of the webpage. Could you post the main log file. Which is located in the SCM-Manager home directory, normally ~/.scm/logs. EofException's are mostly thrown on client aborts. The MaxNewSize value looks very strange (17592186044415 MB ???). Could you post your jvm options?

Sebastian

--
You received this message because you are subscribed to the Google Groups "scmmanager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scmmanager+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guillaume Riflet]

Hi,

Thank for answering!

Our current scm version is 1.35.

If i do a "ps aux | grep jsvc" I find the following options with the jsvc process:

jsvc.exec -cp :/scm/conf:/scm/lib/scm-server-1.35.jar:/scm/lib/commons-daemon-1.0.10.jar:/scm/lib/jetty-server-7.6.14.v20131031.jar:/scm/lib/javax.servlet-2.5.0.v201103041518.jar:/scm/lib/jetty-continuation-7.6.14.v20131031.jar:/scm/lib/jetty-http-7.6.14.v20131031.jar:/scm/lib/jetty-io-7.6.14.v20131031.jar:/scm/lib/jetty-webapp-7.6.14.v20131031.jar:/scm/lib/jetty-xml-7.6.14.v20131031.jar:/scm/lib/jetty-servlet-7.6.14.v20131031.jar:/scm/lib/jetty-security-7.6.14.v20131031.jar:/scm/lib/jetty-jmx-7.6.14.v20131031.jar:/scm/lib/jetty-util-7.6.14.v20131031.jar:/scm/lib/jetty-ajp-7.6.14.v20131031.jar -Djava.awt.headless=true -Dlogback.configurationFile=logging.xml -outfile /scm/var/log/scm-server.out -errfile /scm/var/log/scm-server.err -pidfile /scm/var/scm-server.pid -jvm server -home /usr/lib/jvm/java-7-oracle -Dapp.name=scm-server -Dapp.repo=/scm/lib -Dbasedir=/scm sonia.scm.server.ScmServerDaemon star


I'm not sure about which logs do you want, so I send you both the .err and the .out in a zipped tar in attachment.

Any help or hint is greatly appreciated, thank you very much,

Best regards,

Guillaume Riflet

[Guillaume Riflet]

Hello Sebastian,

I'm sorry to insist but if you could, please please, manage to find some time to look into our problem, it would really help us :)

Merry holidays! I hope you had a great year and that the following will be an even greater year!

Thank you very much,

Guillaume Riflet

--
You received this message because you are subscribed to a topic in the Google Groups "scmmanager" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scmmanager/swfmaYoC7c8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scmmanager+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

mob. : (+351) 934 100 491

http://www.google.com/profiles/guillaume.riflet

[Sebastian Sdorra]

Hi,

I need the log file from the scm home directory, which is normally located in ~/.scm/logs. Could you also try to upgrade scm-manager to the latest version? Because version 1.44, contains the latest version of jgit. You could also try to increase the max memory limit, by adding -Xmx512m to EXTRA_JVM_ARGUMENTS. How did you installed scm-manager (standalone, rpm, deb)?

Sebastian

[Guillaume Riflet]

Hi Sebastian,

Thank you for the answer. We will consider the upgrade to the latest version of scm ,1.44. It has to be carefully planned because it is a production server.

We installed scm as a deb. 

If I do a cat /proc/version I get:

nb19963@CF-SVNGIT:~$ cat /proc/version

Linux version 3.16.0-25-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014

The tip to increase the max memory limit will probably be easier to test and implement, thank you. I'll post feedback for that.

I send you as a link in google drive the logs in the tar gzip format. Please let me know if you have any trouble in downloading and extracting them.

Also keep in mind that we serve the scm behind another apache server that acts as a reverse-proxy.

Thank you so much!

Guillaume Riflet​

 scm.tgz

​