BUG - SVN tries to connect to "akamai" - 15 second timeout - CRL - ctldl.windowsupdate.com
Cameron Sours
**Issue:** After invoking SVN on the command line on a firewalled server, nothing visible happens for 15 seconds, then the program quits with the following error:
svn: E170013: Unable to connect to a repository at URL 'SVN.REPOSITORY.REDACTED'
svn: E730054: Error running context: An existing connection was forcibly closed by the remote host.
**Investigation:** Internet research on the above errors did not uncover any pertinent information (hopefully they will now)
Process Tracing (procmon) showed a connection attempt to an Akamai (cloud services) server after the SSL/TLS handshake to the SVN Server. The hostname for the server was not shown in Process tracing. Reverse DNS lookup showed a184-51-112-88.deploy.static.akamaitechnologies.com or a184-51-112-80.deploy.static.akamaitechnologies.com as the hostname, and the IP was either 184.51.112.88 or 184.51.112.80 (2 entries in DNS cache).
Packet capture tool (MMA) showed a connection attempt to the hostname ctldl.windowsupdate.com after the SSL/TLS Handshake to the SVN server.
The windows Crypto API was attempting to connect to Windows Update to retrieve Certificate revocation information (CRL – certificate revocation list). The default timeout for CRL retrieval is 15 seconds. The timeout for authentication on the server is 10 seconds; as 15 is greater than 10, this fails.
**Resolution:** Internet research uncovered the following: (also see picture at bottom)
Solution 1: Decrease CRL timeout Group Policy -> Computer Config ->Windows Settings -> Security Settings -> Public Key Policies -> Certificate Path Validation Settings -> Network Retrieval
https://subversion.open.collab.net/ds/viewMessage.do?dsForumId=4&dsMessageId=470698
support.microsoft.com/en-us/kb/2625048
blogs.technet.com/b/exchange/archive/2010/05/14/3409948.aspx
Solution 2: Open firewall for CRL traffic
support.microsoft.com/en-us/kb/2677070
Solution 3: SVN command line flags (untested)
serverfault.com/questions/716845/tortoise-svn-initial-connect-timeout -
alternate svn command line flag solution.
**Additional Information:** Debugging this issue was particularly difficult. SVN 1.8 disabled support for the Neon HTTP RA (repository access) library in favor of the Serf library which removed client debug logging. [1] In addition, the SVN error code returned did not match the string given in svn_error_codes.h [2] Also, SVN Error codes cannot be mapped back to their ENUM label easily, this case SVN error code E170013 maps to SVN_ERR_RA_CANNOT_CREATE_SESSION.
1. stackoverflow.com/questions/8416989/is-it-possible-to-get-svn-client-debug-output
3. code.google.com/archive/p/serf/issues/172
**Suggested SVN Changes:**
1. Enable Verbosity option on the command line for all operations
2. Add error ENUM name to stderr
3. Add config flag for Serf Library debug logging.
Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.
Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.
Andreas Mohr
On Wed, Jan 27, 2016 at 08:32:45PM +0000, Cameron Sours wrote:
> **Additional Information:** Debugging this issue was particularly
> difficult. SVN 1.8 disabled support for the Neon HTTP RA (repository
> access) library in favor of the Serf library which removed client debug
> logging. [1] In addition, the SVN error code returned did not match the
> string given in svn_error_codes.h [2] Also, SVN Error codes cannot be
> mapped back to their ENUM label easily, this case SVN error code E170013
> maps to SVN_ERR_RA_CANNOT_CREATE_SESSION.
>
>
>
> 1.
> stackoverflow.com/questions/8416989/is-it-possible-to-get-svn-client-debug-output
(SvnBridge protocol compatibility analysis),
and while I knew that this logging is Neon-specific
(it's called neon-debug-mask after all...)
I had (stupidly?) expected Serf mode
to offer something comparable in functionality.
So, to restate Stackoverflow's question:
"So how does it work now?"
Don't tell me that one is expected to apply liberal use of packet analyzers now...
(well, not-so-"liberal" in that case, that is, for obvious reasons...)
OTOH serf-trunk (of http://serf.apache.org/contribute )
does seem to have logging serf:ed ;) in several areas,
so possibly svn does provide log enabling,
or would be able to implement such relatively easily,
and that would then possibly also spew the data of interest:
$ grep serf__log *
context.c: serf__log_init(ctx);
incoming.c: serf__log(LOGLVL_DEBUG, LOGCOMP_CONN, __FILE__,
client->config,
logging.c:apr_status_t serf__log_init(serf_context_t *ctx)
logging.c:void serf__log_nopref(apr_uint32_t level, apr_uint32_t comp,
logging.c:void serf__log(apr_uint32_t level, apr_uint32_t comp, const
char *prefix,
logging.c:int serf__log_enabled(apr_uint32_t level, apr_uint32_t comp,
serf_config_t *config)
logging.c:apr_status_t serf__log_init(serf_context_t *ctx)
logging.c:void serf__log_nopref(apr_uint32_t level, apr_uint32_t comp,
logging.c:void serf__log(apr_uint32_t level, apr_uint32_t comp, const
char *prefix,
logging.c:int serf__log_enabled(apr_uint32_t level, apr_uint32_t comp,
serf_config_t *config)
...
Congratulations for a very impressively maximally detailed issue description!
Andreas Mohr
Daniel Shahaf
> The windows Crypto API was attempting to connect to Windows Update to
> The timeout for authentication on the server is 10 seconds; as 15 is
> greater than 10, this fails.
> **Suggested SVN Changes:**
>
> 1. Enable Verbosity option on the command line for all operations
command-line through the --config-option flag. Is that what you mean?
> 2. Add error ENUM name to stderr
(unix). When dealing with release builds, you can convert E170013 codes
to symbolic names with tools/dev/which_error.py.
The E730054 code you saw is the sum of
SOCECONNRESET = 10054
APR_OS_START_SYSERR = 720000
(from tools/dev/aprerr.txt), which that script doesn't yet handle.
> 3. Add config flag for Serf Library debug logging.
>
Daniel