Artifact download timed-out

Aleksandr Sorokoumov

unread,

Jun 12, 2016, 6:18:09 AM6/12/16

to Nomad

Hi all,

I have a job which requires an artifact. When I run it, the allocation is failing because the artifact download is being timed out:

# nomad alloc-status 6e64c577
ID            = 6e64c577
Eval ID       = f35bcbe6
Name          = http-task.http-group[0]
Node ID       = 8380894f
Job ID        = http-job
Client Status = pending

==> Task Resources
Task: "http-task"
CPU  Memory MB  Disk MB  IOPS  Addresses
500  256        300      0

==> Task "http-task" is "pending"
Recent Events:
Time                    Type                      Description
12/06/16 11:57:56 CEST  Restarting                Task restarting in 25.440835412s
12/06/16 11:57:56 CEST  Failed Artifact Download  GET error: Get https://raw.githubusercontent.com/FRosner/nomad-docker-wrapper/master/nomad-docker-wrapper: dial tcp 23.235.43.133:443: i/o timeout
12/06/16 11:57:26 CEST  Downloading Artifacts     Client is downloading artifacts
12/06/16 11:56:55 CEST  Restarting                Task restarting in 30.956406963s
12/06/16 11:56:55 CEST  Failed Artifact Download  GET error: Get https://raw.githubusercontent.com/FRosner/nomad-docker-wrapper/master/nomad-docker-wrapper: dial tcp 185.31.17.133:443: i/o timeout
12/06/16 11:56:25 CEST  Downloading Artifacts     Client is downloading artifacts
12/06/16 11:55:59 CEST  Restarting                Task restarting in 26.198607406s
12/06/16 11:55:59 CEST  Failed Artifact Download  GET error: Get https://raw.githubusercontent.com/FRosner/nomad-docker-wrapper/master/nomad-docker-wrapper: dial tcp 23.235.43.133:443: i/o timeout
12/06/16 11:55:29 CEST  Downloading Artifacts     Client is downloading artifacts
12/06/16 11:55:00 CEST  Restarting                Task restarting in 28.654117125s

However, when I try to download it manually, it works:

# curl https://raw.githubusercontent.com/FRosner/nomad-docker-wrapper/master/nomad-docker-wrapper
#!/bin/bash

NOMAD_DOCKER_COMMAND=$*

if [ -z "$NOMAD_DOCKER_CONTAINER_NAME" ]; then
  echo 'Please set $NOMAD_DOCKER_CONTAINER_NAME'
  exit
fi

if [ -z "$NOMAD_DOCKER_COMMAND" ]; then
  echo 'Please set $NOMAD_DOCKER_COMMAND'
  exit
fi
 
echo "Stopping container $NOMAD_DOCKER_CONTAINER_NAME"
docker stop $NOMAD_DOCKER_CONTAINER_NAME 

echo "Removing container $NOMAD_DOCKER_CONTAINER_NAME"
docker rm $NOMAD_DOCKER_CONTAINER_NAME

echo "Starting container $NOMAD_DOCKER_CONTAINER_NAME"
exec docker run \
  --name $NOMAD_DOCKER_CONTAINER_NAME \
  $NOMAD_DOCKER_COMMAND

There is a proxy, which is set with http_proxy and https_proxy variables.

I am not sure what might cause these timeouts. Can it be that go-getter is not picking up proxy settings? I tried to pass http(s) proxy variables in the env section of the job, but it did not help.

I hope someone can help me with this issue.

Thank you in advance,

Aleksandr

Alex Dadgar

unread,

Jun 12, 2016, 10:53:01 AM6/12/16

to Aleksandr Sorokoumov, Nomad

Hey can you file an issue with go-getter. I would not be surprised if it didn't look at the proxy environment variables.

Thanks,

Alex

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/77f48249-cc4b-449a-bf50-2cb4a17339bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aleksandr Sorokoumov

unread,

Jun 13, 2016, 3:56:59 AM6/13/16

to Nomad

Hi Alex!

Thank you for the response. I created an issue in go-getter: https://github.com/hashicorp/go-getter/issues/27

Best regards,

Aleksandr

Mitchell Hashimoto

unread,

Jun 13, 2016, 4:08:34 AM6/13/16

to Aleksandr Sorokoumov, Nomad

Hey,

I see your issue and will respond there, but go-getter does indeed support these and check for them (the underlying Go HTTP client does this automatically). I'll try to write a test and verify that is the case and if not, fix it.

Best,

Mitchell

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/7bf25404-5aef-4fd4-8c98-0f3c28ea8277%40googlegroups.com.

Mitchell Hashimoto

unread,

Jun 13, 2016, 4:36:35 AM6/13/16

to Aleksandr Sorokoumov, Nomad

Just a follow up for the mailing list...

I've verified go-getter respects these env vars: https://github.com/hashicorp/go-getter/issues/27

So it appears to be something else.

Diptanu Choudhury

unread,

Jun 13, 2016, 5:13:06 AM6/13/16

to Mitchell Hashimoto, Aleksandr Sorokoumov, Nomad

Hi Aleksandr,

The artifacts get downloaded by the Nomad client process. Can you please share with us what environment variables the Nomad client process is inheriting?

I am hoping Nomad client sees your http_proxy related env vars.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/CACOXMX7-5PUNF_GYtDmUFMDWs7rokREJZ7W3CVLk8--p_sCPfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

Aleksandr Sorokoumov

unread,

Jun 13, 2016, 5:27:29 AM6/13/16

to Nomad

Hi Diptanu,

After your question about the Nomad client's environment variables, I double checked them and found out that indeed http(s)_proxy were not inherited. After setting them for the process, the problem was solved.

Thank you for the support on this issue!

Best regards,

Aleksandr

Reply all

Reply to author

Forward