"no such host" with many get requests

[stpr...@gmail.com]

Please forgive the newb-ness of this question.

I ran into the "no such host" issue described at https://code.google.com/p/go/issues/detail?id=3575#makechanges. I was told to post here.

In short, I tried the "go install -a -tags netgo net my-pkg" workaround but get a permission denied error. I eventually reinstalled go as as that led to other errors (see the post for more details).

Another proposed solution there is to mutex net.cgoLookupIPCNAME. How do I modify http://play.golang.org/p/C1WhYh4_jl to lock net.cgoLookupIPCNAME? (I do realize it won't run on the playground)

[James Bardin]

On Monday, November 24, 2014 1:34:21 PM UTC-5, stpr...@gmail.com wrote:

Please forgive the newb-ness of this question.

I ran into the "no such host" issue described at https://code.google.com/p/go/issues/detail?id=3575#makechanges. I was told to post here.

In short, I tried the "go install -a -tags netgo net my-pkg" workaround but get a permission denied error. I eventually reinstalled go as as that led to other errors (see the post for more details).

This is because you don't have permission to write to GOROOT. That command is trying to install the net package, which is part of the std library.

 

Another proposed solution there is to mutex net.cgoLookupIPCNAME. How do I modify http://play.golang.org/p/C1WhYh4_jl to lock net.cgoLookupIPCNAME? (I do realize it won't run on the playground)

You can't. You would need to patch net/cgo_unix.go. (try the former workaround first though)

 

[stpr...@gmail.com]

"This is because you don't have permission to write to GOROOT".

I don't have GOROOT set since I installed Go in the default location.
From the docs "Note: GOROOT must be set only when installing to a custom location."

[James Bardin]

On Mon, Nov 24, 2014 at 4:50 PM, <stpr...@gmail.com> wrote:

"This is because you don't have permission to write to GOROOT".
I don't have GOROOT set since I installed Go in the default location.
From the docs "Note: GOROOT must be set only when installing to a custom location."

Even though you're not overriding it, there's always a GOROOT. 

You can see all the variable settings with `go env`

[stpr...@gmail.com]

I see....Now I guess I'm just confused because in the Go install guide it says "Typically these commands must be run as root or through sudo.)". So, I should NOT be installing go as root? Sorry for being thick...it's just the way I am ;)

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Dave Cheney]

You should not try to rebuild your go distributon (-a) unless you've installed it from source, otherwise you tend to hit the permission issues you've hit.

I think it might be worth investigsting other avenues before performing surgery on your go install.

The most common cause of the error you are seeing is running out of file descriptors, or at least having more than 1024 of them open at any one time.

Does that sound like a possible avenue of investigation?

[stpr...@gmail.com]

I used the -a switch b/c that was what was suggested in the first link I linked to. I reinstalled everything from source. I am running only 100 and even with 25 or even 50 get the same errors.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Dave Cheney]

Can you share your code, or some code which demonstrates the issue? Please include details about the platform your are deploying on.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/-2E2pt6OuaU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[stpr...@gmail.com]

Sorry for the delay. I've attached an example...I'll warn you in advance that I am a Go newb ;)
I am running Linux Mint 17 64-bit, which is based on Ubuntu 14.04. I have a fiber connection so the 100 goroutines I launched at a time shouldn't be an issue in terms of bandwith. (Even if I change it to 25 or 1000 the same situation occurs). I've tried installing this the normal way as well as with "go install -tags netgo net testfolder/tester" and neither seem to work for me. My Go version is 1.3.3 linux/amd64.

I am experiencing 2 issues:

1. The 100 goroutines are launched in a never-ending loop and for about 15% of those 100 I get a "dial tcp: lookup example.com: no such host" error. Even though my timeouts are set at around 20-25 seconds currently this error occurs at around 15 seconds (In the code I log each error and the duration of the request). If I increase the timeout to 40-45 seconds then the no such host error seems to occur at about 20 seconds in (which seems a bit odd since I never increase anything from 15 to 20 seconds...I couldn't find a way to control that).

2. If I keep the crawler going for a few hours (in addition to the "no such host" errors) I'll start to experience a different error "net/http: request canceled while waiting for connection". Initially it will only affect a few of the goroutines but as time goes on more and more of the goroutines will experience this....until 100% of the 100 goroutines are affected and effectively no successful requests are returned. To me (a novice, keep in mind ;)) it seems that the connections aren't being released or something to that effect...I've tried calling "defer Tr.CancelRequest(req)" after each request but that doesn't seem to help.

I don't get any errors suggesting I am running out of file descriptors..."cat /proc/sys/fs/file-max" shows 911,302 (I haven't changed it).


thanks!

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Jason Playne]

DNS is a complicated beast - One thing to check is that you are not triggering some sort of DOS prevention thing with too many lookups too quickly?

can you replicate this problem on a local lan with a closed test environment (even if you are requesting the same page over and over again)?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Dave Cheney]

Thanks for posting your sample code.

1. I'm not an expert, but those Tr.CancelRequests are probably not going to help. I cannot give more specific advice than that.

2. The locking inside fetcher is wrong. You must protect every access to robots, not just when you are updating it. I recommend running your code under the race detector, go build -race ... , which should highligh the problem (although it's unlikely to be related to the other issues)

3. []byte(string(body)) where body was a []byte is redundant.

[stpr...@gmail.com]

Jason:
I thought that's what the "go install -tags netgo net testfolder/tester" was supposed to fix, no? Do you mean just requesting "http://127.0.0.1" over and over again?

Dave:
1. Correct, it doesn't seem to do anything. I was hoping it would fix issue #2 that I am having but unfortunately it doesn't ;(
2. Again, you are correct....there is a race condition. That's a good tip, especially for a newb.
3. You are 3 for 3.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Dave Cheney]

Do you want to clean up the example code and repost it, that will give
us a good starting point with some expected failure modes.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/-2E2pt6OuaU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[stpr...@gmail.com]

attached. thx!

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[stpr...@gmail.com]

Jason,

The weirdest thing about the entire thing is that the first few loops will be fine then I'll see more and more "no such host" and other errors as time goes on and then a few hours in there will be no requests that successfully complete. Makes me think the connections aren't being released or something to that effect but I am far from a networking expert.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Jason Playne]

stpra123,

The more you learn about networking - the more you will realise that there is a lot of random weird things that happen (some even for real reasons!)

What I wanted to know is if you remove the external network from the loop does the problem still happen? So yes, just requesting http://127.0.0.1/ would do the trick (no DNS to resolve, no external network to do weird things). It's all about reducing the number of things that can go wrong!

Cheers

Jason

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Andrew Bursavich]

I don't think this is the problem, but it looks like that CheckRedirect function lets requests get stuck in infinite redirect loops.

Cheers,

Andy

[stpr...@gmail.com]

Andrew: Very good catch. Explains why the connections weren't releasing. I've attached a version with a better CheckRedirect function.

I am still experiencing the "no such host errors"....it starts out fine but as time goes on the number of errors increases.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[stpr...@gmail.com]

Is there a way to enable DNS caching? I imagine that would reduce the number of DNS lookups & probably reduce the issues we're experiencing.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Benjamin Measures]

On Sunday, 7 December 2014 20:27:00 UTC, stpr...@gmail.com wrote:

Is there a way to enable DNS caching?

You can install a caching name server. Personally, I use pdnsd.

[Andrew Bursavich]

You'll still need to address the root problem, but (shameless plug) I wrote a package with simple DNS caching a while back.

Cheers,

Andy

[stpr...@gmail.com]

Thanks for the links! The DNS caching does help quite a bit but, again, after a few hours of running it will start returning 0 successful requests as before.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[stpr...@gmail.com]

Here is an updated version with dns caching. For me this runs for just over 2 hours then all requests return "net/http: request canceled while waiting for connection". Installing with "go install -tags netgo net testfolder/tester" doesn't seem to have an effect.

I'm completely out of ideas.

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Dave Cheney]

Can you capture the output of lsof -p $(the pid of your program) when
it get's into that condition ?

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/-2E2pt6OuaU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[stpr...@gmail.com]

Here's what I have when it starts returning 0 successful requests:

COMMAND   PID          USER   FD      TYPE DEVICE SIZE/OFF      NODE NAME
tester  15810 mehere  cwd       DIR    8,1     4096    786488 /home/mehere/Desktop
tester  15810 mehere  rtd       DIR    8,1     4096         2 /
tester  15810 mehere  txt       REG    8,1  6786424  24252069 /home/mehere/Desktop/go/bin/tester
tester  15810 mehere  mem       REG    8,1   101240  16256872 /lib/x86_64-linux-gnu/libresolv-2.19.so
tester  15810 mehere  mem       REG    8,1    22952  16256942 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
tester  15810 mehere  mem       REG    8,1    10432  16256954 /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
tester  15810 mehere  mem       REG    8,1    47712  16258751 /lib/x86_64-linux-gnu/libnss_files-2.19.so
tester  15810 mehere  mem       REG    8,1  1845024  16256946 /lib/x86_64-linux-gnu/libc-2.19.so
tester  15810 mehere  mem       REG    8,1   141574  16258757 /lib/x86_64-linux-gnu/libpthread-2.19.so
tester  15810 mehere  mem       REG    8,1   149120  16256948 /lib/x86_64-linux-gnu/ld-2.19.so
tester  15810 mehere    0u      CHR  136,0      0t0         3 /dev/pts/0
tester  15810 mehere    1u      CHR  136,0      0t0         3 /dev/pts/0
tester  15810 mehere    2u      CHR  136,0      0t0         3 /dev/pts/0
tester  15810 mehere   13u     0000    0,9        0      7348 anon_inode
tester  15810 mehere   14r      CHR    1,9      0t0      1034 /dev/urandom
tester  15810 mehere   16u  netlink             0t0 112965001 ROUTE

When I install with "go install -tags netgo net testfolder/tester" I get:

COMMAND   PID          USER   FD      TYPE DEVICE SIZE/OFF      NODE NAME
tester  26055 mehere  cwd       DIR    8,1     4096    786488 /home/mehere/Desktop
tester  26055 mehere  rtd       DIR    8,1     4096         2 /
tester  26055 mehere  txt       REG    8,1  6786424  24252276 /home/mehere/Desktop/go/bin/tester
tester  26055 mehere  mem       REG    8,1   101240  16256872 /lib/x86_64-linux-gnu/libresolv-2.19.so
tester  26055 mehere  mem       REG    8,1    22952  16256942 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
tester  26055 mehere  mem       REG    8,1    10432  16256954 /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
tester  26055 mehere  mem       REG    8,1    47712  16258751 /lib/x86_64-linux-gnu/libnss_files-2.19.so
tester  26055 mehere  mem       REG    8,1  1845024  16256946 /lib/x86_64-linux-gnu/libc-2.19.so
tester  26055 mehere  mem       REG    8,1   141574  16258757 /lib/x86_64-linux-gnu/libpthread-2.19.so
tester  26055 mehere  mem       REG    8,1   149120  16256948 /lib/x86_64-linux-gnu/ld-2.19.so
tester  26055 mehere    0u      CHR  136,0      0t0         3 /dev/pts/0
tester  26055 mehere    1u      CHR  136,0      0t0         3 /dev/pts/0
tester  26055 mehere    2u      CHR  136,0      0t0         3 /dev/pts/0
tester  26055 mehere    6u     0000    0,9        0      7348 anon_inode
tester  26055 mehere   10u  netlink             0t0 120913976 ROUTE
tester  26055 mehere   14r      CHR    1,9      0t0      1034 /dev/urandom

On Monday, November 24, 2014 11:34:21 AM UTC-7, stpr...@gmail.com wrote:

[Dave Cheney]

At a guess, your program is still using the libc resolver. strace -f
might give a clue there, but the application is not statically
compiled so may still be using the libc resolver.

[stpr...@gmail.com]

I ran strace -t -o tester.txt -f -p <pid>....here's a link:
https://drive.google.com/folderview?id=0B39eHMkMpSGNQlhaSVhZVWNIQkE&usp=sharing

This is from when I started the crawler until the end...the file is 147MB and called "tester.txt". I also split the file into 1MB chunks...chunk_aa is the beginning and chunk_fk is the end.

I have no idea what any of it means ;)

[Dave Cheney]

Yup, that program is still using the libc resolver. That may or may
not be the cause of the problem.

It may also be that -netgo is ineffective now.

[Dave Cheney]

It looks like -tags netgo doesn't do what is expected. I am sorry I cannot offer more specific advice, I don't use that procedure to disable cgo.

See also https://groups.google.com/forum/#!topic/golang-nuts/S2WDcm47bhA

lucky(~/src) % go build -tags netgo netgo.go

lucky(~/src) % ldd ./netgo

        linux-vdso.so.1 =>  (0x00007fffd65fe000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f19e2dc3000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f19e29fd000)

        /lib64/ld-linux-x86-64.so.2 (0x00007f19e3001000)

lucky(~/src) % cat netgo.go 

package main

import "net"

import "fmt"

func main() {

        conn, err := net.Dial("tcp", "www.google.com:443")

        fmt.Println(conn, err)

}

[Jason Playne]

Completely silly questions 

Are there any ulimit's in place for your process?

Could it be a file handle leaking thing?

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[stpr...@gmail.com]

Dave - I have tried it with 1.3.3 and 1.4...I don't think that link applies to me (unless I'm mistaken...I'm still a newb)
Jason - Typing "ulimit" on command prompt returns "unlimited"....not sure how to test a file leaky thing ;)

[Tamás Gulácsi]

Type ulimit -a.

To test, lower your file descriptor limit, and see if the program bails out earlier, or not.

[stpr...@gmail.com]

I'm not quite sure how to limit the number of file descriptors...got conflicting info on a quick google search (I just want to be crystal clear on the commands I run and what you guys *think* I am issuing ;).

ulimit -a gives:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71632
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71632
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[Jason Playne]

This is the line you need to be looking at - the number of file descriptors being used.

open files                      (-n) 1024

it says that any executable you start can open 1024 file descriptors.

If you set this value (as Tamas suggested) down to something like 100, if it is a ulimit issue with the number of open files for a process, it should fail a lot quicker.

--

[James Bardin]

If you build the application with the Go resolver, then we can eliminate the libc resolver as a possibility, and stop a lot of guessing. 

Since the netgo tag doesn't work right now, install Go with CGO_ENABLED=0, and rebuild your application. 

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/-2E2pt6OuaU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[stpr...@gmail.com]

Jason,
Reducing the number of open files ("ulimit -n 100") then rebuilding doesn't change the outcome. In the beginning I'll get a couple of the 100 requests returning "too many open files" (as expected). It will run for a couple of hours as it did before and then crap out with the "request canceled while waiting for connection" error on 100% of the responses.

James,
That also doesn't seem to make any difference, sadly. (Since I am new just want to make sure I did things correctly: I open a terminal, type "export CGO_ENABLED=0" then rebuild & run the crawler from that same terminal window)

[James Bardin]

On Friday, December 19, 2014, <

James,
That also doesn't seem to make any difference, sadly. (Since I am new just want to make sure I did things correctly: I open a terminal, type "export CGO_ENABLED=0" then rebuild & run the crawler from that same terminal window)

Did you install your go tool chain with CGO_ENABLED=0? Did you verify that you had a statically linked executable?

[stpr...@gmail.com]

James,

I'm not sure how to do that. (Newb....).

thx

[Dave Cheney]

So, via issue 9405 it may appear that the errors you are receiving are actually timeouts. Can you log the failing URL? There may be some commonality.

--

[stpr...@gmail.com]

Dave,

All the urls are failing.

[stpr...@gmail.com]

I should clarify. In the beginning only a small amount (sometimes zero) of the urls fail with the "request canceled while waiting for connection" error. As it runs over a couple of hours then gradually more urls will start to fail with the "request canceled while waiting for connection"..starts with 1 then a couple and then more and more. Run it for a couple of hours and eventually all of the 100 requests will return "request canceled while waiting for connection".

I found a windows machine the other day, installed go 1.4 and ran it there....ran for the entire day & night with no issues.

Is anyone else having the same issue on Mint/Ubuntu or other flavors of Linux?

[James Bardin]

On Sat, Dec 20, 2014 at 11:18 AM, <stpr...@gmail.com> wrote:

Is anyone else having the same issue on Mint/Ubuntu or other flavors of Linux?

I still think there's a good chance it's an issue with the resolver on your host (though I'm not sure what that problem might be).

To install go without cgo (e.g. in $HOME)

~$ git clone https://github.com/golang/go.git

...

~$ cd go/src/

~/go/src$ git checkout go1.4

...

~/go/src$ CGO_ENABLED=0 ./make.bash

...

# And verifying with a minimal package named "resolve" that imports "net" to lookup a hostname

~$ rm bin/resolve

~$ ./go/bin/go install resolve

~$ ldd bin/resolve

not a dynamic executable

[Jason Playne]

I am just idly wondering if this is related to the other "http.client FD (File Descriptor leak" thread kicking around on the list at the moment?

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[stpr...@gmail.com]

Hey James,

I followed your instructions but "ldd bin/resolve" keeps giving me:

    linux-vdso.so.1 =>  (0x00007fffda7fe000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f802266e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f80222a8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f80228aa000)

[James Bardin]

On Sun, Dec 21, 2014 at 12:41 PM, <stpr...@gmail.com> wrote:

I followed your instructions but "ldd bin/resolve" keeps giving me:

Sorry, `export CGO_ENABLED=0` first instead of just setting when invoking the build script. That way it's there for the entire build process, and when you build the executable.

[stpr...@gmail.com]

James,

That was it. Running it at high speed now and been running very well.

Does this just affect Ubuntu or other linux distros as well?

[James Bardin]

On Wed, Dec 24, 2014 at 11:27 AM, <stpr...@gmail.com> wrote:

Does this just affect Ubuntu or other linux distros as well?

I haven't had time to try and diagnose what the problem might be. Whatever is happening, it probably can affect any system with the same libc. 

[jo...@ryron.net]

I think this problem is about your cross platform compile.

I am compile in window to run in Linux cause this problem.

Do you have any suggestion  to solve  this problem yet? Thanks

[James Bardin]

This problem was solved by disabling the linux cgo resolver, which you wouldn't have if you're cross-compiling from windows. I would start a new thread showing your exact problem.