Application unreachable (by some users) by Google Apps domain name set with CNAME to ghs.google.com

[Janusz Skonieczny]

Hi,

I have a problem with DNS resolution for http://app.dziennik.edu.pl 

Some users seem to have trouble to find a Google App Engine server for this domain name. AFAIK the CNAME is set up correcly, it is registered in the Google Apps admin panel and there was no changes made recently so it should not be a TTL issue.

When asked, my DNS provider have double checked DNS entries and did not find anything wrong, yet I the server is not found.

I am hardly qualified to troubleshoot this and people that have the problem are just regular users. I do not have direct access to any of the machines having problems so any lame checks I did think of, I had to ask people to run by chat/phone/email. 

Can anybody help me?

PS. I have at least two other applications having the same problems.

[Gwyn Howell]

All you need to do is create a cname record for 'app' to point to ghs.google.com (did a dns lookup - seems you have already done that). Then, log into your Google Apps domain CPanel (www.google.com/a/[your-domain]) and click Add More Services, then enter the App ID for your App Engine in the "Enter App ID" field. When you have done that, click Add new URL, and enter the new url.

Note that this will not work if you have configured all your app engine handlers to work over SSL only

[Janusz Skonieczny]

I have done all that a while ago and everything worked fine (at least no one was complaining). This problem only appeared recently (maybe two weeks ago) as users started to that server is unreachable from some locations, like it is working from the office but not from home.

App is working with vanilla HTTP — I understand that HTTPS is only available with *.appspot.com domain name.

As I said, the only some users have this problem. The group is small, but significant, they are trying to reach the server form few networks (it's not just one location) . The solution: "Please use Google DNS servers in you network settings" is not the one I'm looking for ;)

[Gwyn Howell]

hmm, some kind of network/dns caching going on in those remote locations? proxy servers?

[Janusz Skonieczny]

I have asked for flushdns, it did not help. Anyway there was never anything other than ghs.google.com under app.dziennik.edu.pl CNAME, so these people could not have cached anything else.   

[Simon Knott]

It sounds to me like there is a common DNS server which your users are using, which is returning incorrect results.  Flushing their DNS wouldn't help, because all that flushes is the local PCs cache and as soon as it went to the DNS server it would retrieve the incorrect result again.

Get them to do a tracert and see where their DNS servers are actually pointing them.

[Janusz Skonieczny]

This a problem, cause their pointed nowhere. It's not a case of bad IP address or a proxy server looking in the wrong place. Its like there is no DNS entry for my domain.

Call to ipconfig /displaydns gives results like these:

Windows IP Configuration

    ghs.google.com

    ----------------------------------------

    Record Name . . . . . : ghs.google.com

    Record Type . . . . . : 5

    Time To Live  . . . . : 296

    Data Length . . . . . : 8

    Section . . . . . . . : Answer

    CNAME Record  . . . . : ghs.l.google.com

    app.dziennik.edu.pl

    ----------------------------------------

    Name does not exist.

[Simon Knott]

That means that their company's internal DNS server, or the user's ISP's DNS server, isn't resolving your name.

What DNS server are they pointing to?  It looks like their server's cached a bad lookup and hasn't refreshed.

[Janusz Skonieczny]

But how It could have cached a bad lookup? The CNAME was set to ghs.google.com from day one.

Here an example network settings: 

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . : chello.pl

   Description . . . . . . . . . . . : Realtek PCIe GBE Family Controller

   Physical Address. . . . . . . . . : [...]

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Link-local IPv6 Address . . . . . : [...]

   IPv4 Address. . . . . . . . . . . : [...]

   Subnet Mask . . . . . . . . . . . : 255.255.254.0

   Lease Obtained. . . . . . . . . . : 24 wrze˜nia 2011 09:33:25

   Lease Expires . . . . . . . . . . : 30 wrze˜nia 2011 06:09:11

   Default Gateway . . . . . . . . . : 87.207.162.1

   DHCP Server . . . . . . . . . . . : 10.131.40.1

   DHCPv6 IAID . . . . . . . . . . . : 249613134

   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-13-77-89-56-E0-CB-4E-81-94-81

   DNS Servers . . . . . . . . . . . : 62.179.1.63

                                       62.179.1.62 

[Mat Jaggard]

DNS servers can cache negative results as well as positive - albeit
normally for a shorter time. It seems that 62.179.1.63 or 62 are
caching or a pointing to another DNS server that's caching the fact
that app.dziennik.edu.pl was unresolvable.

On Sep 30, 4:41 pm, Janusz Skonieczny <janusz.skoniec...@gmail.com>
wrote:

[Janusz Skonieczny]

What can be the reason whey a  negative result was cached?

[Simon Knott]

Negative results are cached for exactly the same reasons as positive ones - performance.  If a domain is unresolvable, then it's very likely that in 5 minutes time it will still be unresolvable, so the negative result is stored for a period of time to remove the need to do the lookup again.

As Mat stated, negative results are usually cached for shorter periods of time.  However, unless you have some way of getting those DNS servers flushed, then you're a bit stuck.  It should be noted that the servers that are causing the problem may not be the ones from that IP Config - those DNS servers will be carrying out their lookups against another set of DNS servers.

[Janusz Skonieczny]

What I meant to ask was, what have caused those servers to cache a negative result. 

The app.dziennik.edu.pl domain was a CNAME to ghs.google.com from the start, and it is set-up for a couple of months now. My other domain that have the same issues www.bravelabs.pl have been setup and untouched for at least 10+ months. 

Where is the fault? Is it mine? How can I make sure this will go away and never come back?

PS. It seems it may be the issue with google infrastructure: 
https://groups.google.com/d/topic/google-appengine/zjEeCeyhN20/discussion

[Simon Knott]

I don't know what could have caused the blip to be honest - network issues are always are pain to diagnose, especially when you have no control over most of the infrastructure.

Who is hosting your CNAME records?  If there was an underlying issue with Google's network I think they'd be a lot of people moaning that their hosts were unreachable!  I'm not saying they aren't at fault, I just haven't seen any other complaints recently :) Given that you have the same issue on multiple domains, there must be something that's common to both sites.

[Aidan O'Kelly]

Are all the affected users/locations using one ISP. ? I would try and
isolate where the DNS is failing, its probably one DNS server that all
of these locations use, and its having trouble for whatever reason.
Asking your users to use google's dns may be your only option. (or
possibly you could send them a HOSTS file with the relevant
ip/hostnames)

On Fri, Sep 30, 2011 at 3:41 PM, Gwyn Howell <gwyn....@appogee.co.uk> wrote:
> hmm, some kind of network/dns caching going on in those remote locations?
> proxy servers?
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/VQ8TgYidl_gJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[Alexander Konovalenko]

On Fri, Sep 30, 2011, Janusz Skonieczny <janusz.s...@gmail.com> wrote:
> What I meant to ask was, what have caused those servers to cache a negative
> result.
> The app.dziennik.edu.pl domain was a CNAME to ghs.google.com from the start,
> and it is set-up for a couple of months now. My other domain that have the
> same issues www.bravelabs.pl have been setup and untouched for at least 10+
> months.
> Where is the fault? Is it mine? How can I make sure this will go away and
> never come back?

If I understand correctly, the problem arises when some clients can't
get the IP address of app.dziennik.edu.pl and thus can't find the
site.

It looks like one of your name servers, ns12.az.pl (217.153.158.180),
is flaky and sometimes does not send a timely response for DNS
requests. It experiences ping packet loss around 18% from my location
(Almaty, Kazakhstan). Theoretically, that shouldn't cause any harm
because your other name servers seem to work well, but who knows.

Another problem (probably irrelevant) that I discovered is that in
your DNS registrar settings a wrong IP address appears for ns12.az.pl.
See whois dziennik.edu.pl. The server at 89.171.29.77 does not respond
to DNS queries about your dziennik.edu.pl zone at all.

The ultimate way to troubleshoot this problem is to get remote access
to a user's machine where the problem reproducibly occurs and to debug
it from there. What local DNS resolvers does the machine use? Does the
same problem occur with other sites CNAMEd to ghs.google.com? Do your
name servers see any requests from the local resolvers? Do they see
any requests when the problem with your site occurs? If you cannot
view full access logs or packet traces on your name servers, it can be
hard to answer some of these questions.

It might help to know what is the local resolvers' cache policy and
whether they comply with server-supplied TTL and negative TTL or
ignore your settings and cache for a fixed period of time (like 24 or
48 hours). What does the resolver do when a request to a name server
times out? Does it try other name servers responsible for the same
zone? (For instance, if a request to ns12.az.pl times out, does it try
to contact ns10.az.pl or ns11.az.pl?) When a request times out, does
it cache the negative result and for how long?

Once you determine where exactly the problem occurs, you will be on
your way to fix it. Some things that might go wrong:

1. A client's DNS resolver doesn't always get the proper response from
your name server when it asks it for the IP address of
app.dziennik.edu.pl.

2. A client's DNS resolver doesn't always get the proper IP address of
ghs.google.com when it asks Google name servers for it.

3. Some client-side firewall or anti-virus software interferes with
the DNS request.

If you want quick advice, remove the ns12 nameserver from *both* your
DNS config *and* your DNS registrar settings. Verify your changes
using dig (or nslookup) and whois and wait for a day or two for all
possibly involved caches to update. Then see if the problem goes away.

Hope this helps.

Please let me know what you find out, I'm interested to hear.

-- Alexander

[Janusz Skonieczny]

Thx  for this very instructive response, I suck in networking issues so this is all very appreciated. 

I'll get right on it.

Regards. Janusz.