[Rocks-Discuss] DNS resolution problems

Ryan Golhar

unread,

Feb 20, 2012, 7:11:46 PM2/20/12

to Discussion of Rocks Clusters

Hi all - I have Rocks 5.4.3 running on a cluster....I've been having some
DNS resolution issues with it and just discovered that it is running named
and contains the local DNS server in /etc/resolve.conf (in addition to the
authoritative DNS servers). Is there a reason why the headnode is running
named? I assume this is to resolve the compute nodes names. Is this
correct?

If so, that means I can't simply stop named. So, how do I fix this problem
of DNS resolution going out to the other servers when the local DNS server
fails? My /etc/resolv.conf contains:

[ryang@bic init.d]$ cat /etc/resolv.conf
search mygroup.institution.edu
nameserver 127.0.0.1
nameserver 10.24.2.11
nameserver 10.24.2.12
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120220/ff83b0c9/attachment.html

Philip Papadopoulos

unread,

Feb 21, 2012, 12:19:18 AM2/21/12

to Discussion of Rocks Clusters

On Mon, Feb 20, 2012 at 4:11 PM, Ryan Golhar <ngsbioin...@gmail.com>wrote:

> Hi all - I have Rocks 5.4.3 running on a cluster....I've been having some
> DNS resolution issues with it and just discovered that it is running named
> and contains the local DNS server in /etc/resolve.conf (in addition to the
> authoritative DNS servers). Is there a reason why the headnode is running
> named? I assume this is to resolve the compute nodes names. Is this
> correct?
>

correct.

>
> If so, that means I can't simply stop named. So, how do I fix this problem
> of DNS resolution going out to the other servers when the local DNS server
> fails? My /etc/resolv.conf contains:
>

What is the particular problem? I'm not sure I understand what you think is
"broken"?

>
> [ryang@bic init.d]$ cat /etc/resolv.conf
> search mygroup.institution.edu
> nameserver 127.0.0.1
> nameserver 10.24.2.11
> nameserver 10.24.2.12
>

This looks correct for frontend.
resolv.conf is different for compute nodes.

> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120220/ff83b0c9/attachment.html
>
>

--
Philip Papadopoulos, PhD
University of California, San Diego
858-822-3628 (Ofc)
619-331-2990 (Fax)

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120220/646fab36/attachment.html

Ryan Golhar

unread,

Feb 21, 2012, 7:32:12 AM2/21/12

to Discussion of Rocks Clusters

>
> >
> > If so, that means I can't simply stop named. So, how do I fix this
> problem
> > of DNS resolution going out to the other servers when the local DNS
> server
> > fails? My /etc/resolv.conf contains:
> >
> What is the particular problem? I'm not sure I understand what you think is
> "broken"?

The problem is the headnode is unable to resolve DNS names. I noticed this
problem when running yum. For instance, pinging 'mirrorlist.centos.org'
fails to resolve the name. If I ping mirrorlist.centos.org from another
machine on the same subnet, I can then ping mirrorlist from my rocks
cluster.

I tested this other DNS names. My rocks cluster could only resolve those
names AFTER I resolved them on another machine.

Here is an example of it failing using dig:

[ryang@bic ~]$ dig mirrorlist.centos.org

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> mirrorlist.centos.org
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17307
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;mirrorlist.centos.org. IN A

;; Query time: 6 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 20 19:04:45 2012
;; MSG SIZE rcvd: 39

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/243cbdc1/attachment.html

Ian Kaufman

unread,

Feb 21, 2012, 11:42:52 AM2/21/12

to Discussion of Rocks Clusters

Out of curiosity, how is your cluster's private network set up? By
default, it uses 10.X.X.X, but if your extranet is also using 10.X.X.X
(which appears to be the case since the DNS servers are
10.24.2.[11,12], then you are going to have some routing issues.

Ian

--
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu

Ryan Golhar

unread,

Feb 21, 2012, 5:37:16 PM2/21/12

to Discussion of Rocks Clusters

Ah, our external network is also 10.x.x.x network, actually its on a
10.228.x.x subnet. The private network is 10.1.x.x. My routing table is:

[ryang@bic ~]$ /sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
10.228.81.46 10.1.1.1 255.255.255.255 UGH 0 0 0 eth0
255.255.255.255 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
224.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.228.81.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 10.228.81.254 0.0.0.0 UG 0 0 0 eth1

Doing a traceroute to mirrorlist.centos.org gives

[ryang@bic ~]$ traceroute mirrorlist.centos.org
traceroute to mirrorlist.centos.org (64.235.47.134), 30 hops max, 40 byte
packets
1 10.228.81.254 (10.228.81.254) 0.367 ms 0.393 ms 0.451 ms
2 10.0.7.66 (10.0.7.66) 0.558 ms 0.756 ms 0.933 ms
3 10.0.31.24 (10.0.31.24) 2.965 ms 2.770 ms 3.072 ms
...

so in this case it looks like routing is happening correctly.

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html

Ian Kaufman

unread,

Feb 21, 2012, 5:53:22 PM2/21/12

to Discussion of Rocks Clusters

Are you sure? Did the traceroute complete? Is 10.0.31.24 your gateway
out? Or did the traffic stop there?

Can you traceroute and ping the nameservers? Without first pinging an
external system on another host?

Is it possible to reinstall your cluster, but using 192.168.X.X? It
might help keep your networks discrete.

Ryan Golhar

unread,

Feb 21, 2012, 6:28:47 PM2/21/12

to Discussion of Rocks Clusters

I'm pretty sure it did. 10.228.81.254 is my gateway. 10.0.31.24 isn't the
last IP address, after that it prints * for the other entries.

I can ping the nameservers, that's fine. From everything I can tell,
routing is fine.. The cluster is in use at this point so rebuilding it
isn't really an option...maybe a last resort. I'd prefer to figure out
what's wrong before doing that.

Is there any configuration with the nameserver that needs to be done?

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/b95da03f/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 21, 2012, 6:29:50 PM2/21/12

to Discussion of Rocks Clusters

in-line

what are subnet mask of 10.228.x.x and 10.x.x.x
if both are 255.255.255.0 then your are ok
if both are 255.255.0.0 or 255.0.0.0 then you may have problem
in your case may be you should try 192.168.x.x are internal/private network

>
> [ryang@bic ~]$ /sbin/route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 10.228.81.46 10.1.1.1 255.255.255.255 UGH 0 0 0 eth0
> 255.255.255.255 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
> 224.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 10.228.81.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
> 10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
> 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
> 0.0.0.0 10.228.81.254 0.0.0.0 UG 0 0 0 eth1
>
> Doing a traceroute to mirrorlist.centos.org gives
>
> [ryang@bic ~]$ traceroute mirrorlist.centos.org
> traceroute to mirrorlist.centos.org (64.235.47.134), 30 hops max, 40 byte
> packets
> 1 10.228.81.254 (10.228.81.254) 0.367 ms 0.393 ms 0.451 ms
> 2 10.0.7.66 (10.0.7.66) 0.558 ms 0.756 ms 0.933 ms
> 3 10.0.31.24 (10.0.31.24) 2.965 ms 2.770 ms 3.072 ms
> ...
>
> so in this case it looks like routing is happening correctly.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html

--
Hung-Sheng Tsao Ph D.
Founder& Principal
HopBit GridComputing LLC
cell: 9734950840

http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available
Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/06e39f50/laotsao.vcf

Ryan Golhar

unread,

Feb 22, 2012, 12:42:57 PM2/22/12

to Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D., Discussion of Rocks Clusters

>>>> ;<<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.**el5<<>> mirrorlist.centos.org

>> URL: https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>> discussion/attachments/**20120221/03265073/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html>

>>
>
> --
> Hung-Sheng Tsao Ph D.
> Founder& Principal
> HopBit GridComputing LLC
> cell: 9734950840
>
> http://laotsao.blogspot.com/
> http://laotsao.wordpress.com/

> http://blogs.oracle.com/**hstsao/ <http://blogs.oracle.com/hstsao/>

>
>
what are subnet mask of 10.228.x.x and 10.x.x.x
if both are 255.255.255.0 then your are ok
if both are 255.255.0.0 or 255.0.0.0 then you may have problem
in your case may be you should try 192.168.x.x are internal/private network

Good question. eth0 (private network) is 255.255.0.0 eth1(public) is
255.255.255.0.

Ok, I'll buy that. I'll have to take a look at the docs to figure out how
to reconfigure the private network. Thanks everyone for your help,

Ryan

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/a33873b5/attachment.html

Ryan Golhar

unread,

Feb 22, 2012, 2:21:28 PM2/22/12

to Discussion of Rocks Clusters

On Wed, Feb 22, 2012 at 12:42 PM, Ryan Golhar
<ngsbioin...@gmail.com>wrote:

Ok, I've reconfigured the private network on my headnode.

[root@bic RCS]# rocks list network
NETWORK SUBNET NETMASK MTU DNSZONE SERVEDNS
private: 192.168.1.0 255.255.255.0 1500 local True
public: 10.228.0.0 255.255.0.0 1500 cag.chop.edu False

[root@bic RCS]# rocks list host interface
HOST SUBNET IFACE MAC IP NETMASK
MODULE NAME VLAN OPTIONS CHANNEL
bic: private eth0 00:22:19:21:5E:69 192.168.1.1 255.255.255.0
------ bic ---- ------- -------
bic: public eth1 00:22:19:21:5E:6B 10.228.81.46 255.255.0.0
------ bic ---- ------- -------
compute-0-0: private eth0 00:22:19:53:12:73 192.168.1.254 255.255.255.0
------ compute-0-0 ---- ------- -------
compute-0-0: ------- eth1 00:22:19:53:12:75 ------------- -------------
------ ----------- ---- ------- -------
compute-0-1: private eth0 00:22:19:51:db:9b 192.168.1.253 255.255.255.0
------ compute-0-1 ---- ------- -------
compute-0-1: ------- eth1 00:22:19:51:db:9d ------------- -------------
------ ----------- ---- ------- -------
compute-0-2: private eth0 00:22:19:51:b9:11 192.168.1.252 255.255.255.0
------ compute-0-2 ---- ------- -------
compute-0-2: ------- eth1 00:22:19:51:b9:13 ------------- -------------
------ ----------- ---- ------- -------

Ok looks like everything is good. Now, I'm trying to rebuild my compute
nodes. My compute nodes DHCP boots and picks up its new 192.168.1.x ip
address, but when it tries to establish the tftp connection, its just
hangs. I checked my /etc/xinet.d/tftp config file but there is not
indication anything is wrong. netstat reports tftp is running:

[root@bic xinetd.d]# netstat -a | grep tftp
udp 0 0 *:tftp *:*

I can even tftp to it from the headnode:

[root@bic xinetd.d]# tftp 192.168.1.1
tftp>

There is nothing in my iptables indicating a problem either. So, I'm
stuck. How do I get my compute nodes to tftp boot. Am I missing something?

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/6126b940/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 22, 2012, 2:28:48 PM2/22/12

to Discussion of Rocks Clusters

did you run
rocks sync hosts
rocks sync config

http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/f0ce9fa4/laotsao.vcf

Ryan Golhar

unread,

Feb 22, 2012, 2:34:23 PM2/22/12

to Discussion of Rocks Clusters

I just did that and see rebooted a compute node. I see its now hanging on
dhcp. I checked my /etc/dhcpd.conf file and it indeed not pick up the new
subnet. It still lists the old one, so I missed something.

On Wed, Feb 22, 2012 at 2:28 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <
lao...@gmail.com> wrote:

> did you run
> rocks sync hosts
> rocks sync config
>
>
>
> On 2/22/2012 2:21 PM, Ryan Golhar wrote:
>
>> On Wed, Feb 22, 2012 at 12:42 PM, Ryan Golhar

>> <ngsbioin...@gmail.com>**wrote:

>>>>>>> ;<<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.****el5<<>>

>>>>> URL: https://lists.sdsc.edu/****pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>>>> discussion/attachments/****20120221/03265073/attachment.****html<

>>>>> https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>>>> discussion/attachments/**20120221/03265073/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html>
>>>>> >
>>>>>
>>>>> --
>>>> Hung-Sheng Tsao Ph D.
>>>> Founder& Principal
>>>> HopBit GridComputing LLC
>>>> cell: 9734950840
>>>>
>>>> http://laotsao.blogspot.com/
>>>> http://laotsao.wordpress.com/

>>>> http://blogs.oracle.com/****hstsao/ <http://blogs.oracle.com/**hstsao/>
>>>> <http://blogs.oracle.**com/hstsao/ <http://blogs.oracle.com/hstsao/>>

>> URL: https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>> discussion/attachments/**20120222/6126b940/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/6126b940/attachment.html>

>>
>
> --
> Hung-Sheng Tsao Ph D.
> Founder& Principal
> HopBit GridComputing LLC
> cell: 9734950840
>
> http://laotsao.blogspot.com/
> http://laotsao.wordpress.com/

> http://blogs.oracle.com/**hstsao/ <http://blogs.oracle.com/hstsao/>
>
>
-------------- next part --------------

An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/5402107c/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 22, 2012, 2:59:59 PM2/22/12

to Discussion of Rocks Clusters

what are the output of
rocks list attr
rocks list hosts interface

-------------- next part --------------

A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/8b58244a/laotsao.vcf

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 22, 2012, 3:02:10 PM2/22/12

to Discussion of Rocks Clusters

rocks list host attr

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/d5525c91/laotsao.vcf

Ryan Golhar

unread,

Feb 22, 2012, 3:09:01 PM2/22/12

to Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D., Discussion of Rocks Clusters

I just went through all the entries from 'rocks list attr' and 'rocks list
host attr' and sure enough there were remnants of '10.1.1.1'. I fixed them
all, did a 'rocks sync config', rebooted a compute node and its
kickstarting now. Perfect! Thanks for your help!

On Wed, Feb 22, 2012 at 3:02 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <
lao...@gmail.com> wrote:

> rocks list host attr
>
>
> On 2/22/2012 2:59 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." wrote:
>
>> what are the output of
>> rocks list attr
>> rocks list hosts interface
>>
>>
>> On 2/22/2012 2:34 PM, Ryan Golhar wrote:
>>
>>> I just did that and see rebooted a compute node. I see its now hanging
>>> on
>>> dhcp. I checked my /etc/dhcpd.conf file and it indeed not pick up the
>>> new
>>> subnet. It still lists the old one, so I missed something.
>>>
>>> On Wed, Feb 22, 2012 at 2:28 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."<
>>> lao...@gmail.com> wrote:
>>>
>>> did you run
>>>> rocks sync hosts
>>>> rocks sync config
>>>>
>>>>
>>>>
>>>> On 2/22/2012 2:21 PM, Ryan Golhar wrote:
>>>>
>>>> On Wed, Feb 22, 2012 at 12:42 PM, Ryan Golhar

>>>>> <ngsbioin...@gmail.com>****wrote:

>>>>>>>>>> ;<<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.******el5<<>>

>>>>>>>> URL: https://lists.sdsc.edu/******pipermail/npaci-rocks-**<https://lists.sdsc.edu/****pipermail/npaci-rocks-**>
>>>>>>>> <https**://lists.sdsc.edu/**pipermail/**npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>>>>>>> >
>>>>>>>> discussion/attachments/******20120221/03265073/attachment.***
>>>>>>>> ***html<

>>>>>>>> https://lists.sdsc.edu/****pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>>>>>>> discussion/attachments/****20120221/03265073/attachment.****html<
>>>>>>>> https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>>>>>>> discussion/attachments/**20120221/03265073/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>> Hung-Sheng Tsao Ph D.
>>>>>>> Founder& Principal
>>>>>>> HopBit GridComputing LLC
>>>>>>> cell: 9734950840
>>>>>>>
>>>>>>> http://laotsao.blogspot.com/
>>>>>>> http://laotsao.wordpress.com/

>>>>>>> http://blogs.oracle.com/******hstsao/<http://blogs.oracle.com/****hstsao/>
>>>>>>> <http://blogs.oracle.**com/**hstsao/<http://blogs.oracle.com/**hstsao/>>
>>>>>>>
>>>>>>> <http://blogs.oracle.**com/**hstsao/<http://blogs.oracle.**
>>>>>>> com/hstsao/ <http://blogs.oracle.com/hstsao/>>>

>>>>> URL: https://lists.sdsc.edu/****pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>>>> discussion/attachments/****20120222/6126b940/attachment.****html<

>>>>> https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>>>> discussion/attachments/**20120222/6126b940/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/6126b940/attachment.html>>
>>>>>
>>>>>
>>>>> --
>>>> Hung-Sheng Tsao Ph D.
>>>> Founder& Principal
>>>> HopBit GridComputing LLC
>>>> cell: 9734950840
>>>>
>>>> http://laotsao.blogspot.com/
>>>> http://laotsao.wordpress.com/

>>>> http://blogs.oracle.com/****hstsao/ <http://blogs.oracle.com/**hstsao/>
>>>> <http://blogs.oracle.**com/hstsao/ <http://blogs.oracle.com/hstsao/>>
>>>>
>>>>

>>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: https://lists.sdsc.edu/**pipermail/npaci-rocks-**

>>> discussion/attachments/**20120222/5402107c/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/5402107c/attachment.html>

>>>
>>
>>
> --
> Hung-Sheng Tsao Ph D.
> Founder& Principal
> HopBit GridComputing LLC
> cell: 9734950840
>
> http://laotsao.blogspot.com/
> http://laotsao.wordpress.com/
> http://blogs.oracle.com/**hstsao/ <http://blogs.oracle.com/hstsao/>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/01bf04ea/attachment.html

Ryan Golhar

unread,

Feb 23, 2012, 3:24:47 PM2/23/12

to Discussion of Rocks Clusters

I noticed now on my compute nodes that /etc/host contains the headnode's
public IP address instead of the headnode's private IP address.

Shouldn't the /etc/host on the compute nodes contain the private IP address
of the headnode instead of the public IP address?

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/f6adecf0/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 23, 2012, 4:09:56 PM2/23/12

to Discussion of Rocks Clusters

may be it should

the /etc/resolv.conf does contain the private hostname of FE
-LT

-------------- next part --------------

A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/d3f5208c/laotsao.vcf

Philip Papadopoulos

unread,

Feb 23, 2012, 5:09:09 PM2/23/12

to Discussion of Rocks Clusters

On Thu, Feb 23, 2012 at 12:24 PM, Ryan Golhar
<ngsbioin...@gmail.com>wrote:

> I noticed now on my compute nodes that /etc/host contains the headnode's

> public IP address instead of the headnode's private IP address.
>

Why do you believe this is a problem?
The headnode actually has two names (it IS dual homed). The FQDN on the
public and name.local on the
private.
Routing on the compute nodes always forces network connectivity to the FQDN
of the frontend through the
private interface.

>
> Shouldn't the /etc/host on the compute nodes contain the private IP address
> of the headnode instead of the public IP address?
>

No.

>

--
Philip Papadopoulos, PhD
University of California, San Diego
858-822-3628 (Ofc)
619-331-2990 (Fax)

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/d60a7751/attachment.html

Ryan Golhar

unread,

Feb 23, 2012, 5:21:05 PM2/23/12

to Discussion of Rocks Clusters

Because my cluster isn't actually in DNS...so resolving with the FQDN will
always fail unless its properly specified in /etc/hosts.

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/02ad6739/attachment.html

Hung-Sheng Tsao (laoTsao)

unread,

Feb 23, 2012, 8:16:13 PM2/23/12

to Discussion of Rocks Clusters, Discussion of Rocks Clusters

Sent from my iPad

On Feb 23, 2012, at 17:21, Ryan Golhar <ngsbioin...@gmail.com> wrote:

> Because my cluster isn't actually in DNS...so resolving with the FQDN will
> always fail unless its properly specified in /etc/hosts.

not sure what this means
youdo not have external DNS server?

Ryan Golhar

unread,

Feb 23, 2012, 9:29:58 PM2/23/12

to Discussion of Rocks Clusters

Sorry for not being clear. I see the compute nodes have the public IP
address of the headnode in their /etc/hosts. I would have expected the
headnodes private network IP address to be listed instead.

I don't think my conversion from 10.x.x.x.x to 192.168.x.x went completely
correct. From my compute nodes, I can't ping my headnode by its FQDN, but
I can by only the hostname. My /etc/hosts on my compute nodes looks like

[root@compute-0-2 ~]# more /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.228.81.46 bic.xxx.xxx.xxx
192.168.1.252 compute-0-2.local compute-0-2

If I ping by hostname, it works:

[root@compute-0-2 ~]# ping bic
PING bic.local (192.168.1.1) 56(84) bytes of data.
64 bytes from bic.local (192.168.1.1): icmp_seq=1 ttl=64 time=0.099 ms
64 bytes from bic.local (192.168.1.1): icmp_seq=2 ttl=64 time=0.097 ms

BUT, if I ping by FQDN, it doesn't work:

[root@compute-0-2 ~]# ping bic.xxx.xxxx.xxx
connect: Network is unreachable

2012/2/23 Hung-Sheng Tsao (laoTsao) <lao...@gmail.com>

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/a307604d/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Feb 24, 2012, 9:16:28 AM2/24/12

to Discussion of Rocks Clusters

please check the output
rocks list route
to see all gateway are correct
-LT

Founder & Principal

HopBit GridComputing LLC
cell: 9734950840

http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/

-------------- next part --------------

A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120224/4740376c/laotsao.vcf

Ryan Golhar

unread,

Feb 25, 2012, 12:35:42 PM2/25/12

to Discussion of Rocks Clusters

'rocks list route' shows

[root@bic ~]# rocks list route
NETWORK NETMASK GATEWAY
224.0.0.0: 255.255.255.0 private
255.255.255.255: 255.255.255.255 private
0.0.0.0: 0.0.0.0 10.1.1.1
10.228.81.46: 255.255.255.255 10.1.1.1

10.1.1.1 is incorrect. This should be 192.168.1.1 This doesn't appear in
rocks list attr or rocks list host attr.

Also, on my compute node, there is no GATEWAY entry in ifcfg-eth0, and
route -n shows:

[root@compute-0-2 ~]# route -n

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface

255.255.255.255 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

224.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

no gateway entry here either. Where is this set? I just installed a rocks
headnodes and rocks compute node in VirtualBox on my Mac and they show the
gateway correctly on the compute node. Hmmm. I'll try to track these
down.

2012/2/24 "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <lao...@gmail.com>

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120225/f2e7f94e/attachment.html

Ryan Golhar

unread,

Feb 25, 2012, 12:58:07 PM2/25/12

to Discussion of Rocks Clusters

I seem to be screwing up my settings using rocks add host route. Is there
a way I can edit the entries in the mysql database directly and have rocks
pick it up?

On Sat, Feb 25, 2012 at 12:35 PM, Ryan Golhar
<ngsbioin...@gmail.com>wrote:

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120225/06b075ec/attachment.html

Ryan Golhar

unread,

Feb 25, 2012, 1:40:41 PM2/25/12

to Discussion of Rocks Clusters

I think I got everything fixed. I had to go into the mysql db and fix some
entries. One last question (hopefully), 'rocks list host route' shows
SOURCE in the last column of output. Where does this come from? I could
find any reference in the database to it and I'm not familiar enough with
Python to track it down in the code.

On Sat, Feb 25, 2012 at 12:58 PM, Ryan Golhar
<ngsbioin...@gmail.com>wrote:

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120225/298487a8/attachment.html

Hung-Sheng Tsao (laoTsao)

unread,

Feb 25, 2012, 5:37:08 PM2/25/12

to Discussion of Rocks Clusters, Discussion of Rocks Clusters

try rocks list host route

Sent from my iPad

Ryan Golhar

unread,

Feb 26, 2012, 5:08:32 PM2/26/12

to Discussion of Rocks Clusters

Ok. I've *finally* resolved my 10.x.x.x.x -> 192.168.x.x conversion.
However, I just discovered this doesn't effect my DNS resolution issue. I
can ping (from the head node) www.google.com. At least ping can resolve
the IP address. When I try the same with search.cpan.org, it fails to
resolve. All the network settings look correct as far as I can tell. So,
I'm stuck. I suspect may be the DNS server on the headnode, but not sure.
Here's what my configuration now looks like:

[ryang@bic ~]$ ping www.google.com
PING www.l.google.com (74.125.115.99) 56(84) bytes of data.

[ryang@bic bam]$ ping search.cpan.org
ping: unknown host search.cpan.org

[ryang@bic bam]$ more /etc/hosts
# Added by rocks report host #
# DO NOT MODIFY #
# Add any modifications to #
# /etc/hosts.local file #

127.0.0.1 localhost.localdomain localhost

192.168.1.1 bic.local bic
192.168.1.254 compute-0-0.local compute-0-0
192.168.1.253 compute-0-1.local compute-0-1
192.168.1.252 compute-0-2.local compute-0-2
10.228.81.46 bic.xxx.xxx.xxx

# Imported from /etc/hosts.local

10.228.81.47 tux

[ryang@bic bam]$ more /etc/resolv.conf
search xxx.xxx.xxx

nameserver 127.0.0.1
nameserver 10.24.2.11
nameserver 10.24.2.12

[ryang@bic bam]$ more /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:22:19:21:5E:6B
IPADDR=192.168.1.1
NETMASK=255.255.255.0
BOOTPROTO=static
ONBOOT=yes
MTU=1500

[ryang@bic bam]$ more /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:22:19:21:5E:69
IPADDR=10.228.81.46
NETMASK=255.255.255.0
BOOTPROTO=static
ONBOOT=yes
MTU=1500
GATEWAY=10.228.81.254

[ryang@bic ~]$ /sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface

255.255.255.255 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
224.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

10.228.81.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 10.228.81.254 0.0.0.0 UG 0 0 0 eth1

2012/2/25 Hung-Sheng Tsao (laoTsao) <lao...@gmail.com>

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120226/752add83/attachment.html

Hung-Sheng Tsao (laoTsao)

unread,

Feb 26, 2012, 7:30:25 PM2/26/12

to Discussion of Rocks Clusters, Discussion of Rocks Clusters

may be you should check your dns server of
xxx.xxx
xxx.xxx.xxx
also check other client in xxx.xxx.xxx not rockscluster frontend
-LT

Sent from my iPad

Ryan Golhar

unread,

Feb 28, 2012, 12:44:06 PM2/28/12

to Discussion of Rocks Clusters

Already did. The DNS servers match. The network mask of the public IPs
match. TheI can't find any obvious differences. That's why I suspect
something to do with the DNS server on the rocks cluster itself.

2012/2/26 Hung-Sheng Tsao (laoTsao) <lao...@gmail.com>

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120228/bcdaa9d3/attachment.html

Hung-Sheng Tsao (laoTsao)

unread,

Feb 28, 2012, 6:05:43 PM2/28/12

to Discussion of Rocks Clusters, Discussion of Rocks Clusters

FE just dns client so if your DNS server and all correct then it should just work
try to run traceroute <host>
-LT

Sent from my iPad

Hung-Sheng Tsao (laoTsao)

unread,

Feb 28, 2012, 6:23:40 PM2/28/12

to Hung-Sheng Tsao (laoTsao), Discussion of Rocks Clusters

sorry
traceroute <IP>
also do nslook <host>

Sent from my iPad

Gowtham

unread,

Feb 28, 2012, 6:28:16 PM2/28/12

to Discussion of Rocks Clusters

On Tue, 28 Feb 2012, Hung-Sheng Tsao (laoTsao) wrote:

| sorry
| traceroute <IP>
| also do nslook <host>

should be

nslookup <host>

Ryan Golhar

unread,

Mar 5, 2012, 3:01:27 PM3/5/12

to Discussion of Rocks Clusters

So it looks like my cluster is getting to my DNS server but recursion is
not available. Is this a problem on my DNS server or my ROCKS cluster???

[ryang@bic ~]$ ping www.broadinstitute.org
ping: unknown host www.broadinstitute.org

[ryang@bic ~]$ traceroute www.broadinstitute.org
www.broadinstitute.org: Temporary failure in name resolution
Cannot handle "host" cmdline arg `www.broadinstitute.org' on position 1
(argc 1)

[ryang@bic ~]$ nslookup www.broadinstitute.org
;; Got SERVFAIL reply from 127.0.0.1, trying next server
;; Got recursion not available from 10.24.2.11, trying next server
Server: 10.24.2.12
Address: 10.24.2.12#53

Non-authoritative answer:
*** Can't find www.broadinstitute.org: No answer

[ryang@bic ~]$ more /etc/resolv.conf

search xxx.xxx.xxx
nameserver 127.0.0.1
nameserver 10.24.2.11
nameserver 10.24.2.12

2012/2/28 Gowtham <g...@mtu.edu>

> | >>>>>>>>>>>>>>>>>>>> traceroute to mirrorlist.centos.org(64.235.47.134), 30

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/f81dcc35/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Mar 5, 2012, 3:18:47 PM3/5/12

to Discussion of Rocks Clusters

IMHO, it is your DNS server
you should try it on your DNS server

nameserver 10.24.2.11
nameserver 10.24.2.12

in-line
-LT
in r543 rockscluster
> [root@r543 ~]# ping www.broadinstitute.org
> PING www.broadinstitute.org (69.173.64.101) 56(84) bytes of data.
> *<---- see the IP but cannot ping*
>
> --- www.broadinstitute.org ping statistics ---
> 13 packets transmitted, 0 received, 100% packet loss, time 12008ms
>
> [root@r543 ~]# traceroute www.broadinstitute.org
> traceroute to www.broadinstitute.org (69.173.64.101), 30 hops max, 40
> byte packets
> 1 * * *
> 2 192.168.2.1 (192.168.2.1) 12.353 ms 12.326 ms 12.281 ms
> 3 dstswr1-vlan2.rh.rndhnj.cv.net (67.83.249.161) 12.239 ms 12.201
> ms 12.442 ms
> 4 rtr1-ge6-1.mhe.prnynj.cv.net (67.83.249.129) 18.054 ms 18.148
> ms 18.147 ms
> 5 (64.15.2.69) 18.145 ms rtr4-tg11-2.wan.prnynj.cv.net
> (64.15.6.25) 19.089 ms rtr3-tg11-2.wan.prnynj.cv.net (64.15.6.1)
> 19.900 ms
> 6 (64.15.2.146) 18.984 ms rtr1-tg11-1.in.nwrknjmd.cv.net
> (64.15.0.82) 10.880 ms (64.15.2.146) 14.149 ms
> 7 xe-9-2-3.edge2.Newark1.Level3.net (4.30.130.241) 22.629 ms
> 23.512 ms 23.541 ms
> 8 ae-31-51.ebr1.Newark1.Level3.net (4.69.156.30) 13.921 ms 19.739
> ms 19.741 ms
> 9 ae-2-2.ebr1.NewYork1.Level3.net (4.69.132.97) 19.773 ms 20.409
> ms 19.230 ms
> 10 ae-1-8.bar2.Boston1.Level3.net (4.69.140.97) 39.747 ms 39.227
> ms 36.181 ms
> 11 ae-5-5.car2.Boston1.Level3.net (4.69.132.249) 21.194 ms * *
> 12 BROAD-INSTI.car2.Boston1.Level3.net (4.53.50.78) 17.006 ms
> 16.968 ms 16.741 ms
> 13 * * *
> 14 * * *
> 15 * * *
> 16 * * *
> 17 * * *
> 18 * * *
> 19 * * *
> 20 * * *
> 21 * * *
> 22 * 216.55.4.6 (216.55.4.6) 15.496 ms !X *
> [root@r543 ~]#

>> |>>>>>>> http://blogs.oracle.com/hstsao/
>> |>>>>>>>
>> |>>>>>>> -------------- next part --------------
>> |>>>>>>> A non-text attachment was scrubbed...
>> |>>>>>>> Name: laotsao.vcf
>> |>>>>>>> Type: text/x-vcard
>> |>>>>>>> Size: 608 bytes
>> |>>>>>>> Desc: not available
>> |>>>>>>> Url :
>> |>>>>>>>
>> |>>>>>
>> |>>>
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120224/4740376c/laotsao.vcf
>> |>>>>>>>
>> |>>>>>> -------------- next part --------------
>> |>>>>>> An HTML attachment was scrubbed...
>> |>>>>>> URL:
>> |>>>>>
>> |>>>
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120225/f2e7f94e/attachment.html
>> |>>>>>
>> |>>>> -------------- next part --------------
>> |>>>> An HTML attachment was scrubbed...
>> |>>>> URL:
>> |>>>
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120226/752add83/attachment.html
>> |>>>
>> |>> -------------- next part --------------
>> |>> An HTML attachment was scrubbed...
>> |>> URL:
>> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120228/bcdaa9d3/attachment.html
>> |
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/f81dcc35/attachment.html

--

Hung-Sheng Tsao Ph D.
Founder& Principal
HopBit GridComputing LLC
cell: 9734950840

http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/

-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/74e6d17e/attachment.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/74e6d17e/laotsao.vcf

Ryan Golhar

unread,

Mar 5, 2012, 3:38:50 PM3/5/12

to Discussion of Rocks Clusters

Normally, I would agree with you. I have a second cluster that uses the
same DNS servers. If I execute the exact same commands on that cluster, my
DNS resolution works. That's why I still think its something with the way
this particular ROCKs cluster is set up.

[ryang@cc ~]$ more /etc/resolv.conf
; generated by /sbin/dhclient-script
search xxx.xxx.xxx
nameserver 10.24.2.11
nameserver 10.24.2.12
[ryang@cc ~]$ nslookup www.broadinstitute.org
Server: 10.24.2.11
Address: 10.24.2.11#53

Non-authoritative answer:
Name: www.broadinstitute.org
Address: 69.173.64.101

[ryang@cc ~]$ dig www.broadinstitute.org

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> www.broadinstitute.org

;; global options: printcmd
;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11028
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;www.broadinstitute.org. IN A

;; ANSWER SECTION:
www.broadinstitute.org. 43192 IN A 69.173.64.101

;; AUTHORITY SECTION:
broadinstitute.org. 43192 IN NS ns2.broadinstitute.org.
broadinstitute.org. 43192 IN NS ns1.broadinstitute.org.

;; Query time: 0 msec
;; SERVER: 10.24.2.11#53(10.24.2.11)
;; WHEN: Mon Mar 5 15:35:36 2012
;; MSG SIZE rcvd: 92

> 6 (64.15.2.146) 18.984 ms rtr1-tg11-1.in.nwrknjmd.cv.net(64.15.0.82) 10.880 ms (64.15.2.146) 14.149 ms

> [ryang@bic ~]$ traceroute www.broadinstitute.orgwww.broadinstitute.org: Temporary failure in name resolution

> Cannot handle "host" cmdline arg `www.broadinstitute.org' on position 1
> (argc 1)
>
> [ryang@bic ~]$ nslookup www.broadinstitute.org
> ;; Got SERVFAIL reply from 127.0.0.1, trying next server
> ;; Got recursion not available from 10.24.2.11, trying next server
> Server: 10.24.2.12
> Address: 10.24.2.12#53
>
> Non-authoritative answer:
> *** Can't find www.broadinstitute.org: No answer
>
> [ryang@bic ~]$ more /etc/resolv.conf
> search xxx.xxx.xxx
> nameserver 127.0.0.1
> nameserver 10.24.2.11
> nameserver 10.24.2.12
>
>

> 2012/2/28 Gowtham <g...@mtu.edu> <g...@mtu.edu>

> | > On Feb 28, 2012, at 12:44, Ryan Golhar <ngsbioin...@gmail.com> <ngsbioin...@gmail.com>

> | >> 2012/2/26 Hung-Sheng Tsao (laoTsao) <lao...@gmail.com> <lao...@gmail.com>

> | >>
> | >>> may be you should check your dns server of
> | >>> xxx.xxx
> | >>> xxx.xxx.xxx
> | >>> also check other client in xxx.xxx.xxx not rockscluster frontend
> | >>> -LT
> | >>>
> | >>> Sent from my iPad
> | >>>

> | >>> On Feb 26, 2012, at 17:08, Ryan Golhar <ngsbioin...@gmail.com> <ngsbioin...@gmail.com>

> | >>>> 2012/2/25 Hung-Sheng Tsao (laoTsao) <lao...@gmail.com> <lao...@gmail.com>

> | >>>>>>>>>> <ngsbioin...@gmail.com> <ngsbioin...@gmail.com>wrote:

> | >>>>>>>>>>>>>>>>> <ngsbioin...@gmail.com> <ngsbioin...@gmail.com>****wrote:

> | >>>>>>>>>>>>>>>>>>> com/hstsao/ <http://blogs.oracle.com/hstsao/> <http://blogs.oracle.com/hstsao/>>>

> | >>>>>>> Founder & Principal

> Founder & Principal

> HopBit GridComputing LLC
> cell: 9734950840

> http://laotsao.blogspot.com/http://laotsao.wordpress.com/http://blogs.oracle.com/hstsao/

>
>
-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/988e4ff4/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Mar 5, 2012, 4:00:51 PM3/5/12

to Discussion of Rocks Clusters

so
did you do a total new re-installation of your cluster with new IP etc?
-LT

>> Founder& Principal
>> HopBit GridComputing LLC
>> cell: 9734950840

>> http://laotsao.blogspot.com/http://laotsao.wordpress.com/http://blogs.oracle.com/hstsao/
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/988e4ff4/attachment.html

--

Hung-Sheng Tsao Ph D.
Founder& Principal
HopBit GridComputing LLC
cell: 9734950840

http://laotsao.blogspot.com/
http://laotsao.wordpress.com/
http://blogs.oracle.com/hstsao/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: laotsao.vcf
Type: text/x-vcard
Size: 608 bytes
Desc: not available

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/c61f8e50/laotsao.vcf

Ryan Golhar

unread,

Mar 5, 2012, 4:03:45 PM3/5/12

to Discussion of Rocks Clusters

No, I reconfigured the existing installation. I did a new install on a
virtual cluster and compared the settings to make sure everything was
correct.

On Mon, Mar 5, 2012 at 4:00 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <
lao...@gmail.com> wrote:

> so
> did you do a total new re-installation of your cluster with new IP etc?
> -LT
>
>
> On 3/5/2012 3:38 PM, Ryan Golhar wrote:
>
>> Normally, I would agree with you. I have a second cluster that uses the
>> same DNS servers. If I execute the exact same commands on that cluster,
>> my
>> DNS resolution works. That's why I still think its something with the way
>> this particular ROCKs cluster is set up.
>>
>> [ryang@cc ~]$ more /etc/resolv.conf
>> ; generated by /sbin/dhclient-script
>> search xxx.xxx.xxx
>> nameserver 10.24.2.11
>> nameserver 10.24.2.12
>> [ryang@cc ~]$ nslookup www.broadinstitute.org
>> Server: 10.24.2.11
>> Address: 10.24.2.11#53
>>
>> Non-authoritative answer:
>> Name: www.broadinstitute.org
>> Address: 69.173.64.101
>>
>> [ryang@cc ~]$ dig www.broadinstitute.org
>>

>> ;<<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.**el5_4.2<<>>

>>> 6 (64.15.2.146) 18.984 ms rtr1-tg11-1.in.nwrknjmd.cv.net**(64.15.0.82)

>>> 10.880 ms (64.15.2.146) 14.149 ms
>>>

>>> 7 xe-9-2-3.edge2.Newark1.Level3.**net<http://xe-9-2-3.edge2.Newark1.Level3.net>(4.30.130.241) 22.629 ms 23.512
>>> ms 23.541 ms
>>> 8 ae-31-51.ebr1.Newark1.Level3.**net<http://ae-31-51.ebr1.Newark1.Level3.net>(4.69.156.30) 13.921 ms 19.739 ms
>>> 19.741 ms
>>> 9 ae-2-2.ebr1.NewYork1.Level3.**net<http://ae-2-2.ebr1.NewYork1.Level3.net>(4.69.132.97) 19.773 ms 20.409 ms

>>> 19.230 ms
>>> 10 ae-1-8.bar2.Boston1.Level3.net (4.69.140.97) 39.747 ms 39.227 ms
>>> 36.181 ms
>>> 11 ae-5-5.car2.Boston1.Level3.net (4.69.132.249) 21.194 ms * *

>>> 12 BROAD-INSTI.car2.Boston1.**Level3.net<http://BROAD-INSTI.car2.Boston1.Level3.net>(4.53.50.78) 17.006 ms 16.968

>>> ms 16.741 ms
>>> 13 * * *
>>> 14 * * *
>>> 15 * * *
>>> 16 * * *
>>> 17 * * *
>>> 18 * * *
>>> 19 * * *
>>> 20 * * *
>>> 21 * * *
>>> 22 * 216.55.4.6 (216.55.4.6) 15.496 ms !X *
>>> [root@r543 ~]#
>>>
>>>
>>>
>>> On 3/5/2012 3:01 PM, Ryan Golhar wrote:
>>>
>>> So it looks like my cluster is getting to my DNS server but recursion is
>>> not available. Is this a problem on my DNS server or my ROCKS cluster???
>>>
>>> [ryang@bic ~]$ ping www.broadinstitute.org
>>> ping: unknown host www.broadinstitute.org
>>>

>>> [ryang@bic ~]$ traceroute www.broadinstitute.orgwww.**broadinstitute.org<http://www.broadinstitute.orgwww.broadinstitute.org>:

>>> |> On Feb 28, 2012, at 12:44, Ryan Golhar<ngsbioinformatics@**gmail.com<ngsbioin...@gmail.com>>

>>> |>>> On Feb 26, 2012, at 17:08, Ryan Golhar<ngsbioinformatics@**
>>> gmail.com <ngsbioin...@gmail.com>> <ngsbioin...@gmail.com>

>>> |>>>> [ryang@bic bam]$ more /etc/sysconfig/network-**scripts/ifcfg-eth0

>>> |>>>> [ryang@bic bam]$ more /etc/sysconfig/network-**scripts/ifcfg-eth1

>>> |>>>>> On Feb 25, 2012, at 12:35, Ryan Golhar<ngsbioinformatics@**
>>> gmail.com <ngsbioin...@gmail.com>>

>>> |>>>>>>>> On Feb 23, 2012, at 17:21, Ryan Golhar<ngsbioinformatics@**
>>> gmail.com <ngsbioin...@gmail.com>>

>>> >**wrote:

>>> ngsbioin...@gmail.com>******wrote:

>>> 9.3.6-P1-RedHat-9.3.6-16.P1.********el5<<>>

>>> |>>>>>>>>>>>>>>>>>>>> traceroute to mirrorlist.centos.org(64.235.**47.134),

>>> 30
>>> |>>>>> hops
>>> |>>>>>>>>>>> max,
>>> |>>>>>>>>>>>>>>>>>>>> 40
>>> |>>>>>>>>>>>>>>>>>>>> byte
>>> |>>>>>>>>>>>>>>>>>>>> packets
>>> |>>>>>>>>>>>>>>>>>>>> 1 10.228.81.254 (10.228.81.254) 0.367 ms 0.393
>>> ms
>>> |>>> 0.451
>>> |>>>>>>> ms
>>> |>>>>>>>>>>>>>>>>>>>> 2 10.0.7.66 (10.0.7.66) 0.558 ms 0.756 ms
>>> 0.933
>>> ms
>>> |>>>>>>>>>>>>>>>>>>>> 3 10.0.31.24 (10.0.31.24) 2.965 ms 2.770 ms
>>> 3.072 ms
>>> |>>>>>>>>>>>>>>>>>>>> ...
>>> |>>>>>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>>>>>> so in this case it looks like routing is happening
>>> |>>>>> correctly.
>>> |>>>>>>>>>>>>>>>>>>>> -------------- next part --------------
>>> |>>>>>>>>>>>>>>>>>>>> An HTML attachment was scrubbed...
>>> |>>>>>>>>>>>>>>>>>>>> URL:

>>> >
>>> |>>>>>>>>>>
>>> discussion/attachments/******20120221/03265073/attachment.******html<
>>> |>>>>>>>>>>>>>>>>>>>> https://lists.sdsc.edu/****
>>> pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>> |>>>>>>>>>>>>>>>>>>>>
>>> |>>>>> discussion/attachments/****20120221/03265073/attachment.***
>>> *html<
>>> |>>>>>>>>>>
>>> |>>>>>>>
>>> |>>>>>
>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120221/03265073/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120221/03265073/attachment.html>
>>> |>>>>>>>>>>>>>>>>>>>> --
>>> |>>>>>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>>>>> Hung-Sheng Tsao Ph D.
>>> |>>>>>>>>>>>>>>>>>>> Founder& Principal
>>> |>>>>>>>>>>>>>>>>>>> HopBit GridComputing LLC
>>> |>>>>>>>>>>>>>>>>>>> cell: 9734950840
>>> |>>>>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>>>>> http://laotsao.blogspot.com/
>>> |>>>>>>>>>>>>>>>>>>> http://laotsao.wordpress.com/

>>> |>>>>>>>>>>>>>>>>> URL:https://lists.sdsc.edu/*****
>>> *pipermail/npaci-rocks-**<https://lists.sdsc.edu/****pipermail/npaci-rocks-**>
>>> <
>>> |>>>>>>>>>>> https://lists.sdsc.edu/****pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>> >
>>> |>>>>>>>>>>>>>>>>>
>>> |>>>>>>> discussion/attachments/******20120222/6126b940/attachment.***

>>> ***html<
>>> |>>>>>>>>>>>>>>>>> https://lists.sdsc.edu/****pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>> |>>>>>>>>>>>>>>>>>

>>> |>>> discussion/attachments/****20120222/6126b940/attachment.****html<
>>> |>>>>>>>>>>
>>> |>>>>>>>
>>> |>>>>>
>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120222/6126b940/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/6126b940/attachment.html>
>>> |>>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>>> --
>>> |>>>>>>>>>>>>>>>> Hung-Sheng Tsao Ph D.
>>> |>>>>>>>>>>>>>>>> Founder& Principal
>>> |>>>>>>>>>>>>>>>> HopBit GridComputing LLC
>>> |>>>>>>>>>>>>>>>> cell: 9734950840
>>> |>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>> http://laotsao.blogspot.com/
>>> |>>>>>>>>>>>>>>>> http://laotsao.wordpress.com/

>>> |>>>>>>>>>>>>>>>> http://blogs.oracle.com/******hstsao/<http://blogs.oracle.com/****hstsao/>
>>> <
>>> |>>>>>>>>>>> http://blogs.oracle.com/****hstsao/<http://blogs.oracle.com/**hstsao/>
>>> >
>>> |>>>>>>>>>>>>>>>> <http://blogs.oracle.**com/**hstsao/<

>>> |>>>>>>>>>> http://blogs.oracle.com/**hstsao/<http://blogs.oracle.com/hstsao/>
>>> |>>>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>>>> -------------- next part --------------
>>> |>>>>>>>>>>>>>>> An HTML attachment was scrubbed...

>>> |>>>>>>>>>>>>>>> URL: https://lists.sdsc.edu/****
>>> pipermail/npaci-rocks-**<https://lists.sdsc.edu/**pipermail/npaci-rocks-**>
>>> |>>>>>>>>>>>>>>>
>>> discussion/attachments/****20120222/5402107c/attachment.****html<

>>> |>>>>>>>>>>
>>> |>>>>>>>
>>> |>>>>>
>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120222/5402107c/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120222/5402107c/attachment.html>
>>> |>>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>> --
>>> |>>>>>>>>>>>>> Hung-Sheng Tsao Ph D.
>>> |>>>>>>>>>>>>> Founder& Principal
>>> |>>>>>>>>>>>>> HopBit GridComputing LLC
>>> |>>>>>>>>>>>>> cell: 9734950840
>>> |>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>> http://laotsao.blogspot.com/
>>> |>>>>>>>>>>>>> http://laotsao.wordpress.com/

>>> |>>>>>>>>>>>>> http://blogs.oracle.com/****hstsao/<http://blogs.oracle.com/**hstsao/>

>>> <
>>> |>>>>> http://blogs.oracle.com/**hstsao/<http://blogs.oracle.com/hstsao/>
>>> >
>>> |>>>>>>>>>>>>>
>>> |>>>>>>>>>>>>>
>>> |>>>>>>>>>>> -------------- next part --------------
>>> |>>>>>>>>>>> An HTML attachment was scrubbed...
>>> |>>>>>>>>>>> URL:
>>> |>>>>>>>>>>>
>>> |>>>>>>>>>>
>>> |>>>>>>>
>>> |>>>>>

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120223/f6adecf0/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/f6adecf0/attachment.html>

>>> |>>>>>>>>>>>
>>> |>>>>>>>>>>
>>> |>>>>>>>>>> --
>>> |>>>>>>>>>> Philip Papadopoulos, PhD
>>> |>>>>>>>>>> University of California, San Diego
>>> |>>>>>>>>>> 858-822-3628 (Ofc)
>>> |>>>>>>>>>> 619-331-2990 (Fax)
>>> |>>>>>>>>>> -------------- next part --------------
>>> |>>>>>>>>>> An HTML attachment was scrubbed...
>>> |>>>>>>>>>> URL:
>>> |>>>>>>>>>>
>>> |>>>>>>>
>>> |>>>>>

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120223/d60a7751/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/d60a7751/attachment.html>

>>> |>>>>>>>>>>
>>> |>>>>>>>>> -------------- next part --------------
>>> |>>>>>>>>> An HTML attachment was scrubbed...
>>> |>>>>>>>>> URL:
>>> |>>>>>>>
>>> |>>>>>

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120223/02ad6739/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120223/02ad6739/attachment.html>

>>> |>>>>>>> http://blogs.oracle.com/**hstsao/<http://blogs.oracle.com/hstsao/>
>>> |>>>>>>>
>>> |>>>>>>> -------------- next part --------------

>>> |>>>>>>> A non-text attachment was scrubbed...
>>> |>>>>>>> Name: laotsao.vcf
>>> |>>>>>>> Type: text/x-vcard
>>> |>>>>>>> Size: 608 bytes
>>> |>>>>>>> Desc: not available
>>> |>>>>>>> Url :
>>> |>>>>>>>
>>> |>>>>>

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120224/4740376c/laotsao.vcf<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120224/4740376c/laotsao.vcf>

>>> |>>>>>>>
>>> |>>>>>> -------------- next part --------------
>>> |>>>>>> An HTML attachment was scrubbed...
>>> |>>>>>> URL:
>>> |>>>>>

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120225/f2e7f94e/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120225/f2e7f94e/attachment.html>

>>> |>>>>>
>>> |>>>> -------------- next part --------------
>>> |>>>> An HTML attachment was scrubbed...
>>> |>>>> URL:

>>> |>>>https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120226/752add83/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120226/752add83/attachment.html>

>>> |>>>
>>> |>> -------------- next part --------------
>>> |>> An HTML attachment was scrubbed...

>>> |>> URL:https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120228/bcdaa9d3/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120228/bcdaa9d3/attachment.html>

>>> |
>>>
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...

>>> URL: https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>>> discussion/attachments/**20120305/f81dcc35/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/f81dcc35/attachment.html>

>>>
>>>
>>> --
>>> Hung-Sheng Tsao Ph D.
>>> Founder& Principal
>>> HopBit GridComputing LLC
>>> cell: 9734950840

>>> http://laotsao.blogspot.com/**http://laotsao.wordpress.com/**
>>> http://blogs.oracle.com/**hstsao/<http://laotsao.blogspot.com/http://laotsao.wordpress.com/http://blogs.oracle.com/hstsao/>

>>>
>>>
>>> -------------- next part --------------
>> An HTML attachment was scrubbed...

>> URL: https://lists.sdsc.edu/**pipermail/npaci-rocks-**
>> discussion/attachments/**20120305/988e4ff4/attachment.**html<https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/988e4ff4/attachment.html>

>>
>
> --
> Hung-Sheng Tsao Ph D.
> Founder& Principal
> HopBit GridComputing LLC
> cell: 9734950840
>
> http://laotsao.blogspot.com/
> http://laotsao.wordpress.com/

> http://blogs.oracle.com/**hstsao/ <http://blogs.oracle.com/hstsao/>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/43d36492/attachment.html

"Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D."

unread,

Mar 5, 2012, 4:24:44 PM3/5/12

to Discussion of Rocks Clusters

may be try do do a new re-install:-(

Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/5f114b6f/laotsao.vcf

Ryan Golhar

unread,

Mar 5, 2012, 4:32:08 PM3/5/12

to Discussion of Rocks Clusters

That's what I was afraid of :) Hmmm. I might have to. Thanks.

On Mon, Mar 5, 2012 at 4:24 PM, "Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D." <
lao...@gmail.com> wrote:

> >>>> |> On Feb 28, 2012, at 12:44, Ryan Golhar<ngsbioinformatics@**

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20120305/2597ee4a/attachment.html

Reply all

Reply to author

Forward