Execute script and OVS problem

[İlker Özçelik]

Hi all

I am having problem with executing script on my VMs. I am using Slice Jacks Beta. It looks like install script downloads the code but execute script fails.

Also I am running my switch on Kentucky PKS2 Instageni and I chose Ubuntu12-64-OVS image. In the node, when I run "sudo ovs-vsctl add-br br0" or any "ovs-vsctl" command I get the error below:

2014-11-26T19:21:34Z|00002|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock: connection attempt failed (Connection refused)
ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database connection failed (Connection refused)

I would appreciate any help or suggestion you can give.

PS: My rspect file is attached.

Thanks.

[Sarah Edwards]

Hi Ilker,

I'm trying to reproduce your problem now.

While I'm doing that can you confirm some things:

 1) The ovs-vsctl commands fail both when you run them on the command line as well as in the install script?

 2) Can you run any commands using other ovs commands this: sudo ovs-ofctl dump-flows br0 ?

Thanks,

Sarah

--
GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.
 
If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
---
You received this message because you are subscribed to the Google Groups "GENI Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<rspecPKS2allLMT.xml>

*******************************************************************************

Sarah Edwards

GENI Project Office

BBN Technologies

Cambridge, MA

phone:    (617) 873-2329

email:    sedw...@bbn.com

[Ilker Özçelik]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Sarah
1) I tried both and failed.

2) I tried show option and it did not work either.

Thanks.

>You received this message because you are subscribed to a topic in the

>Google Groups "GENI Users" group.

>To unsubscribe from this topic, visit
>https://groups.google.com/d/topic/geni-users/cvKQQtvhE4A/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to

>geni-users+...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

- --
Ilker Ozcelik

PhD Candidate
Holcombe Department of Electrical and Computer Engineering
Clemson University

13 Riggs Hall
Clemson, SC 29631
USA
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFRBAEBCgA7BQJUdkS7NBxJbGtlciBPemNlbGlrIChubyBwYXNzd29yZCkgPGlv
emNlbGlAZy5jbGVtc29uLmVkdT4ACgkQ5/egE1JZaytS2wgAgh900D2gE7l3JIrk
KnWPJJoYULRxn9WR7+/W0rcxLHx/GtHmS/nOf5Ga+GodapzAk+/pbpamssQxRpUe
uSXSfS2cHn4r57H/WubDHDp3NrhbDo4gDqL5AabEOR0XEmHBKns/2xsXI8jTKw/D
C+tmUz12dgVOkPyLTzM4KkLXk8rVChAK630CnOvJ1tAacb4/SYWDDLtFhexhcB4Y
y2ggJCpFkDyhPSf4C+ncH8UqMt6wVxFg9s2q8LmRyqMpQu0Ixr2uUeHlClG1zeiL
+HPCHyybnCsF5zkJwtFANHfOdTiPAFEXkHfEjvBf8HDJCyZrJa7tkQZRoVn4J1vS
3OjNNQ==
=DI0E
-----END PGP SIGNATURE-----

[Sarah Edwards]

Hi Ilker,

Thank you.

I'm able to reproduce your problem using your RSpec modified to all be in a single aggregate.

I also tried a variation without running the install/execute script on the OVS node and that also fails.

Now I'm starting to worry about the image. Still looking...

Sarah

[Ilker Özçelik]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Sarah,

Also execution problem is not limited to OVS image. My scripts did not work on regular ubuntu 12 images in different AMs also.

Thanks.

iQFRBAEBCgA7BQJUdkthNBxJbGtlciBPemNlbGlrIChubyBwYXNzd29yZCkgPGlv
emNlbGlAZy5jbGVtc29uLmVkdT4ACgkQ5/egE1JZaytjoQf+JXst36KZGjFDngDN
ReBD42lWEEsD6YRJUyfSdxLZZUiHOZRtG2ope6sZFNKAUCnTtPAdbOC45mFVLmo2
33b2qDyvVmr9eMLTWF/WcS7J2ZGcJT78co6gAkXbQyos26kyBZaRRILUb2WUuBlD
jXm1gvfMJFmJE4sPeQ47xc3C6YF9fOv+MJNDZUAWHGJvduQy5pmCWdwyWs3KJCZr
VL+95/y0iRcGWMVtgBlfa+BtvdizLWf+dfXVNZtE6Se+kLfUGz8Uv16MRF2S+p5d
YjQ//pJ9nSi0vS83uqNmmT30sb/FhqnhUVOR62Z+ZdXD76Uuxl66lT+nfcnVP+TA
Nan6eA==
=enDH
-----END PGP SIGNATURE-----

[Niky Riga]

Hi Ilker,

We have seen the OVS problem and the only way that I managed to find to work around it was delete and recreate, but if Sarah can recreate the problem then
that would not help. Sarah which AM are you using?

For the execute scripts some suggestions:
  1. you are calling python but your path might not be set, so I would suggest invoking python with the full path, "which python" should give you that info
  2. there is a useful wiki page that tells you where to look for logs:
http://www.protogeni.net/wiki/ServicesFaq

Cheers,
Niky

[Sarah Edwards]

Hi all,

(To answer Niky's question, I've been using ukypks2-ig and Illinois-ig mostly.)

Vic and I are starting to suspect a broader issue involving install scripts not working at InstaGENI at all as Vic has been having a problem writing an install script today as well.

We're currently looking at the fact that few if any of the slices we've created today have a geniuser home directory.

In particular, I reserved the hellogeni example (using a xen VM) and when it tries to run the install script it logs a message in /var/emulab/logs/runlog.*:
"Could not determine UID for geniuser"

Note there is no home directory for geniuser on my host "server". (Although oddly Vic's OpenVZ nodes do have home directory for geniuser, but he still sees the above error.)

This is slice: urn:publicid:IDN+ch.geni.net:GPO-Project+slice+ostest
Aggregate: clemson-ig
RSpec: http://www.gpolab.bbn.com/experiment-support/HelloGENI/hellogeni.rspec

(To be clear we are seeing similar issues that appear withe same symptom of "install script doesn't appear to have been executed" on a variety of images and xen and OpenVZ, but the above is just our cleanest example.)

If someone happens to be around, can you take a look? (I know it's unlikely due to the time and holiday.)

Ilker, can you look in /var/emulab/logs/runlog.* and report back what you see?

Thank you,
Sarah

>>>> For more options, visit https://groups.google.com/d/optout
>>>> .
>>>>

>>> *******************************************************************************
>>> Sarah Edwards
>>> GENI Project Office
>>>
>>> BBN Technologies
>>> Cambridge, MA
>>> phone: (617) 873-2329
>>> email:
>>> sedw...@bbn.com
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> GENI Users is a community supported mailing list, so please help by
>>> responding to questions you know the answer to.
>>>
>>> If this is your first time posting a question to this list, please
>>> review
>>>
>>> http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "GENI Users" group.
>>> To unsubscribe from this topic, visit
>>>
>>> https://groups.google.com/d/topic/geni-users/cvKQQtvhE4A/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>>
>>> geni-users+...@googlegroups.com
>>> .

>>> For more options, visit
>>> https://groups.google.com/d/optout
>>> .
>>>
>>

>> - --
>> Ilker Ozcelik
>>
>> PhD Candidate
>> Holcombe Department of Electrical and Computer Engineering
>> Clemson University
>>
>> 13 Riggs Hall
>> Clemson, SC 29631
>> USA
>> -----BEGIN PGP SIGNATURE-----
>> Version: APG v1.1.1
>>
>> iQFRBAEBCgA7BQJUdkthNBxJbGtlciBPemNlbGlrIChubyBwYXNzd29yZCkgPGlv
>> emNlbGlAZy5jbGVtc29uLmVkdT4ACgkQ5/egE1JZaytjoQf+JXst36KZGjFDngDN
>> ReBD42lWEEsD6YRJUyfSdxLZZUiHOZRtG2ope6sZFNKAUCnTtPAdbOC45mFVLmo2
>> 33b2qDyvVmr9eMLTWF/WcS7J2ZGcJT78co6gAkXbQyos26kyBZaRRILUb2WUuBlD
>> jXm1gvfMJFmJE4sPeQ47xc3C6YF9fOv+MJNDZUAWHGJvduQy5pmCWdwyWs3KJCZr
>> VL+95/y0iRcGWMVtgBlfa+BtvdizLWf+dfXVNZtE6Se+kLfUGz8Uv16MRF2S+p5d
>> YjQ//pJ9nSi0vS83uqNmmT30sb/FhqnhUVOR62Z+ZdXD76Uuxl66lT+nfcnVP+TA
>> Nan6eA==
>> =enDH
>> -----END PGP SIGNATURE-----
>>
>>
>

[İlker Özçelik]

Sarah,

Sorry about the late reply.
I checked the log file and this is my output;

iozceli@sw:~$ cat /var/emulab/logs/runlog.zulN1o

Could not determine UID for geniuser

Thanks.

[Leigh Stoller]

> If someone happens to be around, can you take a look? (I know it's unlikely due to the time and holiday.)

Hi Sarah. Kirk fixed a bug yesterday, and I just pushed that fix out
to all of the racks. Let me know (soon) if there are still problems.

Leigh

[Sarah Edwards]

Hi Leigh,

I just tried it again (same RSpec and slice but at Georgia Tech) and got the same error.

Hum....
Sarah

[Leigh Stoller]

> I just tried it again (same RSpec and slice but at Georgia Tech) and got the same error.

Is this this slice: urn:publicid:IDN+ch.geni.net:GPO-Project+slice+ostest

Leigh

[Sarah Edwards]

Yes.

[Leigh Stoller]

> Yes.

Okay, give it another try please … thanks!

Leigh

[Sarah Edwards]

Ok. I tried the hellogeni rspec again at GeorgiaTech and this time it seems to have worked.

Thank you so much!

Is the fix deployed everywhere?

Ilker, can you give it a try and see if your problem is fixed?

Sarah

[Leigh Stoller]

> Is the fix deployed everywhere?

No, not yet. I need to install it at the DDC ands let it run for
a few days before pushing it out.

Leigh

[İlker Özçelik]

I just tried my rspect file.  Install scripts are working.

But I still have the OVS problem. I tried two times. OVS image in Kentucky PKS2 Instageni gives this error when I run ovs commands;

iozceli@sw:~$ sudo ovs-vsctl show
2014-12-01T17:45:29Z|00001|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock: connection attempt failed (Connection refused)

ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database connection failed (Connection refused)

iozceli@sw:~$
iozceli@sw:~$
iozceli@sw:~$
iozceli@sw:~$ sudo ovs-vsctl add-br br0
2014-12-01T17:46:28Z|00002|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock: connection attempt failed (Connection refused)

ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database connection failed (Connection refused)

I will try it on a different AM.

Thanks.

[Sarah Edwards]

Hi Leigh,

Just to clarify, the new code is running at all of the InstaGENI racks except for Utah DDC? It's fine either way, I just wanted to know which ones to avoid.

Thanks,
Sarah

[Leigh Stoller]

> Just to clarify, the new code is running at all of the InstaGENI racks except for Utah DDC? It's fine either way, I just wanted to know which ones to avoid.

No, the fix is running at Gatech only.

Leigh

[Sarah Edwards]

Hi Ilker,

I tried a single AM version of your topology at Georgia Tech and it looks to me like the OVS commands return properly now.

Can you try the single AM case and tell me if everything looks good? (I've attached the RSpec i used.)

Thanks,

Sarah

[İlker Özçelik]

OVS image works at UtahDDC InstaGENI.

[İlker Özçelik]

Hi Sarah,

I tried the rspect you send. It successfully finished. I checked all nodes. I could login only switch and controller nodes and they were setup properly (install script worked). But I could not login other nodes. For some reason I get connection refused message.

pardus@diderot:~$ ssh ioz...@pc5.instageni.rnoc.gatech.edu -p 31549 -i DDM/keys/id_geni_ssh_rsa
ssh: connect to host pc5.instageni.rnoc.gatech.edu port 31549: Connection refused

[Sarah Edwards]

Well, that's definitely progress.  

Where each of the nodes in the topology green?  Maybe only some of the nodes were up?  

Sarah

[İlker Özçelik]

Hi Sarah,

I tried my topology (using 7 AMs). It worked except the OVS image. I am still getting the same error.

sudo ovs-vsctl show
2014-12-01T20:51:46Z|00001|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock: connection attempt failed (Connection refused)

ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database connection failed (Connection refused)

I think this is a corrupted image problem. OVS image worked at UtahDCC Instageni but it fails at Kentucky PSK2 Instageni. Luisa suggested Kentucky PSK2 Instageni. It is the only place which has enough bandwidth to support 5 stitched link with 20Mbps at the same time. I would really appreciate if you could help me solve this issue.

[Vicraj Thomas]

Hi İlker,

The problem with execute scripts not running when a node boots up has
been fixed on the Georgia Tech rack but not on the other racks. That's
why you are continuing to see this problem on the Kentucky racks. The
InstaGENI dev team plans on doing some more testing before pushing the
fix to all racks. We are hoping it soon gets pushed to the racks you
need---Sarah has emailed the dev team about this.

I'm sorry for the trouble you've had. Thanks for hanging in there.

< Vic

On 12/1/14, 3:21 PM, İlker Özçelik wrote:
> Hi Sarah,
>
> I tried my topology (using 7 AMs). It worked except the OVS image. I am
> still getting the same error.
>
> sudo ovs-vsctl show
> 2014-12-01T20:51:46Z|00001|reconnect|WARN|unix:/usr/local/var/run/openvswitch/db.sock:
> connection attempt failed (Connection refused)
> ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database
> connection failed (Connection refused)
>
>
> I think this is a corrupted image problem. OVS image worked at UtahDCC
> Instageni but it fails at Kentucky PSK2 Instageni. Luisa suggested
> Kentucky PSK2 Instageni. It is the only place which has enough bandwidth
> to support 5 stitched link with 20Mbps at the same time. I would really
> appreciate if you could help me solve this issue.
>
> On Monday, December 1, 2014 12:52:43 PM UTC-5, Sarah Edwards wrote:
>
> Ok. I tried the hellogeni rspec again at GeorgiaTech and this time
> it seems to have worked.
>
> Thank you so much!
>
> Is the fix deployed everywhere?
>
> Ilker, can you give it a try and see if your problem is fixed?
>
> Sarah
>
> On Dec 1, 2014, at 12:32 PM, Leigh Stoller <lbst...@gmail.com

> <javascript:>> wrote:
>
> >> Yes.
> >
> > Okay, give it another try please … thanks!
> >
> > Leigh
> >
> >
> >
> >
> >
> > --
> > GENI Users is a community supported mailing list, so please help
> by responding to questions you know the answer to.
> >
> > If this is your first time posting a question to this list,
> please review
> http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
> <http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList>
>
> > ---
> > You received this message because you are subscribed to the
> Google Groups "GENI Users" group.
> > To unsubscribe from this group and stop receiving emails from it,

> send an email to geni-users+...@googlegroups.com <javascript:>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> *******************************************************************************
>
> Sarah Edwards
> GENI Project Office
>
> BBN Technologies
> Cambridge, MA
> phone: (617) 873-2329

> email: sedw...@bbn.com <javascript:>

>
>
>
>
>
> --
> GENI Users is a community supported mailing list, so please help by
> responding to questions you know the answer to.
>
> If this is your first time posting a question to this list, please
> review
> http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
> ---
> You received this message because you are subscribed to the Google
> Groups "GENI Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to geni-users+...@googlegroups.com

> <mailto:geni-users+...@googlegroups.com>.

[Leigh Stoller]

> The problem with execute scripts not running when a node boots up has been fixed on the Georgia Tech rack but not on the other racks. That's why you are continuing to see this problem on the Kentucky racks. The InstaGENI dev team plans on doing some more testing before pushing the fix to all racks. We are hoping it soon gets pushed to the racks you need---Sarah has emailed the dev team about this.

The fix has been pushed to all racks ...

Leigh

[Sarah Edwards]

Thank you!

Sarah

> --
> GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.
>
> If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
> ---
> You received this message because you are subscribed to the Google Groups "GENI Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[İlker Özçelik]

Thanks.