New feature; repository based profiles

[Leigh Stoller]

Some of you might already have noticed, but there is a new button on the
Create Profile page named "Git Repo". This is a new feature we are trying
out, we would love to hear feedback about it.

In short, this new button allows you to link your Cloudlab profile to a
public git repository.

More info about "repository-based profiles" can be found in a demonstration
git repository at https://github.com/lbstoller/my-profile

The public Cloudlab profile that is linked to the above repository is at:
https://www.cloudlab.us/p/PortalProfiles/RepoBased

Leigh

[Divyashri Bhat]

Hi Leigh,

Thank you for incorporating this feature.

I understand that this is a new feature but is it possible to specify a git branch from where one wishes to clone and is it possible to specify different branches for various nodes? 

I am currently working on a public git repo for a system that I have instantiated using CloudLab. This requires me to execute different scripts on different types of nodes in my profile. I believe that CloudLab allows me to execute various scripts but clones the entire git repo on all nodes. Is there anyway that I can choose to clone only a certain branch, maybe? 

Leigh

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-users+unsubscribe@googlegroups.com.
To post to this group, send email to cloudlab-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/01510F98-6912-4053-97CE-6A4CF5F360D8%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Divyashri Bhat

Graduate Research Assistant

University of Massachusetts, Amherst

E-mail: divyash...@gmail.com

[Leigh Stoller]

> I understand that this is a new feature but is it possible to specify a
> git branch from where one wishes to clone and is it possible to specify
> different branches for various nodes?

Hi. So glad to have a user try it out! So the way we worked out the
interface is that you select the branch to instantiate at the Portal, we
clone the entire repo to each node and then switch the clones to that
branch (or tag).

So, you could add an execute service to each node that switches the clone
on that node to whatever branch you need to be in (in /local/repository).
And then you can run your other execute services.

Would this solve your problem?

Leigh

[Divyashri Bhat]

That makes sense :) thanks Leigh!

I will try this and keep you posted 

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
To post to this group, send email to cloudla...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/C6ECACA1-EC86-4A1E-84E5-1D62F6DF18EF%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Divyashri Bhat]

Hi Leigh,

I was successful in creating a Profile using a GitHub repo and when I instantiated the profile it also appears that the repo has been cloned into the nodes.

Here are some issues I ran into:

1) When I tried to use the Update button to pull the latest repo I would always get an error message "Bad dataset" and I had to delete and create a new profile in order to update. 

2) I did enable the automatic webhooks on my GitHub repo which went through without errors but I did not see an update on the Profile after pushing to my GitHub repo.

3) The execute command I have on my nodes does not seem to be executed. I poked around in the Manifest a bit and found that there are two exec commands inside the services tag; one which executes the git repo clone and branch selection and another with my execute command. Does it not get executed because of the multiple execute tags, maybe?

• The link to my CloudLab profile: https://www.cloudlab.us/manage_profile.php?action=edit&uuid=a6183489-17b2-11e7-ac8d-90e2ba22fee4
  
• The link to my experiment: https://www.cloudlab.us/status.php?uuid=bc86c9b6-17b2-11e7-ac8d-90e2ba22fee4.
  
• The link to my GitHub repo: https://github.com/dbhat/cloudlab_SABR
  

The instructions were really easy to follow :) Please let me know if you would like me try things out as I am currently working on automation and repeatability of one of my large-scale CloudLab experiments.

Thank you!

On Wed, Mar 29, 2017 at 1:58 PM, Divyashri Bhat <divyash...@gmail.com> wrote:

That makes sense :) thanks Leigh!

I will try this and keep you posted 

On Wed, Mar 29, 2017 at 11:03 PM Leigh Stoller <lbst...@gmail.com> wrote:

> I understand that this is a new feature but is it possible to specify a
> git branch from where one wishes to clone and is it possible to specify
> different branches for various nodes?

Hi. So glad to have a user try it out! So the way we worked out the
interface is that you select the branch to instantiate at the Portal, we
clone the entire repo to each node and then switch the clones to that
branch (or tag).

So, you could add an execute service to each node that switches the clone
on that node to whatever branch you need to be in (in /local/repository).
And then you can run your other execute services.

Would this solve your problem?

Leigh




--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-users+unsubscribe@googlegroups.com.
To post to this group, send email to cloudlab-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/C6ECACA1-EC86-4A1E-84E5-1D62F6DF18EF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Divyashri Bhat

Graduate Research Assistant

University of Massachusetts, Amherst

E-mail: divyash...@gmail.com

[Leigh Stoller]

Thanks for the info! Can you please extend your experiment for another
day, I will look into everything first thing Monday morning.

Thanks!
Leigh

> On Apr 2, 2017, at 12:57 PM, Divyashri Bhat wrote:
>
> Hi Leigh,
>
> I was successful in creating a Profile using a GitHub repo and when I instantiated the profile it also appears that the repo has been cloned into the nodes.
>
> Here are some issues I ran into:
> 1) When I tried to use the Update button to pull the latest repo I would always get an error message "Bad dataset" and I had to delete and create a new profile in order to update.
> 2) I did enable the automatic webhooks on my GitHub repo which went through without errors but I did not see an update on the Profile after pushing to my GitHub repo.
> 3) The execute command I have on my nodes does not seem to be executed. I poked around in the Manifest a bit and found that there are two exec commands inside the services tag; one which executes the git repo clone and branch selection and another with my execute command. Does it not get executed because of the multiple execute tags, maybe?
> • The link to my CloudLab profile: https://www.cloudlab.us/manage_profile.php?action=edit&uuid=a6183489-17b2-11e7-ac8d-90e2ba22fee4
> • The link to my experiment: https://www.cloudlab.us/status.php?uuid=bc86c9b6-17b2-11e7-ac8d-90e2ba22fee4.
> • The link to my GitHub repo: https://github.com/dbhat/cloudlab_SABR
> The instructions were really easy to follow :) Please let me know if you would like me try things out as I am currently working on automation and repeatability of one of my large-scale CloudLab experiments.
>
> Thank you!
>

> On Wed, Mar 29, 2017 at 1:58 PM, Divyashri Bhat wrote:
> That makes sense :) thanks Leigh!
>
> I will try this and keep you posted
>

> On Wed, Mar 29, 2017 at 11:03 PM Leigh Stoller wrote:
> > I understand that this is a new feature but is it possible to specify a
> > git branch from where one wishes to clone and is it possible to specify
> > different branches for various nodes?
>
> Hi. So glad to have a user try it out! So the way we worked out the
> interface is that you select the branch to instantiate at the Portal, we
> clone the entire repo to each node and then switch the clones to that
> branch (or tag).
>
> So, you could add an execute service to each node that switches the clone
> on that node to whatever branch you need to be in (in /local/repository).
> And then you can run your other execute services.
>
> Would this solve your problem?
>
> Leigh
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/C6ECACA1-EC86-4A1E-84E5-1D62F6DF18EF%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> --
> Regards,
> Divyashri Bhat
> Graduate Research Assistant
> University of Massachusetts, Amherst
> E-mail: divyash...@gmail.com
>
>
>
> --
> Regards,
> Divyashri Bhat
> Graduate Research Assistant
> University of Massachusetts, Amherst
> E-mail: divyash...@gmail.com
>

Leigh

[Divyashri Bhat]

Thank you! I have extended it by 3 days.

> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-users+unsubscribe@googlegroups.com.
> To post to this group, send email to cloudlab-users@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/C6ECACA1-EC86-4A1E-84E5-1D62F6DF18EF%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.
> --
> Regards,
> Divyashri Bhat
> Graduate Research Assistant
> University of Massachusetts, Amherst
> E-mail: divyash...@gmail.com
>
>
>
> --
> Regards,
> Divyashri Bhat
> Graduate Research Assistant
> University of Massachusetts, Amherst
> E-mail: divyash...@gmail.com
>

Leigh

--
You received this message because you are subscribed to the Google Groups "cloudlab-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-users+unsubscribe@googlegroups.com.
To post to this group, send email to cloudlab-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/6537D238-23A4-4A0C-BA1C-E51A5EDD97C0%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Leigh Stoller]

Hi Divyashri, here is where things stand:

> 1) When I tried to use the Update button to pull the latest repo I would
> always get an error message "Bad dataset" and I had to delete and
> create a new profile in order to update.

This is fixed.

> 2) I did enable the automatic webhooks on my GitHub repo which went
> through without errors but I did not see an update on the Profile
> after pushing to my GitHub repo.

This is also fixed.

Do take a moment to make sure that your push url is up to date, it will
change with each new profile, even if the repourl is the same.

> 3) The execute command I have on my nodes does not seem to be executed. I
> poked around in the Manifest a bit and found that there are two exec
> commands inside the services tag; one which executes the git repo
> clone and branch selection and another with my execute command. Does
> it not get executed because of the multiple execute tags, maybe?

I looked at this one. Your script is executing after the repo clone.
Take a look /var/emulab/logs on one of the nodes running the clients.sh
script, you will see the execution status. The very last line of this file
is the script that runs, and if you look in that file you can see where
the output is redirected (into files in /var/tmp). There is an error
reported when running your clients.sh script. It might be our error, but
this info should be enough to chase it down further and let us know.

Thanks!
Leigh

[jdelli...@gmail.com]

Hi Leigh,

I'm a little confused as to under which user the execution services are running under. 

<execute shell="sh" command="/local/repository/setup.sh"/>

It seems like maybe it's geniuser? What's odd, however, is that my setup script encounters permission denied errors on certain things like 

env > info.txt

Error from /var/tmp/startup-1.txt:

/local/repository/setup.sh: line 25: info.txt: Permission denied

or 

wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz

Shows this in /var/tmp/startup-1.txt:

--2017-10-23 22:44:26--  http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 195257604 (186M) [application/x-gzip]
hadoop-2.6.0.tar.gz: Permission denied


Cannot write to 'hadoop-2.6.0.tar.gz' (Permission denied).

But other things seem to execute just fine:

cat > ~/.bashrc <<EOM
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-amd64
export EDITOR=vim
EOM

cat > ~/.ssh/config <<EOM

Host *

    StrictHostKeyChecking no

EOM

This is all from my profile here:

https://github.com/jdellithorpe/ldbc-snb-datagen-cloudlab-profile.git

What is the execution environment that node execution services run in?

Best,

Jonathan

[Leigh Stoller]

> wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
>
> Shows this in /var/tmp/startup-1.txt:
>
> --2017-10-23 22:44:26-- http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
> Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
> Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 195257604 (186M) [application/x-gzip]
> hadoop-2.6.0.tar.gz: Permission denied

Hi. I think the basic problem is that the directory you start out in is
unspecified by the system, and since you do not chdir or provide a -O
option to wget, you are probably writing to a directory you do not have
permission on.

Try making the directory explicit in your setup script (chdir /tmp).

Leigh

[jdelli...@gmail.com]

Hmmm I see...

I'm wondering if I've misunderstood how I should be interacting with a cloudlab cluster. Up to now I've been using my user account 'jde', but seeing as how my setup.sh script appears to be running as 'geniuser' (commands like touch ~/test.txt create files in /users/geniuser with owner geniuser), I'm wondering if I should be interacting with the cluster as 'geniuser' instead. This is particularly the case because I'd like my repository based CloudLab profile to be generally usable by any other user, and so don't want to make any assumptions about user names in the setup scripts (e.g. hardcoding my username somewhere in there).

However, another problem is that I don't seem to have access to the geniuser account...

So essentially two questions:

1) Should I be using the geniuser account?

2) If so, how do I get access to the geniuser account?

3) If not, how do I make my setup scripts generic so anyone can use it with their accounts?

Best,

Jonathan

[Leigh Stoller]

> So essentially two questions:
> 1) Should I be using the geniuser account?

Can you point me to a running example please, so I can look.

Thanks!
Leigh

[jdelli...@gmail.com]

Hi Leigh,

Sure! I have a running experiment @ 

jde-QV29797 This is part of the RAMCloud project (ramcloud-pg0) Jonathan

[Leigh Stoller]

OK, just wanted to make sure things were behaving as the should. The
execution environment is a little confusing, cause of how we instantiate
experiments on remote clusters.

> So essentially two questions:
> 1) Should I be using the geniuser account?

The geniuser account is just a pseudo user that we use when instantiating
on remote clusters. The startup scripts end up running as that user.

> 2) If so, how do I get access to the geniuser account?

You don't really want to. Just use "sudo" in your script to do what you
need to (for example, install software or change system settings).

> 3) If not, how do I make my setup scripts generic so anyone can use it
> with their accounts?

So the main thing to remember is that the set of user accounts will be
different depending on who is instantiating your profile. For that reason
you do not want to depend on any user account. Software should be installed
in standard locations (/usr/local, /opt/, etc) and should be runnable by
whoever is logged into the machine.

Leigh

[jdelli...@gmail.com]

Hmm ok, that makes sense. One thing that I am working on in particular is installing SSH keys for the user so that they can ssh between machines without needing to go to through root... but it's occurred to me now that that kind of setup would need to be done for all users on the machine so as to be sort of user-agnostic...

Well anyway, thanks again for your help!

Jonathan

[Leigh Stoller]

> Hmm ok, that makes sense. One thing that I am working on in particular is
> installing SSH keys for the user so that they can ssh between machines
> without needing to go to through root... but it's occurred to me now that
> that kind of setup would need to be done for all users on the machine so
> as to be sort of user-agnostic...

Hi, if you want your users to only use the browser based console, then yes
this is a problem since there is no way to forward your private key with an
ssh agent.

But if your users ssh in from their desktop machine, they can ssh between
machines without you have to generate keys. They just need to upload their
public key to the Portal web interface, and run an ssh agent to forward
their private key.

We point people to this nice tutorial: https://help.github.com/articles/connecting-to-github-with-ssh/

Leigh

[Rajvardhan Deshmukh]

Hi Leigh,

How large can the repository be? Can its size be around 14-16G?

Thanks,
Raj

[Leigh Stoller]

> How large can the repository be? Can its size be around 14-16G?

That would be too big. Consider that the repo needs to cloned to all
of the nodes, and the file system won't have enough space for it.

At the moment we enforce a max size of 50MB on the repo, but we could
relax that to 1G if that would help.

Leigh