Re: [protogeni-users] Trying to use ftp on GENI

[Vicraj Thomas]

Hi Leland,

First, did you set IP addresses on the interfaces on both sides of the
link connecting your VMs? You can do this manually or using the auto-ip
button if you are using the Portal/Jacks.

Next, make sure you can ping from the client to the server. If your VM
at Clemson is called VMC and the VM at Utah is VMU, log into VMC and
ping VMU

If that works, your link is working. But the ftp command you are using
is incorrect. To ftp from the Clemson VM (VMC) to the Utah VM (VMU),
the command you need at VMC is:
ftp VMU (if your server is listening on the standard ftp port 21)
or
ftp VMU:nnn (if your server is listening on port nnn)

The problems with the ftp commands you are using are:
1. pc10 is not the username---it is part of the hostname where your VM
is located. The name of the host where your VM was allocated is
pc10.utahddc.geniracks.net.
2. 34363 is the port you use to log into your Utah VM from the Internet.
This is not the port on which your ftp server is listening.
3. From the client VM (VMC) you should refer to your server VM (VMU) by
the name you gave it (e.g. VMU) or by the IP address you assigned its
interface. You cannot get to it using the name pc10.utahddc.geniracks.net.

Finally, you might want to stay away from the Utah resources (UtahDDC,
Utah InstaGENI and Utah Protogeni) for now. UtahDDC will be down for
maintenance for a bit starting Monday. Connectivity to the other Utah
resources may be iffy during this time as they make changes to their
network.

Best wishes,

< Vic



On 4/4/15 2:00 PM, ldu...@g.clemson.edu wrote:
> ProtoGeni Users Group,
>
> I am an undergrad researcher trying to use ftp on GENI. Here's what I've
> done so far:
>
> I have allocated resources to two GENI VMs, one at Clemson and one at
> Utah, which are connected.
> I installed vsftpd and edited the .conf file on both VMs with the
> following steps:
> 1) sudo apt-get install vsftpd
> 2) sudo nano/etc/vsftpd.conf
> 3) uncommented anonymous_enable=yes
> 4) uncommented local_enable=yes
> 5) uncommented write_enable=yes
> 6) uncommented anon_upload_enable=yes
> 7) wrote the changes and exited nano
> 8) sudo service vsftpd restart
>
> I then tried to connect to the Utah node from the Clemson node via ftp
> by typing (34363 is the port number):
> ftp pc...@utahddc.geniracks.net 34363
>
> I keep getting errors saying that I'm not connected. Any help would be
> much appreciated. Please see attached; this is what the output and error
> looks like when I try to "put" a file to the other VM.
> I'm also not sure what "SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1" is.
>
> Thank you again! You guys are the best.
> Leland Dunwoodie
>
> --
> You received this message because you are subscribed to the Google
> Groups "protogeni-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protogeni-use...@googlegroups.com
> <mailto:protogeni-use...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Vicraj Thomas]

Hi Leland,

I'm not sure why your ftp connection is timing out. Here are a couple
of things you might want to try as you debug this:
1. ftp a very small file and see if that succeeds.
2. if the small file does not succeed, put your client and server on the
same rack and see if that succeeds.
In general, it is a good idea to debug your experiment on one rack
before going across racks. Also, start small.

Are you using a stitched link or a GRE tunnel to connect your VMs? As
you transfer large files, keep in mind that GRE tunnels goes over the
GENI control plane and traffic there is limited to 100 Mbps. If you are
going over a stitched link, the link bandwidth is set to 20 Mbps by
default, but you can increase it. In most cases you should use stitched
links instead of GRE tunnels.

I'm copying geni-users on my response as it reaches users in addition to
those on protogeni-users. Perhaps somebody in the larger community will
have other ideas on why this isn't working for you. To recv by email
responses sent to this group you should join it. See
http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList.

< Vic



On 4/4/15 10:04 PM, Leland Dunwoodie wrote:
> Vic,
>
> Thanks so much for your help!
>
> I made a new set of VMs. node-0 is at Clemson and node-1 is at U
> Chicago. I can ping between the nodes just fine. I downloaded vsftpd
> on both VMs just fine. I tried to ftp from Clemson to U Chicago and
> was able to connect but did not have success transferring data. Here's
> what happened:
> note: SRR039884.sra is the experimental file I'm trying to transfer;
> it is 81 MB in size. The connection timed out after about 30 minutes
> of displaying a blank line after "150 Ok to send data"
>
> ldunwoo@node-0:~$ ftp node-1
> Connected to node-1-link-0.
> 220 (vsFTPd 2.3.5)
> Name (node-1:ldunwoo): ldunwoo
> 331 Please specify the password.
> Password:
> 230 Login successful.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> put SRR039884.sra
> local: SRR039884.sra remote: SRR039884.sra
> 200 PORT command successful. Consider using PASV.
> 150 Ok to send data.
> netout: Connection timed out
> 421 Data timeout. Reconnect. Sorry.
> ftp>
>
>
> No rush to reply - I'm not sure why I'm doing this on Saturday night
> while the Final Four is on!

> Any help would be much appreciated.
>

> Thanks,
> Leland Dunwoodie

>> You received this message because you are subscribed to a topic in the
>> Google Groups "protogeni-users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/protogeni-users/VwoX2NdlyXY/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> protogeni-use...@googlegroups.com.

[Niky Riga]

Hi Leland,

Another thing you might want to try is to play with the MTU, you can either set the MTU in both of your
nodes to 1000 and see if that helps, or you just try a ping test:

ping <other_host>

ping -s 2000 <other_host>

Cheers,
Niky

Vicraj Thomas

April 5, 2015 at 6:52 PM

Hi Leland,

I'm not sure why your ftp connection is timing out.  Here are a couple of things you might want to try as you debug this:
1. ftp a very small file and see if that succeeds.
2. if the small file does not succeed, put your client and server on the same rack and see if that succeeds.
In general, it is a good idea to debug your experiment on one rack before going across racks.  Also, start small.

Are you using a stitched link or a GRE tunnel to connect your VMs?  As you transfer large files, keep in mind that GRE tunnels goes over the GENI control plane and traffic there is limited to 100 Mbps.  If you are going over a stitched link, the link bandwidth is set to 20 Mbps by default, but you can increase it.  In most cases you should use stitched links instead of GRE tunnels.

I'm copying geni-users on my response as it reaches users in addition to those on protogeni-users.  Perhaps somebody in the larger community will have other ideas on why this isn't working for you.  To recv by email responses sent to this group you should join it.  See http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList.

< Vic

Leland Dunwoodie

April 4, 2015 at 10:04 PM

Vicraj Thomas

April 4, 2015 at 6:09 PM

Hi Leland,

First, did you set IP addresses on the interfaces on both sides of the link connecting your VMs?  You can do this manually or using the auto-ip button if you are using the Portal/Jacks.

Next, make sure you can ping from the client to the server.  If your VM at Clemson is called VMC and the VM at Utah is VMU, log into VMC and
    ping VMU

If that works, your link is working.  But the ftp command you are using is incorrect.  To ftp from the Clemson VM (VMC) to the Utah VM (VMU), the command you need at VMC is:
   ftp VMU (if your server is listening on the standard ftp port 21)
or
   ftp VMU:nnn (if your server is listening on port nnn)

The problems with the ftp commands you are using are:
1. pc10 is not the username---it is part of the hostname where your VM is located.  The name of the host where your VM was allocated is pc10.utahddc.geniracks.net.
2. 34363 is the port you use to log into your Utah VM from the Internet.  This is not the port on which your ftp server is listening.
3. From the client VM (VMC) you should refer to your server VM (VMU) by the name you gave it (e.g. VMU) or by the IP address you assigned its interface.  You cannot get to it using the name pc10.utahddc.geniracks.net.

Finally, you might want to stay away from the Utah resources (UtahDDC, Utah InstaGENI and Utah Protogeni) for now.  UtahDDC will be down for maintenance for a bit starting Monday.  Connectivity to the other Utah resources may be iffy during this time as they make changes to their network.

Best wishes,

< Vic

ldu...@g.clemson.edu

April 4, 2015 at 2:00 PM

ProtoGeni Users Group,

I am an undergrad researcher trying to use ftp on GENI. Here's what I've done so far:

I have allocated resources to two GENI VMs, one at Clemson and one at Utah, which are connected.

I installed vsftpd and edited the .conf file on both VMs with the following steps:

1) sudo apt-get install vsftpd

2) sudo nano/etc/vsftpd.conf

3) uncommented anonymous_enable=yes

4) uncommented local_enable=yes

5) uncommented write_enable=yes

6) uncommented anon_upload_enable=yes

7) wrote the changes and exited nano

8) sudo service vsftpd restart

I then tried to connect to the Utah node from the Clemson node via ftp by typing (34363 is the port number):

ftp pc...@utahddc.geniracks.net 34363

I keep getting errors saying that I'm not connected. Any help would be much appreciated. Please see attached; this is what the output and error looks like when I try to "put" a file to the other VM.

I'm also not sure what "SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1" is.

Thank you again! You guys are the best.

Leland Dunwoodie

--
You received this message because you are subscribed to the Google Groups "protogeni-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protogeni-use...@googlegroups.com.

[Niky Riga]

Hi Leland,

I suspect that you do have an MTU problem, just set the MTU for the data interfaces to 1000 on both nodes:

sudo ifconfig eth1 mtu 1000

The two scp commands you have do use different interfaces:

ldunwoo@node-0:~$ scp -P 30267 SRR039884.sra ldu...@pc1.geni.uchicago.edu:/users/ldunwoo 
Uses the control plane, i.e. the internet

ldunwoo@node-0:~$ scp -P 30267 SRR039884.sra node-1:/users/ldunwoo

Uses the GENI network

To see info about the memory of your VM:
cat /proc/meminfo

Best,
niky

Leland Dunwoodie

April 6, 2015 at 3:18 PM

I am able to transfer small text files (~1100 bytes) across the GENI network between VMs at U Chicago and Clemson. I am not able transfer large files (~81 MB) across the same network. I went back and did a successful scp command to make sure that was working. I did a second scp command using the names of the VMs on the GENI network and that did not work. I get the sense that the successful scp command is not using the GENI network but the unsuccessful scp command is. Are both of these commands using the GENI network? Also, how do I see how much RAM is on my VM?

ldunwoo@node-0:~$ scp -P 30267 SRR039884.sra ldu...@pc1.geni.uchicago.edu:/users/ldunwoo

ldu...@pc1.geni.uchicago.edu's password:  100% 81MB 11.5MB/s 00:07

SRR039884.sra

ldunwoo@node-0:~$ scp -P 30267 SRR039884.sra node-1:/users/ldunwoo

The authenticity of host '[node-1]:30267 ([192.168.1.1]:30267)' can't be established.

RSA key fingerprint is 47:0b:36:98:17:3c:89:09:05:f3:4d:c1:fe:4a:4d:7f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '[node-1]:30267,[192.168.1.1]:30267' (RSA) to the list of known hosts.

 

ldunwoo@node-1's password:

SRR039884.sra 2% 2112KB 0.0KB/s - stalled - 

--

=========================
Leland Dunwoodie
Clemson '18
Biochemistry
Calhoun Honors College

"Do your best and forget the rest!"

Niky Riga

April 5, 2015 at 7:08 PM

Hi Leland,

Another thing you might want to try is to play with the MTU, you can either set the MTU in both of your
nodes to 1000 and see if that helps, or you just try a ping test:

ping <other_host>

ping -s 2000 <other_host>

Cheers,
Niky

--
GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.
 
If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
---
You received this message because you are subscribed to the Google Groups "GENI Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Leland Dunwoodie]

Ms. Riga,

Thank you very much!

Setting the mtu to 1000 allows me to use both scp and ftp over the GENI network! These are the commands I used that worked (node-0 is at Clemson and node-1 is at Wisconsin; SRR039884.sra is an 81MB sample file):

****Successful scp command****
ldunwoo@node-1:~$ sudo ifconfig eth1 mtu 1000                                                                                                        
ldunwoo@node-1:~$ scp SRR039884.sra node-0:/users/ldunwoo                                                                                            
ldunwoo@node-0's password:                                                                                                                           
SRR039884.sra                                                                                                      100%   81MB   2.3MB/s   00:35


****Successful ftp command****
****first, I installed vsftpd and edited its .conf file so anonymous_enable, local_enable,
write_enable, and anon_upload were all uncommented and =yes. Then, I restarted vsftpd.

ldunwoo@node-0:~$ ftp node-1                                                                                                                         
Connected to node-1-link-0.                                                                                                                          
220 (vsFTPd 2.3.5)                                                                                                                                   
Name (node-1:ldunwoo): ldunwoo                                                                                                                       
331 Please specify the password.                                                                                                                     
Password:                                                                                                                                            
230 Login successful.                                                                                                                                
Remote system type is UNIX.                                                                                                                          
Using binary mode to transfer files.                                                                                                                 
ftp> put SRR039884.sra                                                                                                                               
local: SRR039884.sra remote: SRR039884.sra                                                                                                           
200 PORT command successful. Consider using PASV.                                                                                                    
150 Ok to send data.                                                                                                                                 

226 Transfer complete.                                                                                                                               
84545295 bytes sent in 34.34 secs (2404.4 kB/s)

Thank you again!
Leland Dunwoodie