File Transfer

[CinnamonDonkey]

Hi All,

I'm fairly new to Python and very new to RPyC but it's all been eye
opening after 15 years of C/C++ :-).

I was wondering, what would be the best approach to take to implement
file transfer functionality using RPyC? I want to be able to transfer
a large local file to the remote machine placing it at a specified
location. The file will be binary.

All help will be very much appreciated!

Regards,
Shaun

[redbaron]

Are you sure that RPyC is a good way to do it? I'm not sure but try to
open file and then send file handler to remote service. As it said in
docs all objects are passed by reference, so reading from that handler
on remote side will actually make transfer from local machine to
remote onee. I'm not sure will it work or not, but you could try it =)

f = open(/path/to/file,"rb")
... send of on remote side
....do f.read() on remote side.

be careful that f.read() will allocate memory equal to file size, if
its really big then try to read it chunk by chunk like:

g = open("/path/where/to/write","wb")
chunk = f.read(1024*1024)
while chunk:
g.write(chunk)
chunk = f.read(1024*1024)

[CinnamonDonkey]

Thanx for the reply RedBaron.

I agree, I don't think it is the best way of doing a file transfer but
based on my limited knowledge of Python and RPyC I'd rather spend my
time learning a few systems well than lots of systems badly. At least
that is my theory at the moment, everything is subject to change ;-).

I do need, RPC for some genuine tasks later so I figured this would be
a good starting point as it satisfies two goals. How to transfer my
file and how to use RPyC :-D.

I have to say, I am very impressed with this Python'ing lark... Oh
what I have been missing all these years ;-).

[CinnamonDonkey]

I have written this on my server side:


class MyService( rpyc.Service ):
class exposed_FileTransfer( ):
def exposed_Open( filename ):
print "FILE TRANSFER OPEN FUNCTION CALLED - " + filename
return 0

if __name__ == "__main__":
s = ThreadedServer( MyService, port = 1234, reuse_addr = True )
s.start()

But on my client side I can't figure how to call it. I have tried:

if __name__ == "__main__":
connection = rpyc.connect( options.serviceHostName,
options.servicePortNum )
connection.root.FileTransfer.Open( "SPANKING~!!!" )

But I get the error:

TypeError: unbounded method expose_Open() must be called with
exposed_FileTransfer instance as first argument.

eh? I tried:

if __name__ == "__main__":
connection = rpyc.connect( options.serviceHostName,
options.servicePortNum )
ft = connection.root.FileTransfer
ft.Open( "SPANKING~!!!" )

and that does not work... I'm confussed.

Please help.




On 17 Feb, 09:17, CinnamonDonkey <CinnamonDon...@googlemail.com>
wrote:

[CinnamonDonkey]

No worries! I worked it out :)

SERVER SHOULD BE:

class MyService( rpyc.Service ):
class exposed_FileTransfer( ):

def exposed_Open( self, filename ):

print "FILE TRANSFER OPEN FUNCTION CALLED - " + filename
return 0

if __name__ == "__main__":
s = ThreadedServer( MyService, port = 1234, reuse_addr = True )
s.start()

CLIENT SHOULD BE:

if __name__ == "__main__":
connection = rpyc.connect( options.serviceHostName,
options.servicePortNum )

tf = connection.root.FileTransfer()
tf.Open( "SPANKING~!!!" )



On 17 Feb, 12:25, CinnamonDonkey <CinnamonDon...@googlemail.com>
wrote:

[tomer filiba]

nice, but you might want to check out rpyc/utils/classic.py -- it already has file transfer functionality :)
it does require the classic mode, but assuming you can get a remote file object,
you can just use shutil.copyfileobj (you should check out shutil, btw, if you're new to python)

i.e.

# == server ==
class FileService(rpyc.Service):
    def exposed_open(self, filename, mode = "r"):
        return open(filename, mode)

# == client ==
c = rpyc.connect(host, port)

# copy to client
remote = c.root.open("/foo/bar")
local = open("/tmp/foo/bar", "w")
shutil.copyfileobj(remote, local)

# copy to server
local = open("/spam/bacon")
remote = c.root.open("/tmp/spam/bacon", "w")
shutil.copyfileobj(local, remote)


hope it helps,
-tomer

--
An NCO and a Gentleman

[CinnamonDonkey]

It looks like this can't be done this way.

both using manual read/write and using shutil results in the same
error message.

File "c:\python25\Lib\Site-packages\rpyc\core\protocol.py", line
329, in sync_request raise obj
AttributeError: cannot access write.



:-(

On 17 Feb, 12:52, tomer filiba <tomerfil...@gmail.com> wrote:
> nice, but you might want to check out rpyc/utils/classic.py -- it already
> has file transfer functionality :)
> it does require the classic mode, but assuming you can get a remote file
> object,
> you can just use shutil.copyfileobj (you should check out shutil, btw, if
> you're new to python)
>
> i.e.
>
> # == server ==
> class FileService(rpyc.Service):
>     def exposed_open(self, filename, mode = "r"):
>         return open(filename, mode)
>
> # == client ==
> c = rpyc.connect(host, port)
>
> # copy to client
> remote = c.root.open("/foo/bar")
> local = open("/tmp/foo/bar", "w")
> shutil.copyfileobj(remote, local)
>
> # copy to server
> local = open("/spam/bacon")
> remote = c.root.open("/tmp/spam/bacon", "w")
> shutil.copyfileobj(local, remote)
>
> hope it helps,
> -tomer
>

> On Tue, Feb 17, 2009 at 14:30, CinnamonDonkey <CinnamonDon...@googlemail.com

[CinnamonDonkey]

This seems to work as a solution -

SERVER SIDE:

class MyService( rpyc.Service ):
class exposed_File():
def exposed_open( self, filename, mode = "r" ):
print "Received call to Open file: " + filename + ", in mode: " +
mode
self.__filehandle = open( filename, mode )
self.__filename = filename

def exposed_write( self, bytes ):
return self.__filehandle.write( bytes )

def exposed_close( self ):
print "Closing file: " + self.__filename
return self.__filehandle.close()

CLIENT SIDE:
connection = rpyc.connect( HostName, PortNum )

remote = connection.root.File()
remote.open( "c:\\temp\\remote.test", "wb" )
local = open( "c:\\temp\\local.test", "rb" )

chunk = local.read( 1024*1024 )
while chunk:
remote.write( chunk )
chunk = local.read( 1024*1024 )

remote.close()
local.close()


:-)



On 17 Feb, 14:02, CinnamonDonkey <CinnamonDon...@googlemail.com>
wrote:

[tomer filiba]

yeah, sorry, i forgot you're not using the classic mode, so you have to explicitly expose stuff.
cool :)

by the way, shutil.copyfileobj does just what you did down there -- so instead of writing it yourself,
i'd recommend using it.


enjoy
-tomer

[CinnamonDonkey]

Thanx for the advice and help tomer!

On 17 Feb, 15:25, tomer filiba <tomerfil...@gmail.com> wrote:
> yeah, sorry, i forgot you're not using the classic mode, so you have to
> explicitly expose stuff.
> cool :)
>
> by the way, shutil.copyfileobj does just what you did down there -- so
> instead of writing it yourself,
> i'd recommend using it.
>
> enjoy
> -tomer
>

> On Tue, Feb 17, 2009 at 17:13, CinnamonDonkey <CinnamonDon...@googlemail.com

[M S Vishwanath Bhat]

Hi,

I am a newbie to python in general and rpyc in particular. Please forgive my ignorance.

How can I achieve file transfer.

I added the file open part in my service.py in server. And then I ran both my own rpyc service and rpyc_classic.py.

Now I use the client to to copy_to_server and I still get the write error.

Any help is appreciated.

Best Regards,
Vishwanath

[Tomer Filiba]

It seems you're not using the right configuration. See http://rpyc.readthedocs.org/en/latest/api/core_protocol.html#rpyc.core.protocol.DEFAULT_CONFIG

You should set allow_all_attrs = true. Anyway, if you're using the ClassicService on both sides of the connection, it should set these attributes automatically. 

-tomer

-----------------------------------------------------------------

Tomer Filiba  tomerfiliba.com

On Wed, Mar 19, 2014 at 8:44 PM, M S Vishwanath Bhat <msv...@gmail.com> wrote:

On Thursday, 20 March 2014 00:07:12 UTC+5:30, M S Vishwanath Bhat wrote:

Hi,

I am a newbie to python in general and rpyc in particular. Please forgive my ignorance.

How can I achieve file transfer.

I added the file open part in my service.py in server. And then I ran both my own rpyc service and rpyc_classic.py.

Now I use the client to to copy_to_server and I still get the write error.

 

Any help is appreciated.

I have connected to my service and the file is being opened with same connection. Should I connect to classic service and use that to open a file remotely.
BTW this is the traceback I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "run_helper.py", line 99, in copy_to_server
    shutil.copyfileobj(localc, remotec)
  File "/usr/lib64/python2.6/shutil.py", line 31, in copyfileobj
    fdst.write(buf)
  File "/usr/lib/python2.6/site-packages/rpyc/core/netref.py", line 150, in __getattr__
    return syncreq(self, consts.HANDLE_GETATTR, name)
  File "/usr/lib/python2.6/site-packages/rpyc/core/netref.py", line 71, in syncreq
    return conn.sync_request(handler, oid, *args)
  File "/usr/lib/python2.6/site-packages/rpyc/core/protocol.py", line 438, in sync_request
    raise obj
AttributeError: cannot access 'write'

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/rpyc/core/protocol.py", line 300, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/usr/lib/python2.6/site-packages/rpyc/core/protocol.py", line 538, in _handle_getattr
    return self._access_attr(oid, name, (), "_rpyc_getattr", "allow_getattr", getattr)
  File "/usr/lib/python2.6/site-packages/rpyc/core/protocol.py", line 501, in _access_attr
    raise AttributeError("cannot access %r" % (name,))
AttributeError: cannot access 'write'


MS

--

---
You received this message because you are subscribed to the Google Groups "rpyc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rpyc+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yehonatan Arad]

Hi Tomer,

I want to get file from server to the client.

I configured the open function on the rpyc_classic.py but I didn't success to call it from the other computer.

I got this error:

Can you help me?

Thanks

>>> remote = c.root.open("FILE")

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "C:\Python26\lib\site-packages\rpyc\core\netref.py", line 150, in __getattr__

    return syncreq(self, consts.HANDLE_GETATTR, name)

  File "C:\Python26\lib\site-packages\rpyc\core\netref.py", line 71, in syncreq

    return conn.sync_request(handler, oid, *args)

  File "C:\Python26\lib\site-packages\rpyc\core\protocol.py", line 441, in sync_request

    raise obj

AttributeError: 'SlaveService' object has no attribute 'exposed_open'

========= Remote Traceback (1) =========

Traceback (most recent call last):

  File "C:\Python26\lib\site-packages\rpyc\core\protocol.py", line 305, in _dispatch_request

    res = self._HANDLERS[handler](self, *args)

  File "C:\Python26\lib\site-packages\rpyc\core\protocol.py", line 541, in _handle_getattr

    return self._access_attr(oid, name, (), "_rpyc_getattr", "allow_getattr", getattr)

  File "C:\Python26\lib\site-packages\rpyc\core\protocol.py", line 507, in _access_attr

    return accessor(obj, name, *args)

  File "C:\Python26\lib\site-packages\rpyc\core\service.py", line 69, in _rpyc_getattr

    return getattr(self, name)

AttributeError: 'SlaveService' object has no attribute 'exposed_open'

"

בתאריך יום שלישי, 17 בפברואר 2009 14:52:05 UTC+2, מאת Tomer Filiba:

[Tomer Filiba]

Use c.builtin.open (or builtins, can't remember), not c.root.open

--

[Yehonatan Arad]

Thanks very much. I success now to copy the file, on classic mode.

Now i want to change the method to service mode. (because security issues).

What I need to do for this?

On server I ran this file:

class DoStuffService(rpyc.Service):

   def on_connect(self):

       "Do some things when a connection is made"

   def on_disconnect(self):

       "Do some things AFTER a connection is dropped"

   def exposed_func1(self, *args, **kws):

       "Do something useful and maybe return a value"

   def exposed_func2(self, *args, **kws):

       "Like func1, but do something different"

if __name__ == '__main__':

   rpyc.utils.server.ThreadedServer(DoStuffService).start()

On client I tried to connect such:

c = rpyc.connect(IP,PORT,service = "DoStuffService")

I get this error, why it can be?

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "C:\Python26\lib\site-packages\rpyc\utils\factory.py", line 89, in connect

    s = SocketStream.connect(host, port, ipv6 = ipv6, keepalive = keepalive)

  File "C:\Python26\lib\site-packages\rpyc\core\stream.py", line 132, in connect

    return cls(cls._connect(host, port, **kwargs))

  File "C:\Python26\lib\site-packages\rpyc\core\stream.py", line 102, in _connect

    s.connect(sockaddr)

  File "<string>", line 1, in connect

socket.error: [Errno 10061] No connection could be made because the target machine actively refused it

בתאריך יום רביעי, 10 בספטמבר 2014 08:02:07 UTC+3, מאת Tomer Filiba:

[Yehonatan Arad]

I succeed:

on the server:

import rpyc

from rpyc.utils.server import ThreadedServer, ForkingServer, OneShotServer

class DpaServerService(rpyc.Service):

def exposed_open(self, filename):

return open(filename, "r")

if __name__ == '__main__':

server = ThreadedServer(DpaServerService, port = 12345)

server.start()

and on the client:

c = rpyc.connect("IP","PORT")

remote= c.root.open("File")

local = open("File","w")

shutil.copyfileobj(remote, local)

remote.close()

local.close() 

בתאריך יום רביעי, 10 בספטמבר 2014 09:38:33 UTC+3, מאת Yehonatan Arad:

[Yehonatan Arad]

Hi,

I want to run this service on the boot computer.

I think to run it on service, but I have a problems on it.

Do you have idea?

Thanks,

Yehonatan

בתאריך יום רביעי, 10 בספטמבר 2014 12:28:39 UTC+3, מאת Yehonatan Arad:

[M S Vishwanath Bhat]

On 15 September 2014 12:56, Yehonatan Arad <yon...@gmail.com> wrote:

Hi,

I want to run this service on the boot computer.

I think to run it on service, but I have a problems on it.

Do you have idea?

I had the same requirement. This is what I did.

(works only on Linux systems)

I modified the service to run as a daemon (like a service in linux). Wrote a init script inside and /etc/init.d And then added the service to chkconfig to start at boot time.

Best Regards,

Vishwanath

--

---
You received this message because you are subscribed to a topic in the Google Groups "rpyc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rpyc/91kMD-wy5vk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rpyc+uns...@googlegroups.com.

[Yehonatan Arad]

Thanks, I need it also to windows.

In addition, Can you write here more about your process on linux?

Thanks

בתאריך יום שני, 15 בספטמבר 2014 10:30:59 UTC+3, מאת M S Vishwanath Bhat:

[Matthew Cummings]

Hi Yehonatan,

This is something I've been wanting to do myself (both on Windows and Linux).  In my case, with my current project, I'm going try running SSH servers on my Windows hosts and spawning RPyC processes on demand via SSH.  BTW, that's what the Zero Deployment utility does http://rpyc.readthedocs.org/en/latest/api/utils_zerodeploy.html.  

I'm not entirely convinced it's going to work well on Windows, because SSH isn't native to Windows, not because I don't think RPyC won't work correctly - I feel like I'm the biggest fan of this library, it's one of my favorite Python modules!  :)

That being said, if you google around, you'll find various recipes for running Python scripts as Windows services.  Again, like running SSH servers, that's not really a "natural" thing to do on Windows.  Windows services usually run executables, not scripts.

Here's one page that shows how to do it:

http://stackoverflow.com/questions/32404/is-it-possible-to-run-a-python-script-as-a-service-in-windows-if-possible-how

Cheers,
Matthew Cummings

[dmnicol]

Just learned about rpyc last week, very impressive, lots of promise.

if I could just learn to to make it do what I wanted on my Mac

Right now I having trouble with copying files, using Mac OS X Mavericks, in case that matters.  

Experiment #1 --- use classic mode, follow instructions to import rpyc and make a connection.

The very same unhappiness below happens with different python implementations (2.7 and 3.4.1)

dmnicol-MacBook-Pro-5:dmnicol$ python3

Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import rpyc

>>> conn = rpyc.classic.connect('localhost')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/utils/classic.py", line 68, in connect

    return factory.connect(host, port, SlaveService, ipv6 = ipv6, keepalive = keepalive)

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/utils/factory.py", line 89, in connect

    s = SocketStream.connect(host, port, ipv6 = ipv6, keepalive = keepalive)

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/core/stream.py", line 132, in connect

    return cls(cls._connect(host, port, **kwargs))

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/core/stream.py", line 102, in _connect

    s.connect(sockaddr)

ConnectionRefusedError: [Errno 61] Connection refused

-----------

well, huh.

I actually started first with service oriented mode, and looked at classic mode only when

I couldn't get service oriented stuff to work.

Here's the server code

------- fileServer.py ------

import rpyc

import sys

import argparse

from rpyc.utils.server import ThreadedServer

# this code comes straight off a 'this works for me' example

class FileService(rpyc.Service):

    def exposed_open(self,filePath, mode = 'r'):

        return open(filePath,mode)

# argparse is overkill for this, but I'm a fan

parser = argparse.ArgumentParser()

parser.add_argument(u'-port',  metavar = u'use this port', dest= u'port', required=True)

args = parser.parse_args( sys.argv[1:] )

if __name__ == "__main__":

    # just like numerious examples on the web

    t = ThreadedServer(FileService, port=int(args.port))

    t.start()

--------------

now the client code

------- fileClient.py -------

from __future__ import print_function

import rpyc

import argparse

import os

import sys

import shutil

# after this args.directory holds directory path and args.port the port number of the server

parser = argparse.ArgumentParser()

parser.add_argument(u'-port',  metavar = u'use this port', dest= u'port', required=True)

parser.add_argument(u'-dir',  metavar = u'local directory name', dest= u'directory', required=True)

args = parser.parse_args( sys.argv[1:] )

# proof that the problem is not junk coming in from the command line

if not os.path.isdir(args.directory):

    print ('directory', args.directory,'not accessible')

    exit (1)

# use os.path.join to show problem is not due to badly formated path name

localPath     = os.path.join(args.directory,'destination.txt')

remotePath    = os.path.join(args.directory,'source.txt')

# test each line separately to isolate the failure

try:

localFile     = open(localPath, 'w')

except:

        print('cannot open',localPath)

        exit

try:

    conn          = rpyc.connect('localhost', port = int(args.port))

except:

    print('rpyc.connect failed')

    exit

try:

remoteFile    = conn.root.open( remotePath, 'r' )

except:

print('conn.root.open failed')

exit

try:

    shutil.copyfileobj(remoteFile, localFile)

except:

    print ('copyfileobj error:', sys.exc_info()[0])

---------------

so let's take it out for a spin

-----------

dmnicol-MacBook-Pro-5:np-live dmnicol$ 

dmnicol-MacBook-Pro-5:np-live dmnicol$ ls testing/

source.txt

dmnicol-MacBook-Pro-5:np-live dmnicol$ python3 fileServer.py -port 11111 &

[1] 24764

dmnicol-MacBook-Pro-5:np-live dmnicol$ python3 fileClient.py -port 11111 -dir 'testing'

copyfileobj error: <class 'AttributeError'>

-----------

and what attribute error might that be?  Bring in pdb, step to the offending line and see

-> shutil.copyfileobj(remoteFile, localFile)

(Pdb) shutil.copyfileobj(remoteFile, localFile)

*** AttributeError: cannot access 'read'

========= Remote Traceback (1) =========

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/core/protocol.py", line 305, in _dispatch_request

    res = self._HANDLERS[handler](self, *args)

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/core/protocol.py", line 541, in _handle_getattr

    return self._access_attr(oid, name, (), "_rpyc_getattr", "allow_getattr", getattr)

  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/rpyc/core/protocol.py", line 504, in _access_attr

    raise AttributeError("cannot access %r" % (name,))

AttributeError: cannot access 'read'

------------------------

I'm not so sophomoric to believe I'm not doing something wrong here, but the code is taken nearly

exactly as I'm finding examples to follow.   Any help will be appreciated.