ImportError: Import by filename is not supported when unpickleing

[Larry Martell]

When I try and unpickle an object with pickle.loads it fails with:

ImportError: Import by filename is not supported when unpickleing

I've never used pickle before. Why do I get this and how can I fix it?

[Jason Benjamin]

Try using *pickle.load* instead of *pickle.loads*. pickle.loads is for
strings. Retrieved from documentation: help(pickle) after importing
pickle.



--
[The Computer] was the first machine man built that assisted the power of
his brain instead of the strength of his arm. - Grace Hopper

[Larry Martell]

On Wednesday, July 27, 2016, Jason Benjamin <hexus...@gmail.com

<javascript:_e(%7B%7D,'cvml','hexus...@gmail.com');>> wrote:

> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote:
>
> > When I try and unpickle an object with pickle.loads it fails with:
> >
> > ImportError: Import by filename is not supported when unpickleing
> >
> > I've never used pickle before. Why do I get this and how can I fix it?
>
> Try using *pickle.load* instead of *pickle.loads*. pickle.loads is for
> strings. Retrieved from documentation: help(pickle) after importing
> pickle.
>

I am unpickling a string created with pickle.dumps

[Larry Martell]

On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin <hexus...@gmail.com> wrote:
> If it has and 's' on the end it will only work on strings. *dumps* refers
> to a string too.

Yes, I know. I have an object, which I pickle with dumps, which turns
it into a string. Then I try to unpickle it with loads and I get that
error.


(Pdb) type(args.target)
<type 'str'>
(Pdb) pickle.loads(args.target)
*** ImportError: Import by filename is not supported.


> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell <larry....@gmail.com>
> wrote:

>
>
>
> On Wednesday, July 27, 2016, Jason Benjamin <hexus...@gmail.com> wrote:
>>
>> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote:
>>
>> > When I try and unpickle an object with pickle.loads it fails with:
>> >
>> > ImportError: Import by filename is not supported when unpickleing
>> >
>> > I've never used pickle before. Why do I get this and how can I fix it?
>>
>> Try using *pickle.load* instead of *pickle.loads*. pickle.loads is for
>> strings. Retrieved from documentation: help(pickle) after importing
>> pickle.
>
>

[Larry Martell]

On Wed, Jul 27, 2016 at 7:45 PM, Jason Benjamin <hexus...@gmail.com> wrote:
> Look at this: https://wiki.python.org/moin/UsingPickle
> It uses *pickle.dump* not *pickle.dumps*

Yes that uses a file. I do not want to use a file. I want to pass the
object as a string.

> If you still don't get it send me the code for the function with a working
> code example the calls that function revealing the error.
>
> I've had a lot of experience with Python and it looks like its the sequence
> of functions that causes the problem and not just one Python function call
> on its own.

I have an object of type Target:

(Pdb) type(target)
<class 'workitem.Target'>

And I pickle it like this:

(Pdb) type(pickle.dumps(target))
<type 'str'>

And then it looks like this:

(Pdb) pickle.dumps(target)
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

And I pass it into a subprocess.Popen call. Then in the program that
is called, it comes in as a string:

(Pdb) type(args.target)
<type 'str'>

With the same content:

(Pdb) args.target
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

And when I unpickle it I get the error:

(Pdb) pickle.loads(args.target)
***ImportError: Import by filename is not supported.


Also, please keep the discussion on the list, and please don't top post.

>
>
>
> On Wed, Jul 27, 2016 at 4:29 PM, Larry Martell <larry....@gmail.com>
> wrote:
>
> On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin <hexus...@gmail.com>
> wrote:
>
> If it has and 's' on the end it will only work on strings. *dumps* refers to
> a string too.
>
> Yes, I know. I have an object, which I pickle with dumps, which turns it
> into a string. Then I try to unpickle it with loads and I get that error.
> (Pdb) type(args.target) <type 'str'> (Pdb) pickle.loads(args.target) ***
> ImportError: Import by filename is not supported.
>
> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell <larry....@gmail.com>
> wrote: On Wednesday, July 27, 2016, Jason Benjamin <hexus...@gmail.com>
> wrote:
>

> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote: > When I try and
> unpickle an object with pickle.loads it fails with: > > ImportError: Import
> by filename is not supported when unpickleing > > I've never used pickle
> before. Why do I get this and how can I fix it? Try using *pickle.load*
> instead of *pickle.loads*. pickle.loads is for strings. Retrieved from
> documentation: help(pickle) after importing pickle.
>

[Larry Martell]

Also let me add that initially I was calling Popen with shell=False
and the arguments in a list, and that was failing with:

arg 2 must contain only strings

And when I debugged I found that this was happening in the execvp call
in subprocess and arg 2 was a str. So then I changed it using
shell=False with the command line in a string, and that is when I get
the ImportError error.

>> On Wed, Jul 27, 2016 at 4:29 PM, Larry Martell <larry....@gmail.com>
>> wrote:
>>
>> On Wed, Jul 27, 2016 at 6:54 PM, Jason Benjamin <hexus...@gmail.com>
>> wrote:
>>
>> If it has and 's' on the end it will only work on strings. *dumps* refers to
>> a string too.
>>
>> Yes, I know. I have an object, which I pickle with dumps, which turns it
>> into a string. Then I try to unpickle it with loads and I get that error.
>> (Pdb) type(args.target) <type 'str'> (Pdb) pickle.loads(args.target) ***
>> ImportError: Import by filename is not supported.
>>
>> On Wed, Jul 27, 2016 at 3:44 PM, Larry Martell <larry....@gmail.com>
>> wrote: On Wednesday, July 27, 2016, Jason Benjamin <hexus...@gmail.com>
>> wrote:
>>

>> On Wed, 27 Jul 2016 17:25:43 -0400, Larry Martell wrote: > When I try and
>> unpickle an object with pickle.loads it fails with: > > ImportError: Import
>> by filename is not supported when unpickleing > > I've never used pickle
>> before. Why do I get this and how can I fix it? Try using *pickle.load*
>> instead of *pickle.loads*. pickle.loads is for strings. Retrieved from
>> documentation: help(pickle) after importing pickle.
>>

[Steven D'Aprano]

On Thursday 28 July 2016 12:39, Larry Martell wrote:

> I have an object of type Target:
>
> (Pdb) type(target)
> <class 'workitem.Target'>
>
> And I pickle it like this:
>
> (Pdb) type(pickle.dumps(target))
> <type 'str'>
>
> And then it looks like this:
>
> (Pdb) pickle.dumps(target)
>
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-
CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

Gag me with a spoon!

What happens if you do

s = pickle.dumps(target)
pickle.loads(s)

Do you get the same error?

> And I pass it into a subprocess.Popen call. Then in the program that
> is called, it comes in as a string:
>
> (Pdb) type(args.target)
> <type 'str'>
>
> With the same content:
>
> (Pdb) args.target
>
"ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-

CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."

Are you sure it's the same content, and not just the same to the naked eye?

> And when I unpickle it I get the error:
>
> (Pdb) pickle.loads(args.target)
> ***ImportError: Import by filename is not supported.

Does the target process already have workitem imported? What happens if you
import workitem first?




By the way, just in case this is relevant to you... pickle is an insecure
format. If your target process is running in a context where it can receive
pickles from untrusted clients, you're vulnerable to them running arbitrary
code on your machine.

If you are trying to do a remote procedure call, rather than invent your own,
you should use a reliable, trusted library. I've used both rpyc and Pyro and
can recommend them both:

https://pypi.python.org/pypi/rpyc/
https://pypi.python.org/pypi/Pyro4/


--
Steve

[Larry Martell]

On Thu, Jul 28, 2016 at 1:58 AM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> On Thursday 28 July 2016 12:39, Larry Martell wrote:
>
>> I have an object of type Target:
>>
>> (Pdb) type(target)
>> <class 'workitem.Target'>
>>
>> And I pickle it like this:
>>
>> (Pdb) type(pickle.dumps(target))
>> <type 'str'>
>>
>> And then it looks like this:
>>
>> (Pdb) pickle.dumps(target)
>>
> "ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-
> CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
> CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."
>
> Gag me with a spoon!
>
> What happens if you do
>
> s = pickle.dumps(target)
> pickle.loads(s)
>
> Do you get the same error?

No that works fine. I get back a target object with the same content.

>> And I pass it into a subprocess.Popen call. Then in the program that
>> is called, it comes in as a string:
>>
>> (Pdb) type(args.target)
>> <type 'str'>
>>
>> With the same content:
>>
>> (Pdb) args.target
>>
> "ccopy_reg\n_reconstructor\np0\n(cworkitem\nTarget\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'histologySections'\np6\n(lp7\nsS'registrationTransforms'\np8\nV\np9\nsS'valueMaps'\np10\n(dp11\nsS'targetID'\np12\nVRight-
> CarotidArtery\np13\nsS'targetInitializer'\np14\ng9\nsS'regions'\np15\n(dp16\nsS'bodySite'\np17\nVRightCarotid\np18\nsS'targetLocalFolderName'\np19\nV21597135/wi-54976537/Right-
> CarotidArtery\np20\nsS'readingsLocalFileName'\np21\ng9\nsS'probabilityMaps'\np22\n(dp23\nsS'targetPath'\np24\ng9\nsb."
>
> Are you sure it's the same content, and not just the same to the naked eye?
>
>> And when I unpickle it I get the error:
>>
>> (Pdb) pickle.loads(args.target)
>> ***ImportError: Import by filename is not supported.
>
> Does the target process already have workitem imported? What happens if you
> import workitem first?

Yes, workitem is already imported.

Here is something interesting I just discovered. In the called
program, if I print out argv[8] (which is the pickled item) and then
copy/paste it into loads it works fine. But if I call loads directly
with argv[8] I get the error. But I don't know what the significance
is of that data point is.

> By the way, just in case this is relevant to you... pickle is an insecure
> format. If your target process is running in a context where it can receive
> pickles from untrusted clients, you're vulnerable to them running arbitrary
> code on your machine.
>
> If you are trying to do a remote procedure call, rather than invent your own,
> you should use a reliable, trusted library. I've used both rpyc and Pyro and
> can recommend them both:
>
> https://pypi.python.org/pypi/rpyc/
> https://pypi.python.org/pypi/Pyro4/

I'm calling a local program, and this is an in house run script.

[Nobody]

On Wed, 27 Jul 2016 22:47:15 -0400, Larry Martell wrote:

> Also let me add that initially I was calling Popen with shell=False and
> the arguments in a list, and that was failing with:
>
> arg 2 must contain only strings

That indicates that you're calling Popen() incorrectly.

> And when I debugged I found that this was happening in the execvp call in
> subprocess and arg 2 was a str. So then I changed it using shell=False
> with the command line in a string, and that is when I get the ImportError
> error.

This is just hiding the error. Passing the command line as a string will
result in the string being parsed in some way (the details depend upon the
OS). Unless the string was assembled with the correct quoting, this will
corrupt the string.

I'd suggest starting off by ensuring that the string received by the child
process is the one which you intended to pass. Calculate and display (or
log) its hash (see the hashlib module) in both the sender and receiver and
confirm that they're the same.

[Larry Martell]

Thanks, but I have abandoned using pickle. I am now converting my
objects to JSON, writing them to files, passing the file names to the
process and reading them in and converting them back to objects there.
In addition to that working, it makes the process much easier to
debug.

[Lutz Horn]

Hi,

Am 07/28/2016 um 03:48 PM schrieb Larry Martell:
> Thanks, but I have abandoned using pickle. I am now converting my
> objects to JSON, writing them to files, passing the file names to
> the process and reading them in and converting them back to objects
> there. In addition to that working, it makes the process much easier
> to debug.

I think that is a very good idea. JSON is the universal format today and
mapping to and from Python is very easy.

Lutz

--
https://www.lhorn.de/
https://emailselfdefense.fsf.org/de/