Include folders in --onefile new question

[Laurence Anthony]

Back in 2010, Martin Zibricky wrote about adding a folder to --onefile.
The method was simply to change the following:
coll = COLLECT(exe, a.binaries, ...

to:
coll = COLLECT(exe, a.binaries + [('testFolder', 'C:/testFolder',
'DATA')], ...

I'm trying to do this now, but I keep getting an "IOError: [Errno 13]
Permission denied" error, even when it's a simple folder containing just
a plain text file. I've tried with various folders in various locations.
The problem is the same. I also confirmed that the above line causes the
error. Removing the + [('testFolder', 'C:/testFolder', 'DATA')] part of
collect results in a perfect build with no errorrs. Here's the traceback:

Traceback (most recent call last):
File "Z:\@@portable_apps\PortableApps\PyInstaller\utils\Build.py",
line 56, in <module>
PyInstaller.build.main(args[0], **opts.__dict__)
File
"Z:\@@portable_apps\PortableApps\PyInstaller\PyInstaller\build.py", line
1709, in main
build(specfile, buildpath)
File
"Z:\@@portable_apps\PortableApps\PyInstaller\PyInstaller\build.py", line
1666, in build
execfile(spec)
File
"Z:\@@eclipse_workspace\RPyProjects\bundleAll\PypeR_demo_001.spec", line
22, in <module>
name=os.path.join('dist', 'PypeR_demo_001'))
File
"Z:\@@portable_apps\PortableApps\PyInstaller\PyInstaller\build.py", line
1257, in __init__
self.__postinit__()
File
"Z:\@@portable_apps\PortableApps\PyInstaller\PyInstaller\build.py", line
312, in __postinit__
self.assemble()
File
"Z:\@@portable_apps\PortableApps\PyInstaller\PyInstaller\build.py", line
1289, in assemble
shutil.copy2(fnm, tofnm)
File
"Z:\@@portable_apps\PortableApps\PythonPortable\App\lib\shutil.py", line
128, in copy2
copyfile(src, dst)
File
"Z:\@@portable_apps\PortableApps\PythonPortable\App\lib\shutil.py", line
82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 13] Permission denied: 'C:/testFolder'

Can someone confirm that Martin's approach really works, and perhaps
suggest where I'm going wrong? If you have an alternative approach, that
would be fine, too.

Thank you!
Laurence.

Below is Martin's complete solution:

#####################
Pyinstaller should support that. However I didn't use that yet. But:

In your spec file you should tweak the collect section.

coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=False,
name=os.path.join('dist', __testname__))

like:

coll = COLLECT(exe,
a.binaries +
[('skins', '/my/project/skins', 'DATA')],
a.zipfiles,
a.datas,
strip=False,
upx=False,
name=os.path.join('dist', __testname__))

the skins folder should be accessible at runtime like:

os.path.join(os.environ["_MEIPASS2], 'skins'))

#####################

[Hartmut Goebel]

Am 12.05.2013 13:07, schrieb Laurence Anthony:

Back in 2010, Martin Zibricky wrote about adding a folder to --onefile.  The method was simply to change the following:
coll = COLLECT(exe, a.binaries, ...

to:
coll = COLLECT(exe, a.binaries + [('testFolder', 'C:/testFolder', 'DATA')], ...

I doubt, this ever as been working, as DATA is marking a file ever since.

You could try adding a hidden file, which should create the directory:

  [('testFolder/.dummy', 'some_empty_file', 'DATA')]

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP

Goebel Consult
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/2011-02-fleisige-datensammler-fur-lukratives-geschaeftsmodell-gesucht
Blog: http://www.goebel-consult.de/blog/20060121

Goebel Consult ist Mitglied bei http://www.7-it.de/

[Laurence]

On Monday, 13 May 2013 00:13:16 UTC+9, Hartmut Goebel wrote:

Am 12.05.2013 13:07, schrieb Laurence Anthony:

Back in 2010, Martin Zibricky wrote about adding a folder to --onefile.  The method was simply to change the following:
coll = COLLECT(exe, a.binaries, ...

to:
coll = COLLECT(exe, a.binaries + [('testFolder', 'C:/testFolder', 'DATA')], ...

I doubt, this ever as been working, as DATA is marking a file ever since.

You could try adding a hidden file, which should create the directory:

  [('testFolder/.dummy', 'some_empty_file', 'DATA')]

--

Thanks for the reply. If I want to add the entire contents of a directory (and its sub directories), does this mean that the best solution is to walk through the source directory structure and add each file one at a time using some script? I saw a similar suggestion on StackOverflow: http://stackoverflow.com/a/13656127/548598. If I do this, can I then refer to the copied data structure using "testFolder" (perhaps combined with the very useful script here: http://stackoverflow.com/a/7675014/548598)?

Laurence.

[Laurence]

Following from my earlier post, I've now solved the issue:

In Pyinstaller 2, instead of adding files to COLLECT, I just add the following after a = Analysis line in the spec file:

### STARTS HERE
def extra_datas(mycwd,mydir):
    import os
    oldcwd = os.getcwd()
    os.chdir( mycwd )
   
    def rec_glob(p, files):
        import os
        import glob
        for d in glob.glob(p):
            if os.path.isfile(d):
                files.append(d)
            rec_glob("%s/*" % d, files)
           
    files = []
    rec_glob("%s/*" % mydir, files)
    extra_datas = []
    for f in files:
        extra_datas.append((f, os.path.join(mycwd,f), 'DATA'))
       
    os.chdir( oldcwd )
    return extra_datas



# append the 'mydir' directory
mycwd= <path to parent folder of mydir>
a.datas += extra_datas(mycwd,mydir)

### ENDS HERE

I got the basic idea from http://stackoverflow.com/a/12033695/548598, but needed to adapt it to handle relative and absolute paths. This is because Pyinstaller uses the following pattern:
a.datas += [ ( <relativepath stored in dist>, <absolutepath of file to add>, DATA)]

I'm not sure if there is a way to avoid all the relative/absolute path hacks, but if there is, I'd like to know. Also, I read that TREE can be used to walk through a directory. However, I couldn't get this to work and had use the above script to walk through the directory myself.

Laurence.

[Hartmut Goebel]

Hi Laurence,

def extra_datas(mycwd,mydir):

This looks like you are reimplementing os.walk(). And to be frank: this is ugly, ugly code (I'm glad that it's not yours, so I can say this :-)

that TREE can be used to walk through a directory. However, I couldn't get this to work and had use the above script to walk through the directory myself.

Well, the function is called "Tree", not "TREE", you can find the documentation here:
<http://www.pyinstaller.org/export/v2.0/project/doc/Manual.html?format=raw#tree>

What is your exact problem with it? (I have not used it yet, but it looks as if it would solve your problem. Thanks for ponting to it :-)

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP

Goebel Consult
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/2012-12-in-die-cloud-in-die-cloud-ab-wo-soll-die-sein
Blog: http://www.goebel-consult.de/blog/20050401

[Laurence Anthony]

Hi again,

 

def extra_datas(mycwd,mydir):

This looks like you are reimplementing os.walk(). And to be frank: this is ugly, ugly code (I'm glad that it's not yours, so I can say this :-)

Yes, it's ugly and it's not mine!! :-)  But, I can confirm that it does work and it works well. I could bundle the entire R stats library and it works perfectly. I'll now tidy up the code, but I wanted to give credit to the person who originally suggested the approach.

that TREE can be used to walk through a directory. However, I couldn't get this to work and had use the above script to walk through the directory myself.

Well, the function is called "Tree", not "TREE", you can find the documentation here:
<http://www.pyinstaller.org/export/v2.0/project/doc/Manual.html?format=raw#tree>

What is your exact problem with it? (I have not used it yet, but it looks as if it would solve your problem. Thanks for ponting to it :-)

It might work. I just don't understand the docs! :-). If I call Tree(mydir), what exactly is returned. A list? A string? An iterator?
If mydir is like the following:

c:/mydir
c:/mydir/text1.txt
c:/mydir/text2.txt

for the a.datas addition approach to work, I need the following structure:

a.datas += [ ('/mydir/text1.txt', 'c:/mydir/text1.txt', DATA),  ('/mydir/text2.txt', 'c:/mydir/text2.txt', DATA) ]

So if I call Tree('c:/mydir'), what is returned?

I know that I should just try this myself. But, my hack already works and I've already spent hours on this! Any quick pointers would be great.

Laurence.

[Hartmut Goebel]

Am 14.05.2013 13:36, schrieb Laurence Anthony:

It might work. I just don't understand the docs! :-). If I call Tree(mydir), what exactly is returned. A list? A string? An iterator?

Yes, it's a bit short :-) Thanks to Nat Picker there is a better description available now:
<https://rawgithub.com/pyinstaller/pyinstaller/develop/doc/Manual.html#the-tree-class>

Is this understandable for you? (Otherwise we should again improve  it)

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP

Goebel Consult
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/2011-09-kommerz-uber-recht-fdp-die-gefaellt-mir-partei
Blog: http://www.goebel-consult.de/blog/20060920

[Laurence Anthony]

On Tue, May 14, 2013 at 8:53 PM, Hartmut Goebel <h.go...@goebel-consult.de> wrote:

Am 14.05.2013 13:36, schrieb Laurence Anthony:

It might work. I just don't understand the docs! :-). If I call Tree(mydir), what exactly is returned. A list? A string? An iterator?

Yes, it's a bit short :-) Thanks to Nat Picker there is a better description available now:
<https://rawgithub.com/pyinstaller/pyinstaller/develop/doc/Manual.html#the-tree-class>

Is this understandable for you? (Otherwise we should again improve  it)

Wow... I just read through the entire manual. It is so much clearer. Nat Picker has done a great job.

Unfortunately, the Tree section is still a bit cryptic. Here are the relevant bits:

####
TOC Class (Table of Contents)
...
Basically a TOC object contains a list of tuples of the form
    (name,path,typecode)
....
The Tree Class

The Tree class is a way of creating a TOC that describes some or all of the files within a directory:
    Tree(root, prefix=run-time-folder, excludes=match)

###

So, if Tree is creating a TOC, how does root, prefix, (and excludes) relate to (name, path, typecode) of TOC.

I guess <prefix> become the first part of <name> but it is not clear how the <root> path gets combined with <name>.

Then, clearly <path> and <root> are related, but how is not clear either.

The big problem is <typecode>. What typecode does the Tree command create. Is it always <DATA>?

Perhaps a simple example using a bundle folder (e.g. dist) and a source folder  (e.g. c:/testfolder) would be good. See my earlier example:

c:/mydir
c:/mydir/text1.txt
c:/mydir/text2.txt

So, how do I get c:/mydir into the dist folder?

Is it simply for a one-dir:
collect = COLLECT(a.binaries + Tree('c:/mydir'), ...

And for a one-file bundle:
exe = EXE(..., a.binaries + + Tree('c:/mydir'), ....

Or:
collect = COLLECT(a.binaries + Tree('c:/mydir', 'mydir'), ...
exe = EXE(..., a.binaries + + Tree('c:/mydir', 'mydir'), ....

or something else?

An example explaining this would be really useful.

Laurence.

[Hartmut Goebel]

Am 14.05.2013 14:50, schrieb Laurence Anthony:

So, if Tree is creating a TOC, how does root, prefix, (and excludes) relate to (name, path, typecode) of TOC.

Good point.

From the code I can tell you, that the typecode is always DATA.

For the rest I suggest testing it. I'v done it four you:

print Tree('xx')
# [('yy/zz.txt', 'xx/yy/zz.txt', 'DATA')]

print Tree('xx', 'aaa')
# [('aaa/yy/zz.txt', 'xx/yy/zz.txt', 'DATA')]

print Tree('/usr/share/icons/large', 'my-icons')
# [('my-icons/mageiaupdate.png', '/usr/share/icons/large/mageiaupdate.png', 'DATA'), ...


For me this is quite obviose

So, how do I get c:/mydir into the dist folder?

Simply list it, lika any other TOC:

COLLECT(a.binaries,
   a.pure,

   Tree('c:/mydir'),
  ...

> An example explaining this would be really useful.

Please open an issue for this, including the examples above.

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP

Goebel Consult
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/2012-04-compliance-bringt-keine-sicherheit
Blog: http://www.goebel-consult.de/blog/20050401

[Laurence Anthony]

So, if Tree is creating a TOC, how does root, prefix, (and excludes) relate to (name, path, typecode) of TOC.

Good point.

From the code I can tell you, that the typecode is always DATA.

For the rest I suggest testing it. I'v done it four you:

print Tree('xx')
# [('yy/zz.txt', 'xx/yy/zz.txt', 'DATA')]

print Tree('xx', 'aaa')
# [('aaa/yy/zz.txt', 'xx/yy/zz.txt', 'DATA')]

print Tree('/usr/share/icons/large', 'my-icons')
# [('my-icons/mageiaupdate.png', '/usr/share/icons/large/mageiaupdate.png', 'DATA'), ...


For me this is quite obviose

So, how do I get c:/mydir into the dist folder?

Simply list it, lika any other TOC:

COLLECT(a.binaries,
   a.pure,

   Tree('c:/mydir'),
  ...

> An example explaining this would be really useful.

Please open an issue for this, including the examples above.

Wonderful. So clear and obvious. If the manual had this in it, everything would make sense.

Laurence.

 

[Hartmut Goebel]

Am 14.05.2013 15:42, schrieb Laurence Anthony:

Wonderful. So clear and obvious. If the manual had this in it, everything would make sense.

We would appreciate you patch for the manual :-)

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP

Goebel Consult
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/2012-02-bring-your-own-life-glosse
Blog: http://www.goebel-consult.de/blog/200505010