Use iPython workspace variables in a script

[Ian]

Say I run a script that creates the variable x in my iPython
workspace. I want to re-run the script (or another script) and use
the variable x from my workspace (not one created by the script).

The reason I want to do this is that x was loaded in from disk and is
quite large--therefore re-loading it takes some time. There are other
cases where I modify x and want to use the modified version.

[Will Martin]

Ian,

So the main thing you want to do is the following:
(1) load some big data once
(2) do computations with it interactively (i.e. scripts use the newest version)

Right?

This is a job for the class constructor. The pattern below will let you load the data once and then modify it with scripts (or at the command line). All scripts will use the most updated version of the data.

Let me know if this helps.

Cheers,
Will

[Ethan Coon]

Will's way works, or try a namespace (because namespaces are one
honking great idea, let's use more of those!)

Write a script that loads the data. Save it as filename.py

Then, every time you want to use the data, you can "import filename".
The first time you call import, it will run the loading script. Every
other time you import, it will not run the loading script. If you
want to force a reload of the data, use "reload filename". Now, if
you want to modify the data in-place, without touching the saved data
file, you can simply overwrite/alter/modifiy/whatever to the data,
filename.x. When your script calls "import filename", nothing will
happen, and the modified x will be used. Also, if you want to spoof
data, you can write alternative fake_filename.py scripts, and "import
fake_filename as filename" to get the spoofed data (with, for example,
a small subset of the data for testing purposes).

Note that none of this is ipython-specific, and would work equally
well in python code as in the ipython shell.

Ethan

> --
> William GK Martin
> Graduate Student of Applied Mathematics
> Columbia University
>
> email: wgm...@columbia.edu
> phone: 503-332-7227

--

-------------------------------------------------------------------
Ethan Coon
DOE CSGF - Graduate Student
Dept. Applied Physics & Applied Mathematics
Columbia University
212-854-0415

http://www.ldeo.columbia.edu/~ecoon/
-------------------------------------------------------------------

[Ian Langmore]

Thanks!

I tried this out.  Everything works with one caveat; 'reload' doesn't seem to work from a script.  Consider the following script, use_data.py:

################### use_data.py #####################
import load_data

processor = load_data.processor

# The default name is 'processor'
print processor.name 
processor.name = 'foo'
################################################

## Now consider the following from iPython:

In [1]: run use_data.py
processor

In [2]: run use_data.py
foo

## So the first run loaded the data and changed the name of the processor.  The second run did not reload the data.
## Now I run the same script, except that I have changed the 'import load_data' to 'reload load_data'.  This gives an error.
In [3]: run use_data.py
------------------------------------------------------------
   File "use_data.py", line 2
     reload load_data
                    ^
SyntaxError: invalid syntax

WARNING: Failure executing file: <use_data.py>

## So I get an error upon attempting to have the script reload the data using the script
## However, using the shell does work

In [4]: reload load_data
------> reload(load_data)
Out[4]: <module 'load_data' from 'load_data.pyc'>

In [5]: run use_data.py
processor

In 6]: run use_data.py
foo

[Ethan Coon]

Yeah, that's because the reload is a function, not a keyword (not sure what the technical term is, may not be keyword) like import is.

Note ipython is using magic by replacing your reload with a reload().  I suspect your script would work if you did the same.

reload(use_data)

Note that this is possible (though not necessary) of most of those keywords.  Scanning the docs will find built-in functions for import(), print(), and the like, which often have other options which can be convenient (like using print to write to a file).

I believe python is moving away from these keyword/specially mangled commands.  For instance, you used to be able to do:

raise ValueError, 'invalid value'

But now you must do:

raise ValueError('invalid value')

So you must instantiate the error class yourself, it won't magically do it for you in the interpreter. 

Ethan

[Will Martin]

Hey Ethan,

I just tried out your way. Pretty nifty use namespaces.

The only limitation seems to be that this way will support only one copy of the data. Once reload is called the modified data is lost (unless it is manually saved). And, even if it is saved the scripts will only work on the one copy that is stored as 'filename.x'.

The main benefit of using classes would be the ability to make and modify multiple copies of the data. Each instance has its own data and methods that use that data.

Right?

cheers,
Will

--
William GK Martin
Graduate Student of Applied Mathematics
Columbia University

email: wgm...@columbia.edu
phone: 503-332-7227

[Ethan Coon]

Yeah, that sounds right. Namespaces can almost be thought of as a
singleton class... a design pattern where there can only ever be one
instance, eg: http://en.wikipedia.org/wiki/Singleton_pattern

But that's what you want for a large, single data set. If you wanted
multiple data sets, you'd do better with a class, like you said.

Ethan