Playlist file format and related issues
W. Trevor King
http://yaml.org/spec/
using the PyYAML package
http://pyyaml.org/
for (de)serializing.
Pros:
+ Makes encoding CommandStacks, etc. easier, which makes sense as it's
designed to be a data serialization language and not a markup
language.
Cons (with counter-arguments):
- Introduces an external dependency.
+ PyYAML is pure python, so install is easy
- Stores playlists in a more complicated format.
+ Complicated things (e.g. CommandStacks) will be complicated
anyway. At least with YAML they'll be more compact. Simple
things will be simple in either format, so users should be able to
pick up the basics quickly.
- Breaks backwards compatibility with current XML playlists.
+ It would be easy to write an upgrade script to convert old
playlists to the new format.
Bonus side-effects:
+ Allows replacing my
http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/file/532a566a42c0/hooke/util/convert.py
with calls to yaml.dump and .load
Another alternative would be the pickle module, but then human
readability really suffers.
Also, there currently seems to be no standard way to save "how did I
process that data" information. This is what I'm trying to achieve
with per-curve CommandStacks.
http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/rev/d58538cbc7e2
However:
- Commands/arguments/etc. may change, breaking compatibility with
old CommandStacks.
- CommandStacks don't save (potentially relavant) configuration
data (e.g. the default median window for flat-filtering).
Both the config file and playlists should record the Hooke version
used to create them. This would allow the user to rollback to the
appropriate version of Hooke if they wanted to recreate previous
analysis. It would also allow the construction of (semi-)automatic
scripts upgrading between different version.
Thoughs?
Trevor
--
This email may be signed or encrypted with GPG (http://www.gnupg.org).
The GPG signature (if present) will be attached as 'signature.asc'.
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
W. Trevor King
> Also, there currently seems to be no standard way to save "how did I
> process that data" information. This is what I'm trying to achieve
> with per-curve CommandStacks.
Actually, you can always save the whole list of commands you send to
Hooke with the command line UI, and then load that script with
$ ./bin/hooke --script=your-script --persist
This doesn't answer "what did I do to this curve?" gracefully if
you've been through a few rounds of playlist filtering, but it does
answer "how did I process the data?". Perhaps there should be a
CommandMessage -> command-line converter to automatically save logs at
the CommandEngine level... In any case, I no longer consider this
recording problem an issue.
YAML playlists would still be nice, though, since we need the
CommandStacks to regenerate curves that have been unloaded,
http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/file/c278ad972a7f/hooke/playlist.py#l140
and YAML CommandStacks would allow you to restart analysis at any
playlist along the analysis route.
ms
> On Thu, Aug 12, 2010 at 03:57:53PM -0400, W. Trevor King wrote:
>> Also, there currently seems to be no standard way to save "how did I
>> process that data" information. This is what I'm trying to achieve
>> with per-curve CommandStacks.
>
> Actually, you can always save the whole list of commands you send to
> Hooke with the command line UI, and then load that script with
> $ ./bin/hooke --script=your-script --persist
>
> This doesn't answer "what did I do to this curve?" gracefully if
> you've been through a few rounds of playlist filtering, but it does
> answer "how did I process the data?". Perhaps there should be a
> CommandMessage -> command-line converter to automatically save logs at
> the CommandEngine level... In any case, I no longer consider this
> recording problem an issue.
>
> YAML playlists would still be nice, though, since we need the
> CommandStacks to regenerate curves that have been unloaded,
> http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/file/c278ad972a7f/hooke/playlist.py#l140
> and YAML CommandStacks would allow you to restart analysis at any
> playlist along the analysis route.
>
I admit that I don't know YAML but sounds interesting. What about
keeping both options, with automatic recognition of file format? :)
cheers,
M.
W. Trevor King
> What about keeping both options,
The main motivator was trying to avoid implementing CommandStack
storage in XML. Keeping both would mean maintaining two CommandStack
serializations :(. Unless XML people were not interested in recording
their CommandStacks. This is how things work in the current Hooke.
From macro.py in the default branch:
Note that macros applied to playlists should end by export
commands so the processed curves are not lost
CommandStacks allow you to maintain that information, so you can come
back and continue or review your analysis later on. But typed command
parameters + XML = annoying.
ms
> On Fri, Aug 13, 2010 at 06:22:10PM +0100, ms wrote:
>> What about keeping both options,
>
> The main motivator was trying to avoid implementing CommandStack
> storage in XML. Keeping both would mean maintaining two CommandStack
> serializations :(. Unless XML people were not interested in recording
> their CommandStacks. This is how things work in the current Hooke.
> From macro.py in the default branch:
> Note that macros applied to playlists should end by export
> commands so the processed curves are not lost
> CommandStacks allow you to maintain that information, so you can come
> back and continue or review your analysis later on. But typed command
> parameters + XML = annoying.
>
Ouch, I see.
Since the new GUI/etc. Hooke seems a much different beast than its
humble predecessor, I would say that I have no objections about that.
cheers,
m.
W. Trevor King
http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/rev/8d2db52043a5
On Sat, Aug 14, 2010 at 05:39:48PM +0100, ms wrote:
> Since the new GUI/etc. Hooke seems a much different beast than its
> humble predecessor, I would say that I have no objections about that.
It's not *doing* much that it didn't do before, it's just a bit more
flexible/modular.
ms
> YAML playlists working as of
> http://www.physics.drexel.edu/~wking/code/hg/hgwebdir.cgi/hooke/rev/8d2db52043a5
>
> On Sat, Aug 14, 2010 at 05:39:48PM +0100, ms wrote:
>> Since the new GUI/etc. Hooke seems a much different beast than its
>> humble predecessor, I would say that I have no objections about that.
>
> It's not *doing* much that it didn't do before, it's just a bit more
> flexible/modular.
Doesn't it include the GUI?
ms
> On Sun, Aug 22, 2010 at 06:11:35PM +0100, ms wrote:
>> On 22/08/10 15:16, W. Trevor King wrote:
>>> On Sat, Aug 14, 2010 at 05:39:48PM +0100, ms wrote:
>>>> Since the new GUI/etc. Hooke seems a much different beast than its
>>>> humble predecessor, I would say that I have no objections about that.
>>>
>>> It's not *doing* much that it didn't do before, it's just a bit more
>>> flexible/modular.
>>
>> Doesn't it include the GUI?
>
> Yeah, but the original still had the matplotlib plot, which is the
> main thing. The rest of the GUI is just like the command line, but
> it takes up more of your screen and you have to use your mouse ;).
>
Ok -it's different anyway :)