working with summing multiple images in a macro

[Sarah....@gmail.com]

The ability to easily sum multiple images is invaluable to the kind of
data reduction I want to do with ADM. If there was a way to
incorporate the opening/ summing of multiple files into a macro, that
would be really helpful.

[joshu...@gmail.com]

Hi Sarah! Thanks so much for posting into this discussion group. This
feature can definitely be added in some form or another. The only real
question is how the interface should work. Your idea was: "Maybe it
would be possible to sort the files with the same name into folders
and then have the macro add all the files in each folder and integrate
them. Maybe this is too difficult to be accomplished, but would be a
great feature for me."

In principle, all the steps like making new folders and moving files
around shouldn't be too hard for me to code up. It is the type of
thing that computers are really good at automating. The only thing
that I don't really understand is what exactly you mean by "files with
the same name". The problem I feel is that everybody who will want to
use this feature will have all their files having the same name BUT
with the names the same in slightly different ways. Is the any real
naming convention for diffraction data that you want to add together
later? So it seems like it would be hard to pick a convention that
even most people would agree upon. And it would really suck if
somebody had to rename all of their files just to use my program
because their convention was different.

Another possibility might be to make the user move all of the files
into subfolders manually and then have the user tell the macro the
folder name with all the subfolders inside of it. The program would
then add all of the files in one subfolder, do the analysis of the
combined image, and repeat for each of the subfolders. There would be
no real ambiguity to this approach because if several files were all
in the same subfolder, they would always get added together regardless
of their names. This would seem to be less complicated for the user,
but it would have the downside of requiring the user to manually
create a bunch of folders and then sort their data into them.

I am not convinced either way and there is probably an even better way
that neither of use have though of. Do you have any real preference?
Also, how many files are you dealing with? How long would it take you
to sort all your files into subfolders?

Josh

[Sarah....@gmail.com]

I agree that your idea is better. It would be pretty quick for me to
sort all my files into folders. I actually have a script that does it
according to my naming conventions. But to make it more universal, it
is a good idea to presort them into folders. I think that would work
for most people I know who collect the same type of data.

I also agree that there could be a better way that neither of us have
thought of. If it is easy, could we (I mean you in a friendly, we're
all in this together kind of way) put a band-aid on it while we are
awaiting inspiration for the best solution ever?

[joshu...@gmail.com]

I am not exactly sure what you mean by band-aid but I probably should
just code this feature up soon so that it can officially be included
in the version 1 release. I will try to code it up tonight and see how
it goes.

I think that I will introdcue a new macro command named "Multiple Data
Files:" that is analogous to "Data File:". Then, you can simply use a
syntax like this:

Mulitple Data Files:
[ first_file.mar3450 second_file.mar3450 third_file.mar3450]

so that at least in principle you can explicitly tell the program how
to do the grouping by writing out all the file names directly into the
macro file. Then, I will allow for doing this for multiple groups of
macro files by saying:

Mulitple Data Files:
[ first_file.mar3450 second_file.mar3450 third_file.mar3450]
[ new_first_file.mar3450 new_second_file.mar3450
new_third_file.mar3450]

and finally, you will be able to add folders into this list

Mulitple Data Files:
[ first_file.mar3450 second_file.mar3450 third_file.mar3450]
some_folder [ new_first_file.mar3450 new_second_file.mar3450
new_third_file.mar3450]

When run, all the files in each of the subfolders in some_folder will
be appended to this list with the [] brackets around them so that they
get added together and looped over when running the macros. This
should be flexible enough for the average user and if anybody thinks
of a better way to do it later, we can cross that bridge when we come
to it.

[Sarah....@gmail.com]

I'd love to try it out whenever you have something for me to try.
Sarah

[joshu...@gmail.com]

Hey. I've written it up and I think that it mostly works, but I am
still testing it out. It is really easy for macros to break so I have
to be cautious when changing stuff like this. I can upload the version
that I have now if you want to try it out to find bugs in it.
Otherwise, I'll upload it some time later today.

[Sarah....@gmail.com]

either way. Thanks a bunch for spending time on this!
Sarah

[joshu...@gmail.com]

Hey Sarah. The new version with these features is up. I coded it just
as I described above. The version number for the build of the code
with this new feature is 1.0.0. I hope you enjoy it. Also, if you find
any problems with it or are confused and can't get it to work, make
sure to ask me.

Best,

Josh

[Sarah....@gmail.com]

Josh-
I've spent a bit of time with this last night and this morning, and it
works GREAT! I'm actually super excited!!! I might have some
suggestions, but I'm going to try and solve my own problems first. I
want it to write my output files one directory up (not in the folder
with the files to be combine, but one directory up). I think that I
can do this by changing some stuff in my macro to make that work, but
I'll let you know if I can't figure it out.

Otherwise, this feature is working SUPER GREAT!!!!

Thanks so much!
Sarah

[Sarah....@gmail.com]

When it writes on output file, it names the file"MULTIPLE_FILES."
This isn't optimal because when there is more than one, it causes
problems. Can you ask it to name the output file the name of folder
that it is contained in? That would be better for me. since I would
be able to give the folder a descriptive name. I think that would
still be general enough for every one to use.

Thanks
Sarah

[Joshua Lande]

Thanks for your positive feedback. You are right that MULTIPLE_FILES
is not optimal. It would probably be better if the FILENAME was
replaced with the folder's name. Or maybe we should just introduce a
new macro syntax named SUBFOLDERNAME which does that instead. That
might be a little less confusing to the user and then we would keep
FILENAME still expanding to MULTIPLE_FILES. Maybe we could also
introduce the macro syntax SUBFOLDERPATH which refers to the path that
the current subfolder is in. That way, you could get the files where
you want with the right name by saying

SUBFOLDERPATHNAME/SUBFOLDERNAME_int.dat

Kind of wordy so maybe there is a better way to do this or at least a
better naming convention.

Josh

[Sarah....@gmail.com]

Josh-
The program seems to be choking on saving the files. Ive tried
several different permutations of the file output line. including:
SUBFOLDERPATHNAME/2th/SUBFOLDERNAME_2th.dat
SUBFOLDERPATHNAME/SUBFOLDERNAME_2th.dat
SUBFOLDERPATHNAME/SUBFOLDERNAME_int.dat (not sure what the int does)

But it doesn't want to save the files.

It gives an error message:
<type'exceptions.IOError'>Exception in Tk callback Function: <bound
method Main.runMacro of <_main_.Main instance at 0x01494990>> (type:
<type'instancemethod'>)
Args: ()
Traceback (innermost last):
File "PmwFreeze.pyc", line1784, in_call_
None
File "areadiffractionmachine.py", line 365, in runMacro
None
File "MacroMode.pyc", line 2685, in saveCakeData
none
File "DiffractionData.pyc", line 507, in saveCakeData
None
File "Cake.pyc", line 169, in toFile
None
<type "exceptions.IOError'>:[Errno 2] No such file or directory:
'SUBFOLDERH:/test/T2unrx_glass/SUBFOLDERNAME_int.dat'

Thoughts?

Here is the macro that I would like to use in that I want the macro to
add the files in the folders and then write the ouptut files in
folders for cake, 2th and qsp in the same directory as the folders
themselves. Not sure if that makes sense, so let me give you an
example:
5 files for summing are located in H:/test/tarbutite
So I want the 2theta spectra to get saved in H:/test/2th with the
filename of tarbutite_2th.dat

# Macro File recorded on Thu Feb 21 21:41:29 2008

Multiple Data Files:
H:/test/
Load From File
H:/test/12-15-07_calib.dat
Do Greater Than Mask?
Select
(Pixels Can't Be) Greater Than Mask:
170000
Do Polygon Mask?
Select
Load Mask
H:/test/bstop_msk.dat
AutoCake
Save Caked Image

SUBFOLDERPATHNAME/cake/SUBFOLDERNAME_cakeimage.jpg

Save Caked Data

SUBFOLDERPATHNAME/cake/SUBFOLDERNAME_cakedata.dat

Work in 2theta
AutoIntegrate 2theta-I
Save Integration Data
SUBFOLDERPATHNAME/SUBFOLDERNAME_2th.dat
Work in Q
AutoIntegrate Q-I
Save Integration Data
SUBFOLDERPATHNAME/SUBFOLDERNAME_qsp.dat

[joshu...@gmail.com]

I am sorry. I think that you misinterpreted what I said. I haven't yet
added any of these new macro commands. I was just suggesting that
maybe I should add them to the program and I was asking you to comment
on what you thought of them before I plunged ahead and coded them up.

So, what do you think? I will try to code them up tonight, but I am
not sure I will have the time. We'll see.

Josh

[Sarah....@gmail.com]

You're already super speedy. I think that would work great. I don't
think it's really excessively wordy. Sorry I misinterpreted.
Whenever you have time is cool.

Sarah

[Sarah....@gmail.com]

[joshu...@gmail.com]

What's up Sarah? I decided it was too wordy so I instead went with
macro commands named FOLDERPATH and FOLDERNAME. This to me is more
accurate because even if you are not using the Multiple Data Files
command but instead using the Data File command, you could still in
principle use the FOLDERNAME command (in which case the folder
shouldn't be thought of as a subfolder at all but is just the folder
containing some file).

For the record, FOLDERNAME gets replaced by the name of the folder
contaning the current diffraction file and PATHNAME gets replaced by
the path leading up to this folder.

Thus, if we open some mar3450 file, we could specify where the file is
by using the macro commands

PATHNAME/FILENAME.mar3450
or
FOLDERPATH/FOLDERNAME/FILENAME.mar3450

The new version containing this feature is v1.0.3

I have tested this feature but please report back and tell me if you
find any problems with it.

Josh

[Sarah....@gmail.com]

Hey-
Hope your trip to CA was fabulous. I am testing your changes now and
they look like they work great! This feature is such a lifesaver for
me! Thanks so much for working on this!!! I really appreciate it!
Sarah

[joshu...@gmail.com]

I am glad you like it. Keep me informed if you find any problems,

Josh