Import ASCII data

[stephane...@sintef.no]

Hello,

I have ASCII column-oriented files that have been generated from an EBSD software and that I would like to be readable to the StatsGenerator. These files contain, among others, the three Euler angles (phi1, PHI, phi2). I have built a filter with the following flow:

1. Create Data Container

2. Create Attribute Matrix (a Cell having the same number of tuples as there are lines to be imported)

3. Create Data Array (1 component - Phases)

4. Import ASCII Data

5. Combine Attribute Arrays (combine phi1, Phi and phi2 into EulerAngles)

6. Write StatsGenerator ODF Angle File

My first question concerns step 2 where I have to specify the size of the tuples. The ASCII files I will import may have different number of lines depending on the data: is there any way that the number of tuples in the Cell be defined dynamically? So far, I first set the data to import, then I know the size of my data and then I change the tuple size in the Create Attribute Matrix filter. Is there a better way?

For some files I also got an issue with the Create Attribute Matrix filter, i.e. when I execute the pipeline, I get the message "line 8 has an inconsistent number of data".

First I see that the help does not correspond to what the filter does: it says that the first screen should be about choosing whether the data is limited or with fixed width but that is not the case, I come directly to the choice of the delimiter. Anyway, I believe the number of columns is determined in that step when the delimiter is choosen. Since I use space as delimiter, and since some files do have some long comments before all the data, i.e. with many spaces, the filter ends up with many more columns that I actually have. Note also that all comment lines at the beginning of the files start with #. What should I do to avoid that (appart from deleting the comments from the file), is there something I am doing wrong?

Thanks for any help.

Regards

Stephane

[Sean Donegan]

Hi Stephane,

Currently, the only way to get the tuple dimensions correct for the ASCII file, assuming you don't know them a priori, is to run through the import ASCII wizard to get the tuple sizes then go back to the Create Attribute Matrix filter and set the tuple dimensions correctly there.  Something we would like to add to the filter is an option to create an attribute matrix for the data itself, rather than going through this roundabout process.  This would also make it much easier to use the filter when running from the command line; for now, the way you are getting the tuple sizes is the best way to go about it.

This second issue is a known bug.  The ASCII import wizard allows you to skip header lines, but then it will not properly ignore these lines when figuring the number of columns, which can cause issues as you noticed if the header lines have more delimiters than the actual number of data columns.  This is definitely something that we plan on fixing.  For now, the only real way around it is to manually modify the header lines to match the number of columns of the actual data, which is obviously not ideal. 

hope that helps,

-Sean

[Michael Jackson]

Stephane,
I have attempted to fix some of the issues that you have reported
below. Specifically I have fixed/updated the following:

* Documentation is updated with correct screen shots
* The number of columns are correctly deduced when "Start Import at
Row:" is used. The table will real-time adjust itself with new columns
with each adjustment to the "Start Import at Row" spin box.
* The filter will now properly read all of its parameters from a
pipeline file.
* The data structure tab will properly update itself when the "Remove
File" button is clicked.


To address your issue with files with long header lines do the following:

1: On the first windows of the Wizard, select "Space" for the delimiters
and optionally the "Treat consecutive delimiters as one".
2: Click "Continue"
3: On the next window adjust the "Start import at row:" spin box
according to your file. As you adjust this number the table will adjust
to the proper number of columns (hopefully).
4: Make any other adjustments necessary to import your data.
5: Click "Done"

We will work on adding an option to create the AttributeMatrix during
the filter which will make using this filter a bit easier.

These changes will be in the next nightly build which should be
completed by 03:00 EST which I think is about 09:00 local time for you.
Try out the new version and let us know if these changes allow DREAM.3D
to import your data correctly.


--
Michael A. Jackson
BlueQuartz Software, LLC
[e]: mike.j...@bluequartz.net

> --
> You received this message because you are subscribed to the Google
> Groups "dream3d-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dream3d-user...@googlegroups.com
> <mailto:dream3d-user...@googlegroups.com>.
> To post to this group, send email to dream3...@googlegroups.com
> <mailto:dream3...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dream3d-users/faa288de-44ea-4aa0-b3e4-1abd3abf232a%40googlegroups.com
> <https://groups.google.com/d/msgid/dream3d-users/faa288de-44ea-4aa0-b3e4-1abd3abf232a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[stephane...@sintef.no]

Hello!

It works like a charm; the headers are now ignored  :-)

Thanks very much for your help, it is very much appreciated.

Regards

Stephane