CsvImport Plugin

[Will Riley]

I know that some of you have been waiting to play with the CsvImport
plugin. It's now available in alpha for you to use and test.

The CsvImport Plugin allows you to import items from a comma separated
file (CSV). It allows you to map each of the columns in your file to
item elements, tags, or files.

Using it is pretty simple.

1. Download and install the plugin.

cd plugins
svn co https://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha
CsvImport

(During installation, you need to specify the path to your php 5
executable. It will usually work by default, but some webhosts like
Dreamhost have a different path).

2. Create a CSV file where each row represents data for a single
item. The CSV file should only include information about items of the
same item type. Also, the first row should have column headers. For
multi-line data, or data with commas, you should surround the
information with double quotes.

Here's a properly formatted example file called foobar.csv



name, age, gender, quote, tags, url
"John", "27", "Male", "Gee, I really like Will's plugin.", "smart,
human, Georgia" , "http://www.myysspace.com/pic.gif"
"Laura", "50", "Female", "Wow, I can't believe it's not butter!",
"smart, human, Alaska", "http://www.myysspace.com/pic.gif"



One easy way of creating a CSV file is by saving an Excel Spreadsheet
as a CSV file.

3. Move the CSV files you want to import into the plugins/CsvImport/
csv_files directory on the server. You can use an SFTP client to do
this. This method allows you to ingest large files.

4. Login as an administrator, and click the CsvImport tab, and then
click the Import Items tab.

5. Select your Csv file. Optionally, select an item type and the
collection you want to import it to. Specify whether the items are
all going to be imported as public and/or featured. And if you are
importing files for items from URL data, then specify whether you want
the import process to stop if it cannot download a file. If this is
unchecked, then it will ignore download errors and keep importing
items. Click the Next button.

6. Specify how you want to map your data. Each column in your CSV
file represents a different aspect of your items, including metadata
stored in elements, tags, and files. Let us consider each one in
turn:

Elements:
You can map columns to multiple elements from multiple element sets.
In our example file, the "name" column could map to the "Title"
element of the Dublin Core element set, as well as the "Creator"
element of the Dublin Core element set. To do this, select each
element from the dropdown and then click the add button. To help you
decide, the plugin shows you an example of the data from that column.
If you change your mind, you can remove an added element by clicking
the delete graphic next to each added element.

Also you can map two columns to the same element. For example, you
could map both the "age" column and the "gender" column to the
Description element in the element set associated with the Document
item type. To gain access to this element, you would have had to
select the Document item type for this import.

Tags
In addition to elements, you can also map elements to tags. The
importer will parse comma separated tags and associate them with the
item. To do this, make sure your column data has comma separated
tags, like the example above.

Files
The plugin can also download files based on column data. To do this,
enter a single URL to a file and click the checkbox.

After you have configured your column mappings, click the Import CSV
File button.

7. Monitor your import status. Click the Status tab or reload the
Status page to check the progress of your import. After your import
has completed, you have the option of undoing your import. If you
undo your import, then all the items in that import will be deleted.
Don't worry, the system prompts you for confirmation twice to make
sure that you really want to undo the import. Finally, if you have
many undone imports, you may want to remove the history of those
undone imports, so you don't have to keep looking at them. To do this,
click the Clear History link next your import.

8. Remember this: if you decide that you want to uninstall the
plugin, it will NOT delete the items you have imported. It will
however, prevent you from undoing those imports in the future.

Have fun with the plugin and let us know if it works as specified and
how we can improve it.

[Will Riley]

I forgot to mention that the CsvImport plugin requires Omeka 1.0.

On Mar 12, 5:54 pm, Will Riley <wanderingw...@gmail.com> wrote:
> I know that some of you have been waiting to play with the CsvImport
> plugin.  It's now available in alpha for you to use and test.
>
> The CsvImport Plugin allows you to import items from a comma separated
> file (CSV).  It allows you to map each of the columns in your file to
> item elements, tags, or files.
>
> Using it is pretty simple.
>
> 1. Download and install the plugin.
>
> cd plugins

> svn cohttps://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha

[glass]

Ah, I think I accidentally sent you an email, Will!
Anyway, I keep getting 404 errors when I click the plugin tab. I have
double checked that all the files are in the correct directories and
the CLI points to the right place. Suggestions?

Thanks!

Dan Glass

[Will Riley]

Hey Dan,

I'm not sure why it's doing that and cannot reproduce the problem.

Make sure that you have Omeka 1.0alpha installed. It tells you the
version you are using at the bottom of the main admin page.

Then, uninstall and CsvImport plugin you currently have installed, and
then remove its directory from the plugins directory.

I added a zipped version of the alpha version of the CsvImport plugin
to the Files section, so you can either download it and unzip it into
the plugin directory, or checkout the alpha version from svn as
described above. Then try to install it. Try this and let me know if
you're still getting the same 404 errors.

Thanks,
Will

[glass]

After I uninstalled and reinstalled it worked great: I just imported
about 2500 records from PassPerfect into Omeka. The only question mark
above the importer is that it's important to not that the first
element determines what gets imported; if there's a blank spot in the
first field of the data set the importer will ignore that row in the
CSV.

In my imported dataset the first column was 'AssnNum' which some
records lacked, for whatever reason. It then imported only the records
who had a value for that field, even if they had every other field
complete. It was easy enough to get around with an insertion of a
column in Excel and a copy/paste with the museum ID number, but some
people who don't know how to manipulate spreadsheets might find this
intermediate step confusing.

Everything went perfectly beyond that--great work. Omeka wouldn't be
able to really expand without a feasible data migration option, and
with 1.0 on the way and this plugin very capable of digesting large
data sets, I'm very excited to see where this will go.

Thanks,
Glass

[Tom Scheinfeldt]

Thanks, Glass. Glad the plugin worked for you. I'm sure your
suggestions will be helpful to other users. We will be sure to include
them as we work on the plugin's documentation. You're also absolutely
right about the importance of migration options, which is why we have
coordinated the release of this CSVImport plugin (and the OAI-PHM
harvester) with the release of Omeka 1.0.

Take care,
Tom Scheinfeldt

[Will Riley]

Thanks Dan,

I'm glad it worked. Can you email me the csv file that did not work
for you? I will look at it and try to fix the issue.

Thanks,
Will

[glass]

Will,

I will forward one to you on Monday, but generally speaking any CSV
file I tried to import would only be imported if it had a nonempty
field in the first column.

Dan

[Will Riley]

Thanks Dan for finding the bug. I fixed it and updated the plugin
code in svn on the trunk. Also, I uploaded the revised plugin to
Files section.

Let me know it works for you now.

Cheers,
Will

[amandafrench]

Hey, Will!

I'm new to the list, and it'd be a stretch to call me a developer,
though I've worked a little with PHP and subversion etc., so please
pardon me if I'm asking silly questions. I have the same problem as
Dan: after I checked out CsvImport with the svn command above & set
the PHP-CLI location to what I *think* is the right one, I'm getting
404 errors when I click on the tab in the admin panel.

(By the way, I'm on Dreamhost, and they say that the PHP5 executable
path is /usr/local/php5/bin/php.)

I'm not sure where the "Files section" with the zipped version that
you mention above is -- can you clarify?

Thanks,

Amanda

[John Flatness]

Hi,

The "Files" section that was mentioned is on the Google Groups page
for the mailing list, at http://groups.google.com/group/omeka-dev

Specifically, all the files are at http://groups.google.com/group/omeka-dev/files
. You're looking for CsvImport-1.0-1.0alpha.zip

If you tried to use the svn command above, it's important to note that
the end of the command got wrapped to a second line (at least for me).

-John

[amandafrench]

Thanks, John -- You're absolutely right that the linewrapping was the
problem; I used svn co https://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha
and it worked. It created a directory called 1.0-1.0alpha, which I did
think was weird, but I figured, Who am I to judge others' naming
practices? :)

I've corrected the command and I'm to "svn co
https://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha CsvImport"
and I'm sure it'll work great now.

Amanda

On Mar 26, 1:44 pm, John Flatness <jflat...@vt.edu> wrote:
> Hi,
>
> The "Files" section that was mentioned is on the Google Groups page

> for the mailing list, athttp://groups.google.com/group/omeka-dev
>
> Specifically, all the files are athttp://groups.google.com/group/omeka-dev/files

[Erin Bell]

Hi,

I've been working with the CSV Import plug-in for the last day or so,
but have not had any luck.

I was given a ContentDM export file (tab-delimited), which I converted
to CSV. There are no empty fields and I mapped each column to an
element in Omeka. The final column contains the URL to the file in
ContentDM. For that column, I checked the URL to File box. I tried
this while mapping to an element and not mapping to an element.

Each time, the files are successfully migrated to Omeka, but the
records are not.

In my most recent attempt, I noticed that if you try to import without
choosing file or tag, you get an admin message saying you must choose
at least one "element, file, or tag". So, guessing it wasn't seeing
that I had actually chosen many elements, I went back through and
clicked the "Add Element" button next to each one. I think that the
"Add" button is only for adding additional elements, but here I could
be wrong. In any case, the result of this approach is that the import
is frozen at 0.

One last note that may be important. Reading through Will's initial
announcement, I see that "you can map two columns to the same
element." I have a few instances where I was mapping more than two
columns to the same element. Could this be the root (or part) of my
problem?

The CSVs (I tried three different versions) can be viewed here...
http://137.148.49.110/public/sandbox/items/show/505

Here are the first two rows...
Title,Subject,Subject,Description,Creator,Date of
Original,Collection,Source,Submitting Institution,Rights,Type,Image
Height,Image Width,File Size,Language,Display File Type,Color
Space,Media Type,Reference URL
Flank Markers of the 1st O.V.V.H.A.,Flags -- Ohio," United States --
History -- Civil War, 1861-1865",Flank Marker and Guidon of 1st Ohio
Veteran Volunteer Heavy Artillery. Text on flank marker reads: Lady
Flint.,Ohio Department of the Adjutant General,ca. 1965-1970,Ohio
Battle Flag Audiovisual Collection,"Photographic Print (Color), Not
cataloged, State Archives Series 4605 AV, Box 1, Folder 2, Image AF
9",Ohio Historical Society,"Online access is provided for research
purposes only. For rights and reproduction requests or more
information, go to http://www.ohiohistory.org/images/information",StillImage,2000,2771,1161.635
KB,en,image/jp2,RGB,JP2,http://cdm267401.cdmhost.com/cgi-bin/
showfile.exe?CISOROOT=/flags&CISOPTR=100

In any case, this is clearly a really great plug-in; once I figure out
what I'm doing wrong, I will be extremely grateful that it exists.

Cheers,

Erin

On Mar 12, 5:54 pm, Will Riley <wanderingw...@gmail.com> wrote:

> I know that some of you have been waiting to play with the CsvImport
> plugin.  It's now available in alpha for you to use and test.
>
> The CsvImport Plugin allows you to import items from a comma separated
> file (CSV).  It allows you to map each of the columns in your file to
> item elements, tags, or files.
>
> Using it is pretty simple.
>
> 1. Download and install the plugin.
>
> cd plugins

> svn cohttps://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha

[Will Riley]

Hey Erin,

Thank you for your feedback. I plan to change the user interface for
adding elements because it is confusing. Currently, you need to click
the Add Element button to add elements - if you do not click this
button then it will not add it. We will change this so that selecting
from the dropdown will automatically add the element.

I tested the first file, export_flags1.csv and was able to get it to
import some of the metadata. In was able to map the two "Subject"
columns to the same element. But the system failed when i tried to
map lots of text fields. I also was able to download some of the
files. The files are .JP2. We need to add this mime type so they are
viewable I will debug this and let you know.

-Will

On Mar 27, 9:53 am, Erin Bell <ebellemp...@gmail.com> wrote:
> Hi,
>
> I've been working with the CSV Import plug-in for the last day or so,
> but have not had any luck.
>
> I was given a ContentDM export file (tab-delimited), which I converted
> to CSV.  There are no empty fields and I mapped each column to an
> element in Omeka.  The final column contains the URL to the file in
> ContentDM.  For that column, I checked the URL to File box.  I tried
> this while mapping to an element and not mapping to an element.
>
> Each time, the files are successfully migrated to Omeka, but the
> records are not.
>
> In my most recent attempt, I noticed that if you try to import without
> choosing file or tag, you get an admin message saying you must choose
> at least one "element, file, or tag".  So, guessing it wasn't seeing
> that I had actually chosen many elements, I went back through and
> clicked the "Add Element" button next to each one.  I think that the
> "Add" button is only for adding additional elements, but here I could
> be wrong.  In any case, the result of this approach is that the import
> is frozen at 0.
>
> One last note that may be important.  Reading through Will's initial
> announcement, I see that "you can map two columns to the same
> element."  I have a few instances where I was mapping more than two
> columns to the same element.  Could this be the root (or part) of my
> problem?
>

> The CSVs (I tried three different versions) can be viewed here...http://137.148.49.110/public/sandbox/items/show/505

>
> Here are the first two rows...
> Title,Subject,Subject,Description,Creator,Date of
> Original,Collection,Source,Submitting Institution,Rights,Type,Image
> Height,Image Width,File Size,Language,Display File Type,Color
> Space,Media Type,Reference URL
> Flank Markers of the 1st O.V.V.H.A.,Flags -- Ohio," United States --
> History -- Civil War, 1861-1865",Flank Marker and Guidon of 1st Ohio
> Veteran Volunteer Heavy Artillery.  Text on flank marker reads: Lady
> Flint.,Ohio Department of the Adjutant General,ca. 1965-1970,Ohio
> Battle Flag Audiovisual Collection,"Photographic Print (Color), Not
> cataloged, State Archives Series 4605 AV, Box 1, Folder 2, Image AF
> 9",Ohio Historical Society,"Online access is provided for research
> purposes only. For rights and reproduction requests or more

> information, go tohttp://www.ohiohistory.org/images/information",StillImage,2000,2771,1161.635

[Tom Scheinfeldt]

Erin,

The other thing to note here is that the OAI-PMH harvester plugin should also work with CONTENTdm installations. I don't know if you'd like to try that in addition to CSVImport.

Thanks for your feedback.

Tom

On Mar 27, 2009 7:43 PM, "Will Riley" <wander...@gmail.com> wrote:


Hey Erin,

Thank you for your feedback.  I plan to change the user interface for
adding elements because it is confusing.  Currently, you need to click
the Add Element button to add elements - if you do not click this
button then it will not add it.  We will change this so that selecting
from the dropdown will automatically add the element.

I tested the first file, export_flags1.csv and was able to get it to
import some of the metadata.  In was able to map the two "Subject"
columns to the same element.  But the system failed when i tried to
map lots of text fields.  I also was able to download some of the
files.  The files are .JP2.  We need to add this mime type so they are
viewable  I will debug this and let you know.

-Will

On Mar 27, 9:53 am, Erin Bell <ebellemp...@gmail.com> wrote: > Hi, > > I've been working with the ...

> information, go tohttp://www.ohiohistory.org/images/information",StillImage,2000,2771,1161.635

> KB,en,image/jp2,RGB,JP2,http://cdm267401.cdmhost.com/cgi-bin/ > showfile.exe?CISOROOT=/flags&CISOP...

[Dave Lester]

Hi Will (as well as Erin, Amanda, and everyone else),

The getId3 library we use to extract file metadata doesn't support
jpeg2000 photos, which, I *think*, is why this is dying. If that's
the case, we'll need to automatically bypass those files, and possibly
implement some other type of metadata extraction (I'm not sure, but I
imagine someone has come up with some kind of solution).

Dave

[Erin Bell]

Thanks everyone for the replies.

My import is still frozen at zero, so my first follow-up question is
how to stop that process.

There are a couple reasons I am using the CSV Importer, rather than
the OAI-PMH Harvester. The first is that we only want select records/
items from across several cDM sets -- thus, we are exporting the data
from custom queries. This also gives us the benefit of turning the
reference URLs into download URLs with a quick find-and-replace across
the spreadsheet.

Of course, the latter isn't of much use if the JP2 files won't
display. I am a bit confused on this one though, since the files I
successfully migrated (sans metadata) were displaying fine on the home
and item-browse pages (though not in the item-show page).

Is there a safe and simple hack to fix this, or will I need to convert
to a different format?

If so, does anyone have any advice on writing a script that can
download from all the URLs on my CSV at once?

Thanks for all your help. I'll update again if I make any progress
today.

Erin

[Erin Bell]

So, I'm still frozen up at the end of the day. How can I kill that
process?

Also, I downloaded and converted all of the JPEG2000 files, so I will
probably end up using the Dropbox plug-in for the files, combined with
the CSV Import plug-in for the metadata. Not as ideal as doing it all
in one fell swoop, but not bad all things considered.

I'll let you know how the metadata-only process goes.

Erin

> ...
>
> read more »

[Erin Bell]

I scaled back the number of metadata fields to import (eliminating
"Omeka image metadata" fields and a few others that would increase the
number of columns mapped to one field) and everything worked great --
with and without images (ignoring for the moment the JP2 problem).

I have (hopefully) one last question about the CSV Importer:

Is there a way to import files and give them a different display
name? Right now, when I import from a URL, I get a file name like:

http://cdm267401.cdmhost.com/cgi-bin/showfile.exe?CISOROOT=/flags&CISOPTR=170

I imagine this could be confusing to a user. I'm not sure if this is
an issue to address in this plug-in or in Omeka itself (or to solve
with another plug-in that renames files), but I thought I would bring
that up to see what others think.

Thanks,

Erin

> ...
>
> read more »

[amandafrench]

Just wanted to say that the CSVImport is working great for us! We
imported about 300 items in a matter of minutes, files, metadata, and
all.

We took two tries, though, because the first time around we mapped the
URL columns to the Identifier element AND checked the "URL to file?"
checkbox; when we did that, the image files didn't import. When we did
it again and just checked the checkbox without mapping the URL
columns, it worked perfectly.

Amabda

On Mar 12, 5:54 pm, Will Riley <wanderingw...@gmail.com> wrote:

> I know that some of you have been waiting to play with the CsvImport
> plugin.  It's now available in alpha for you to use and test.
>
> The CsvImport Plugin allows you to import items from a comma separated
> file (CSV).  It allows you to map each of the columns in your file to
> item elements, tags, or files.
>
> Using it is pretty simple.
>
> 1. Download and install the plugin.
>
> cd plugins

> svn cohttps://omeka.org/svn/plugins/CsvImport/tags/1.0-1.0alpha

[Will Riley]

Hey Amanda and Erin,

I just updated the Omeka 1.0 trunk, the Omeka stable-1.0 branch, and
the CsvImport plugin with some bug fixes. Now you should be able to
ingest files from URLs.

Let me know if these fixes work for you.

-Will

[Erin Bell]

Hi Will,

I moved away from our sandbox and am working with what will soon be a
production site, so a few things have changed that may be affecting my
experience. I am running Omeka 1.0 beta, with the latest trunk of the
CSVimport plug-in. I tried a couple CSV files that worked fine on our
sandbox instance, but are throwing errors here. Same server. Same
settings. I'm getting the standard: "An error has occurred in Omeka.
To track the contents of this error, please enable error logging."
The error log shows: "2009-05-14T16:12:41-04:00 ERR (3): Mysqli
prepare error: Unknown column 'error_details' in 'field list'"

Any guesses?

By the way, two other things: 1) To Will, the new UI is much
improved. I like the auto-add from the drop-down. 2) To All, am I
correct in thinking that JP2000 is now supported in Omeka 1.0?

[Will Riley]

Hey Erin,

To remove the unknown column error, I think you need to reinstal from
the CsvImport plugin from the plugin page.
Try uninstalling the CsvImport plugin from the Settings->Plugins admin
page. And then installing it again from that page.

Also, we have improved the file downloading process in the stable-1.0
branch since the 1.0beta release. We have not released the 1.0 stable
yet, but you are free to switch your repository from 1.0beta to the
stable-1.0 branch.

To do this, you run something like this from your project root:

svn switch https://omeka.org/svn/branches/stable-1.0

Even if you switch to the stable-1.0 branch, your files in
export_flags-final-withFiles.csv will not ingest properly because the
requests to these links provide an invalid mime-type: application/
octet-stream

For example, the mime-type for the following file is application/octet-
stream (not image/jp2):
http://cdm267401.cdmhost.com/cgi-bin/showfile.exe?CISOROOT=/flags&CISOPTR=100

As a work around, perhaps you can place these images somewhere else so
that they are served with the image/jp2 mime-type.

At the moment, Omeka allows you upload and associate a JPEG-2000 file
with an item, but it does not generate any thumbnails or display
them. As far as I know, few web browsers support JPEG-2000. Can you
convert the images to another file format?

Let me know if this helps.

Best regards,
Will

[Erin Bell]

Thanks Will,

I'm kind of surprised and embarrassed that I didn't try reinstalling.
As a note, I had to "uninstall" without "deactivating" first --
otherwise, the de-installation wouldn't take. But reinstalling fixed
the problem, and just in time, since I have a meeting in the morning
and needed to show progress. I owe you a beer (or a candybar if
you're not into beer) at THATcamp.

As for the JP2 formatting: yeah, I think I'll need to put the JPEG
versions up on our server and point to that. I forgot that the file
paths in that CSV were configured for downloading from Cdm and do not
work for the migration (or downloading on a Mac as I just figured
out... strange).

Best -- Erin

On May 18, 4:59 pm, Will Riley <wanderingw...@gmail.com> wrote:
> Hey Erin,
>
> To remove the unknown column error, I think you need to reinstal from
> the CsvImport plugin from the plugin page.
> Try uninstalling the CsvImport plugin from the Settings->Plugins admin
> page.  And then installing it again from that page.
>
> Also, we have improved the file downloading process in the stable-1.0
> branch since the 1.0beta release.  We have not released the 1.0 stable
> yet, but you are free to switch your repository from 1.0beta to the
> stable-1.0 branch.
>
> To do this, you run something like this from your project root:
>

> svn switchhttps://omeka.org/svn/branches/stable-1.0

>
> Even if you switch to the stable-1.0 branch, your files in
> export_flags-final-withFiles.csv will not ingest properly because the
> requests to these links provide an invalid mime-type: application/
> octet-stream
>
> For example, the mime-type for the following file is application/octet-

> stream (not image/jp2):http://cdm267401.cdmhost.com/cgi-bin/showfile.exe?CISOROOT=/flags&CIS...

[Dave Lester]

Support for JPEG2000 was added in 1.0beta, and Will is right that
there's no default way of displaying files, but now Omeka doesn't die
when you try to upload the files and they can be included in your
archive. :) It's possible to display JPEG2000 files by wrapping them
in an appropriate flash-wrapper (or whatever other tool is used to
display jpeg2000 files via a web interface).

Dave

[David Dwiggins]

I've been playing with this a bit - overall looks very good. But I noticed two errors:

When trying to import a large file with lots of fields, I got the following in my server error log:

[Tue May 26 13:25:58 2009] [error] [client 68.236.104.199] ALERT - configured POST variable limit exceeded - dropped variable 'csv_import_column_map_file_48' (attacker '68.236.104.199', file '/srv/www/htdocs/omekatest/admin/index.php'), referer: http://cap.hneotis.historicnewengland.org/omekatest/admin/csv-import/index/map-columns
[Tue May 26 13:26:27 2009] [error] [client 68.236.104.199] ALERT - configured POST variable limit exceeded - dropped variable 'csv_import_column_map_file_48' (attacker '68.236.104.199', file '/srv/www/htdocs/omekatest/admin/index.php'), referer: http://cap.hneotis.historicnewengland.org/omekatest/admin
/csv-import/index/map-columns

It then appears that nothing imported.

I know that this is a server configuration thing, but it would be nice if the plugin realized that there would be too many fields to handle before the user spends time trying to map them.

Also, after I reduced the number of columns on the incoming file, it appeared to work but the status screen looked like this:

2009-05-26 14:31:06     jewelry.csv      getImportedItemCount(); ?> / 3232      status; ?>

This is because my default PHP install (on SLES 10.1)  doesn't recognize <? as  a valid start for php.  So the following fixes it

/hne/omeka/stable-1.0/plugins/CsvImport/views/admin/index/status.php

--- status.php  (revision 4052)
+++ status.php  (working copy)
@@ -37,8 +37,8 @@
                 <tr>
                     <td><?php echo $csvImport->added; ?></td>
                     <td><?php echo $csvImport->csv_file_name; ?></td>
-                    <td><? echo $csvImport->getImportedItemCount(); ?> / <?php echo $csvImport->getItemCount(); ?></td>
-                    <td><? echo $csvImport->status; ?></td>
+                    <td><?php echo $csvImport->getImportedItemCount(); ?> / <?php echo $csvImport->getItemCount(); ?></td>
+                    <td><?php echo $csvImport->status; ?></td>
                     <?php
                     if ( $csvImport->status == CsvImport_Import::STATUS_COMPLETED_IMPORT ||
                          $csvImport->status == CsvImport_Import::STATUS_IMPORT_ERROR_INVALID_FILE_DOWNLOAD) {

Looking at the bigger picture: I haven't dug into this too deeply yet, but I'm wondering if there is a provision for loading updated CSV files with new/improved data -- IE, once you've done it once, you then get a new file that has changes to some of the existing records (based on identifier) along with some new records.  This would likely be a requirement for our potential use case, where cataloging data is actually being mantained in another system and mirrored to Omeka.

-David Dwiggins
Historic New England

[Will Riley]

Thank you David for finding and publishing these issues with the CSV
Import plugin.

I will apply your fix for the <? issue immediately and look into the
Suhosin issue to see how to address it. Also, I like your idea about
being able to reimport an item based on a uniquely keyed column and
having those records update. Perhaps we could work on be a re-import
feature that allows user to update the rows that match this unique key
and adds new rows for those that don't. Perhaps it could also have an
option to delete items that were in the previous import, but not in
the current import. What do you think?

-Will

On May 26, 4:14 pm, David Dwiggins <da...@dwiggins.net> wrote:
> I've been playing with this a bit - overall looks very good. But I noticed
> two errors:
>
> When trying to import a large file with lots of fields, I got the following
> in my server error log:
>
> [Tue May 26 13:25:58 2009] [error] [client 68.236.104.199] ALERT -
> configured POST variable limit exceeded - dropped variable
> 'csv_import_column_map_file_48' (attacker '68.236.104.199', file

> '/srv/www/htdocs/omekatest/admin/index.php'), referer:http://cap.hneotis.historicnewengland.org/omekatest/admin/csv-import/...

> [Tue May 26 13:26:27 2009] [error] [client 68.236.104.199] ALERT -
> configured POST variable limit exceeded - dropped variable
> 'csv_import_column_map_file_48' (attacker '68.236.104.199', file

> '/srv/www/htdocs/omekatest/admin/index.php'), referer:http://cap.hneotis.historicnewengland.org/omekatest/admin/csv-import/...

[Will Riley]

I just updated the trunk of the CsvImport plugin so that it uses <?php
instead of <?

-Will

On May 26, 4:14 pm, David Dwiggins <da...@dwiggins.net> wrote:

> I've been playing with this a bit - overall looks very good. But I noticed
> two errors:
>
> When trying to import a large file with lots of fields, I got the following
> in my server error log:
>
> [Tue May 26 13:25:58 2009] [error] [client 68.236.104.199] ALERT -
> configured POST variable limit exceeded - dropped variable
> 'csv_import_column_map_file_48' (attacker '68.236.104.199', file

> '/srv/www/htdocs/omekatest/admin/index.php'), referer:http://cap.hneotis.historicnewengland.org/omekatest/admin/csv-import/...

> [Tue May 26 13:26:27 2009] [error] [client 68.236.104.199] ALERT -
> configured POST variable limit exceeded - dropped variable
> 'csv_import_column_map_file_48' (attacker '68.236.104.199', file

> '/srv/www/htdocs/omekatest/admin/index.php'), referer:http://cap.hneotis.historicnewengland.org/omekatest/admin/csv-import/...

[David Dwiggins]

Thanks for the quick response. Yeah, that's kind of what I was envisioning. However, the deletion idea assumes that all of the data is coming from one and only one source. If a record doesn't exist in the incoming file because it was generated within Omeka, or by importing a different CSV file, then it probably shouldn't be purged.

I also think this implies the need to remember one or more data maps so that they can be reused. Ideally, once you've set up the map initially, the system would remember it, and then it could be reapplied for any subsequent updates. And perhaps the system would remember which map put a record in so that it would also realize if it should be taken out again if it disappears from that data source?

I like the GUI for doing the mapping, but for these subsequent imports I'm thinking along the lines of a cron job that would kick off every night and ingest updates from that day.

I'm sure this makes the whole thing far more complicated than what you may want to tackle, but I figured as long as we're brainstorming, it doesn't hurt to think it out.

(I don't know how much this jives with the needs of others. We're still looking at various options for the online front end to several collections databases, and Omeka is one of the things on the table.)

-David Dwiggins