Redbox CSV import possible?

[Tim O'Connor]

Does Redbox allow importing (via the alert system) of CSV files?   if so, how?

[Andrew Brazzatti]

Hi Tim,

It does support CSV files but not in the greatest way, in particular the hard coded mapping.

Duncan has done a lot of work on improving the alerts code. This is currently in it's own branch and is nearly complete. Just needs a bit of additional work from me as he ran out of time.

Some of the features of the new alerts code:

• Allows alerts to come into different directories, each with their own configuration
• CSV files are now handled in a more comprehensive manner
• There's improved support for importing RIF-CS
• Improved logging
• A test script can be used to test a new alert without having it push into ReDBox

There are institutions waiting on this code so it's high on my list of priorities to complete.

Thanks,

Andrew

On Thu, Nov 1, 2012 at 4:56 PM, Tim O'Connor <tim.o...@rmit.edu.au> wrote:

Does Redbox allow importing (via the alert system) of CSV files?   if so, how?

-- Website: http://www.redboxresearchdata.com.au
 
You received this message because you are subscribed to the Google Groups ReDBox group. To post to this group, send email to redbo...@googlegroups.com. To unsubscribe from this group, send email to redbox-repo...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/redbox-repo?hl=en

[Tim O'Connor]

Hi Andrew - what's the status of this?  How close to being released?

Thanks :)

[Duncan Dickinson]

Hi Tim,

This is some work I'd been doing before I went on leave and need to get the final testing and documentation together (hopefully I'll have time next week). I had been working to improve the functionality of the alerts system after I'd prepared the How-To at http://www.redboxresearchdata.com.au/documentation/how-to/import-rif-cs-into-redbox*. 

Andrew rolled the code into the main branch for me after I headed off. I've set the code up to make it easier to get the code into an existing install. We'd look to release the new alerts stuff with the next release (not currently planned but definitely after Christmas break).

The main files can be found in:

• The newalerts.py HouseKeeping job can be found in https://github.com/redbox-mint/redbox/tree/master/config/src/main/config/portal/default/redbox/scripts/hkjobs
• The codebase (and doco) is available at https://github.com/redbox-mint/redbox/tree/master/config/src/main/config/home/lib/jython/alertlib

You should be able to copy these resources over to an existing install and get started now - if you have any issues let me know and I'll look into them when I'm finalising the testing/doco

* Note that the new code means you shouldn't need the alerts.py script available from that wiki page.

--

Cheers, 

Duncan 

Duncan Dickinson 
QCIF Project Manager 
Central Queensland University 

My contact details: ph: 07 3138 2084 m: 0432 402 511 skype: de.dickinson website | calendar | LinkedIn

Project involvement ReDBox (Research Data Box) TERN Central Portal Tropical Data Hub

Developer? Check out dev8D-AU at  http://dev8dau.info/

---

[Tim O'Connor]

Excellent, thanks Duncan - Ill have a play with this today.

[Tim O'Connor]

Hi again - Im unsure how I integrate these files - Im using the instructions here for an institutional build:

http://www.redboxresearchdata.com.au/documentation/how-to/institutional-builds

After checkout, Im not seeing a directory I can copy these files into...there is no src / main / config / home / lib / jython / alertlib directory that I am seeing for example. (and I suspect Im sounding pretty foolish here, but its the only way to learn :))

On Thursday, November 29, 2012 9:16:05 AM UTC+11, Duncan Dickinson wrote:

[Andrew Brazzatti]

Hi Tim,

Those files are in the core part of Redbox. Change the version numbers of the Redbox dependencies in your institutional build to the latest 1.5.2.3-SNAPSHOT version (it wasn't in the last release) and you should see these files pop up in your built app.

Thanks,

Andrew

On Fri, Nov 30, 2012 at 3:15 PM, Tim O'Connor <tim.o...@rmit.edu.au> wrote:

home 

[Tim O'Connor]

Cool - got the new alert system running.  

Ive put some example rif-cs file in there, and it appears to work, but it doesnt appear anywhere in redbox.  Any ideas?

Logs for the file say:

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Processing file new-rif.rif

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Using the XMLAlertHandler for file with extension rif

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Using metadata file: C:\redbox\home/new-alerts\.processed\2012_12_03_17_55_01\processing/new-rif.rif.1.tfpackage

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Submitting to harvest. Config file is C:\redbox\home/harvest/workflows/dataset.json and meta_file is C:\redbox\home/new-alerts\.processed\2012_12_03_17_55_01\processing/new-rif.rif.1.tfpackage

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Successfully harvested alert item f2f4706db02f7bd63ecad78fc5869d82 from processing file C:\redbox\home/new-alerts\.processed\2012_12_03_17_55_01\processing/new-rif.rif.1.tfpackage

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] Moving successful metadata file [C:\redbox\home/new-alerts\.processed\2012_12_03_17_55_01\processing/new-rif.rif.1.tfpackage] to C:\redbox\home/new-alerts\.processed\2012_12_03_17_55_01\success/.

[INFO] [Default alert] [new-rif.rif] [Mon Dec  3 17:55:01 2012] File processing complete. Successful imports: 1; Failed imports 0

[Andrew Brazzatti]

Hi Tim,

Found the issue. Was racking my brain as there seemed to be nothing wrong with the code but it turns out the issue is in the sample configuration that is on the GitHub page. Our rules file for dataset records assumes that everything stored in it is a string. The baseline data that was configured was as follows:

  "baseline": {
            "viewId": "default",
            "packageType": "dataset",
            "redbox:formVersion": "1.5.2",
            "redbox:newForm": true,
            "redbox:submissionProcess.redbox:submitted": true
        }

Meaning we are setting some boolean values. Simply making them strings as I have done below will fix your problem:

  "baseline": {
            "viewId": "default",
            "packageType": "dataset",
            "redbox:formVersion": "1.5.2",
            "redbox:newForm": "true",
            "redbox:submissionProcess.redbox:submitted": "true"
        }

I'll be updating that sample config shortly.

Thanks,

Andrew