What is suggested way for creating datasets from csv file and partial synchronization?

[Ladislav Nesnera]

We have two use cases:

• dozens of dataset decribed in spreadsheet (xls, csv). Creting them individually by hand is litle bit irritating
• one CKAN instance contains all our datasets, second one only subset of open data (in primary catalog tagged as "open data")

What is suggested way for creating datasets from csv file and publishing open data into the second server?

• loader script like this one? But seems be deprecated.
• use CKAN API (e.g. https://stackoverflow.com/questions/24642317/create-ckan-dataset-using-ckan-api-and-python-requests-library)
• an extension (ckanext-harvest)
  
• or somethin else?

Thank you in advance for your recommendation.   ;?

[Florian May]

Hi Ladislav,

My gut feeling is that the most flexible yet maintainable way for you might be a script in your favourite language rather than using / customising existing harvesting infrastructure (like ckanext-harvest) to do the same.

I found my sweet spot with ckanapi and iPython notebooks - see what you can re-use: https://github.com/datawagovau/harvesters 

Alternatively, R, ckanr and RMarkdown workbooks are able to do the same, but Python's list comprehensions are just a joy to work with.

Write-up of the approaches: http://datawagovau.readthedocs.io/en/latest/operations.html#ckan-to-ckan-harvesting-ckanext-harvest

ckanext-harvest is a very elegant way to automate / schedule very simple jobs (for which harvest has a configuration), but customising ckanext-harvest your specific use case (with proper handling of all edge cases) will require the above work plus modifying the actual extension (adding your own custom harvester). My use case was too one-off (only ran a few times in test/UAT, then in prod) to warrant that kind of sophistication.

Hope that helps!

Florian

--
You received this message because you are subscribed to the Google Groups "CKAN Global User Group (Non-technical questions)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ckan-global-user-group+unsub...@googlegroups.com.
To post to this group, send email to ckan-global-user-group@googlegroups.com.
Visit this group at https://groups.google.com/group/ckan-global-user-group.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/2b45fe69-8831-40d0-9072-d0ccdb01c448%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven De Costa]

You could take a look at https://github.com/DataShades/ckanext-excelimport

We built that to help on a use case where custodians wanted to upload and import a dataset with its resources without having to enter all the metadata via the UI. It doesn't handle bulk creation of datasets from a single excel file but take a look and see if it might help in your case :)

Cheers,

Steven

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAK6Sqm3erD2mn%2B0Wf8HePtd0zTnWHo1ebPuxGqcRzxaF_QJ_kw%40mail.gmail.com.

[Steven De Costa]

Sorry, on the second use case we'd use syndicate and push from the first CKAN into the second :)

You'll find the forked repo we use here: https://github.com/DataShades/ckanext-syndicate

Cheers,

Steven

STEVEN DE COSTA | EXECUTIVE DIRECTOR
www.linkdigital.com.au

   

To unsubscribe from this group and stop receiving emails from it, send an email to ckan-global-user-group+unsubscri...@googlegroups.com.

To post to this group, send email to ckan-global-user-group@googlegroups.com.
Visit this group at https://groups.google.com/group/ckan-global-user-group.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/2b45fe69-8831-40d0-9072-d0ccdb01c448%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "CKAN Global User Group (Non-technical questions)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ckan-global-user-group+unsubscri...@googlegroups.com.

[Ladislav Nesnera]

Guys, thanks a lot for very inspirational posts