HARVEST dataset from DKAN to CKAN

[Lucas .Moran]

To whom it may be concern, 

We are contacting you in order to learn how to harvest datatests from DKAN to CKAN.  We are part of a National Health Care Institution for Senior Citizents in Argentina, INSSJP (www.pami.org.ar). 

This is the material that we have used so far: 

• https://github.com/ckan/ckanext-harvest : It explains how to migrate from CKAN to CKAN but it does not mention how to migrate from DKAN to CKAN. 

• https://github.com/HHS/ckanext-datajson : We have tried several times but it did not work. 

Our DKAN website is: it.pami.org.ar 

It is essencial to clarify that we implement CKAN into dockers. 
  

Best regards. 

[David Read]

The DKAN implementation of the CKAN API was awful when we looked at it
in October. That's why the CKAN harvest doesn't work. We did get a
harvester working for DKAN, which you can see here:
https://github.com/datagovuk/ckanext-harvest/blob/2.0/ckanext/harvest/harvesters/dkanharvester.py
which inherits from our customized ckanharvester and ckanext-harvest,
and there are several bits specific to data.gov.uk. But it gives you
the idea.

David

> --
> You received this message because you are subscribed to the Google Groups
> "CKAN Global User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ckan-global-user-...@googlegroups.com.
> To post to this group, send email to
> ckan-global...@googlegroups.com.
> Visit this group at http://groups.google.com/group/ckan-global-user-group.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/ckan-global-user-group/9c3bec89-b518-4990-9e3f-b0168df07ec0%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Aaron Couch]

hi David:

Can you provide any details about what was sub-optimal about the CKAN-compatible APIs in DKAN?  Would love to improve that if possible. We are trying to update the read APIs as they are reported to us: https://github.com/NuCivic/dkan/labels/CKAN%20Harvest%20Comptibility

best,

-Aaron

Aaron Couch

Director of Engineering for Civic Solutions

Online: http://nucivic.com

Offline: (215) 315-7915

To post to this group, send an email to ckan-global...@googlegroups.com.

Visit this group at http://groups.google.com/group/ckan-global-user-group.

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAOY41VXWrG7nbm8od-5Ued_2CXqjHGdMJ%2Baxbqe-Gg%3DZbwt0%3Dw%40mail.gmail.com.

[David Read]

Aaron,

You said to me last August that your team had not even tried using DKAN with the CKAN harvester. Yet your marketing claims compatibility and even links to it!

http://docs.getdkan.com/dkan-documentation/dkan-api/dataset-api

David

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAO-nyzT2pB%3DJX3cKP42zj4kaHV74%3DUKn-3Ezwcfo3WogXrW8Sw%40mail.gmail.com.

[Aaron Couch]

hi David:

We've created APIs that are as compatible as possible with CKAN's APIs so that folks can Harvest from DKAN to CKAN.  It looks like you've been successful with that which we are glad to hear.  The documentation you linked to lists those APIs.  We would like to improve that experience if possible so please provide any constructive feedback that you can.  We have received tickets for API enhancements or bugs which we've corrected.

best,

-Aaron

Aaron Couch

Director of Engineering for Civic Solutions

Online: http://nucivic.com

Offline: (215) 315-7915

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAOY41VXT8HJE93%3DD0wkGGRm5tx3PJdxCyOSa88qnTB%2BzVbD2Ag%40mail.gmail.com.

[David Read]

Aaron,

Why should I list the numerous issues if your engineers haven't even tried the harvester on DKAN themselves?

David

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAO-nyzSRuoCGCKHpT0bHzpVQUDhZVFS-bwwP9xG9FuCDyCkYqg%40mail.gmail.com.

[Aaron Couch]

hi David:

It looks like some of the issues that you had: https://github.com/datagovuk/ckanext-harvest/commit/7697c15d3554d4eb4d267761c75e7d5550dac477#diff-f7727de02658c14b918525571426659bR58

such as 'name' not being available and 'notes' being mislabeled as 'description' have been addressed in recent updates: http://demo.getdkan.com/api/3/action/package_show?id=gold-prices-london-1950-2008-monthly

I've made a note to do a more thorough review to see if there are other items we can make tickets for that already haven't been addressed.  We've created a schema for CKAN's package_show https://github.com/NuCivic/open_data_schema_map/blob/master/modules/open_data_schema_ckan/data/ckan_dataset_schema.json which we hope can help other systems federate to CKAN without requiring a lot of custom harvesting.  The module used to implement package_show creates a user interface for mapping Drupal entities to the CKAN API so that any Drupal site can implement it. We are hoping this lowers the bar for a lot of government agencies to federate in CKAN compatible ways.

If there are any governments that you are working with that have not been able to incorporate our recent updates please let us know and we'll try and help them to get updated.

best,

-Aaron

Aaron Couch

Director of Engineering for Civic Solutions

Online: http://nucivic.com

Offline: (215) 315-7915

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAOY41VU%3DGgLzoz9CrD3eCX-JO0--gXW9SQ%3DF4MKhC6kGVDZ55g%40mail.gmail.com.

[David Read]

Aaron,

We've discussed a more fundamental problem that has not been addressed as far as I see. With respect, you're coming at this from the wrong angle. Rather than seeing if the API *looks* the same as CKAN, you need to test whether the real CKAN harvester works on your API.

David

To view this discussion on the web, visit https://groups.google.com/d/msgid/ckan-global-user-group/CAO-nyzSOL4qWLe967GEYKVakcPkyf3LHGnCm7UsxCq%3DzDHYueg%40mail.gmail.com.