[Dspace-tech] [KE1019161] Embargo settings on item import

[marco...@kesslernetworks.de]

Hi all,

is it possible to set embergo settings during the import?
The aim is, getting items metadata into DSpace 3.0 not accessable
through the OAI and update the access state later through a second
import or batch modification.

Any ideas?

Regards Marco

[helix84]

On Thu, Jan 3, 2013 at 12:41 PM, <marco...@kesslernetworks.de> wrote:
> is it possible to set embergo settings during the import?

What method of import are you using?

> The aim is, getting items metadata into DSpace 3.0 not accessable
> through the OAI and update the access state later through a second
> import or batch modification.

Can you elaborate a little? What do you mean not accessible through OAI?

Do you wish to use the original 1.6 embargo or the new 3.0 embargo?

Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[Marco Weiß]

Hi Helix,

at the moment i tested the item import but as my system is not productive today i don't mind if i have to use another method.

Oh sorry, i mean that it is not harvestable by the OAI (ListRecord, GetRecord)

I think I have to use the 3.0 embargo because my DSpace are 3.0 or?


Regards,
Marco

--
Kessler GmbH - Netzwerke und mehr

Gewerbegebiet 6
82399 Raisting
Tel.: 08807 / 94 67 7 - 0
Fax: 08807 / 94 67 7 - 99
Web: http://www.kesslernetworks.de

Sitz der Gesellschaft: Raisting
Geschäftsführer: Florian Kessler
Prokura: Marco Weiß
HRB 163710


-----Ursprüngliche Nachricht-----
Von: ivan....@gmail.com [mailto:ivan....@gmail.com] Im Auftrag von helix84
Gesendet: Donnerstag, 3. Januar 2013 13:04
An: marco...@kesslernetworks.de
Cc: dspac...@lists.sourceforge.net
Betreff: Re: [Dspace-tech] [KE1019161] Embargo settings on item import

[helix84]

On Thu, Jan 3, 2013 at 2:23 PM, Marco Weiß
<marco...@kesslernetworks.de> wrote:
> at the moment i tested the item import but as my system is not productive today i don't mind if i have to use another method.

If you mean the old importer ([dspace]/bin/dspace import), this type
of package (Simple Archive Format) does not support storing the
embargo information (resource policies) [1]. I think the AIP format
does preserve it [2] [3]. I recommend you to create a testing item
manually from the admin interface with embargo, then export it to AIP
and examine the AIP contents manualy. There should be resource
policies in the METS file. I'm not completely sure about embargo
terms, but I didn't hear that the new embargo would not be compatible
with AIP. For the old embargo, that's not a problem because it doesn't
use any special fields (it uses standard metadata fields and custom
setter/lifter code).

> I think I have to use the 3.0 embargo because my DSpace are 3.0 or?

In DSpace 3.0 both are supported. The new embargo is easier to use for
common use cases (e.g. change bistream access policy for the specified
date range), the old one still allows you to write a custom embargo
setter/lifeter code so it's more flexible. More information is in the
official documentation.

[1] https://wiki.duraspace.org/display/DSDOC3x/Importing+and+Exporting+Items+via+Simple+Archive+Format
[2] https://wiki.duraspace.org/display/DSDOC3x/AIP+Backup+and+Restore
[3] https://wiki.duraspace.org/display/DSDOC3x/Importing+and+Exporting+Content+via+Packages

[marco...@kesslernetworks.de]

Hi all,

is it possible to perform an get request on the OAI interface and
using not the handle as identifier?
Can i use a field from the metadata scheme like dc.contributor.author?

For example that is a working request

http://10.2.46.252:8080/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:urntest:123456789/7

and i want to do a search like

http://10.2.46.252:8080/oai/request?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:urntest:dc.contributor.author=Goethe


Maybe it is possible to tell DSpace what field of the metadata is used
for identify the record?


Regards Marco

[helix84]

On Thu, Jan 17, 2013 at 9:37 AM, <marco...@kesslernetworks.de> wrote:
> is it possible to perform an get request on the OAI interface and
> using not the handle as identifier?
> Can i use a field from the metadata scheme like dc.contributor.author?

I'm reading the Unique Identifier chapter of the OAI-PMH specification
and it I don't think it explicitly says that there couldn't be
multiple unique identifiers pointing to the same record. But they do
have to be unique, so dc.contributor.author is out of the question.
The OAI in DSpace probably could be adapted to allow multiple
identifier schemas is you really wanted to, but I don't think it's a
good idea at all.

OTOH, if you don't need to conform to OAI-PMH, you just need a search
interface, you can just search for any metadata via Solr [1], which
uses really a simple, XML-based protocol (among other options). Of
course, that way by searching for a value of a metadata field (like
author) you can find the handles and still use OAI-PMH to access those
handles, if you wish. Yet another alternative is SRU/SRW, but I don't
recommend it at this time because it's implemented as an addon (i.e.
not part of standard DSpace) and it's SQL-, not Solr-based (i.e.
relatively slow).

[1] https://wiki.duraspace.org/display/DSPACE/Solr

[marco...@kesslernetworks.de]

Hi Helix,

ok dc.contributor.author was a bad example. In this case we have a
UUID that is unique and i can import it in a field named
dc.identifier.uuid.
So how can i modify DSpace to allow multiple identifier?

I asked this because i had only this idea to get the record i need.
Maybe if i explain to you what i'd like to do, you have an other idea
i don't thought about.

I'd like to get the meta data record from a system/website that knows
nothing from the handles in DSpace.
But the systems knows the UUID!
So if i maintain the UUID in DSpace as dc.identifier.uuid too both
systems know about the UUID and can match the entries.


Regards, Marco




Zitat von helix84 <hel...@centrum.sk>:

[helix84]

That sounds like a valid use case. I think it might be easy:

https://github.com/DSpace/DSpace/blob/dspace-3_x/dspace-oai/src/main/java/org/dspace/xoai/data/DSpaceItem.java#L67

[Hilton Gibson]

Also see: http://hdl.handle.net/10019.1/3161

------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
DSpace-tech mailing list
DSpac...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--

Hilton Gibson

Systems Administrator

JS Gericke Library

Room 1025D

Stellenbosch University

Private Bag X5036

Stellenbosch

7599

South Africa

Tel: +27 21 808 4100 | Cell: +27 84 646 4758

http://library.sun.ac.za

http://scholar.sun.ac.za

http://ar1.sun.ac.za

http://aj1.sun.ac.za

[marco...@kesslernetworks.de]

Hope i understand you the right way...
The only way to get this working is to expand the code?

Regards, Marco


Zitat von helix84 <hel...@centrum.sk>:

[helix84]

On Thu, Jan 17, 2013 at 11:05 AM, <marco...@kesslernetworks.de> wrote:
> Hope i understand you the right way...
> The only way to get this working is to expand the code?

Yes, certainly. But it doesn't look hard, unless there's something I missed.

[João Melo]

Hi marco,

if you are using DSpace 3.0, it's also possible do the same with filters which don't require code changes, only configuration changes.

------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
DSpace-tech mailing list
DSpac...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

--

Thanks, João Melo (My Portfolio) DSpace Department Lyncode: Official website

[marco...@kesslernetworks.de]

Hi Joao,

yes we are using DSpace 3.0. Can you provide me a hint where i can find a documentation of these filters?

Or do you mean the media filters?

 

Regards, Marco

[helix84]

On Thu, Jan 17, 2013 at 1:18 PM, <marco...@kesslernetworks.de> wrote:
> yes we are using DSpace 3.0. Can you provide me a hint where i can find a documentation of these filters?

The filters Joao meant are defined and used here:

https://github.com/DSpace/DSpace/blob/dspace-3.0/dspace/config/crosswalks/oai/xoai.xml

I don't know how they allow you to change what identifiers are
_accepted_ by oai without writing Java code, but of course Joao knows
better, he wrote the code.

[marco...@kesslernetworks.de]

Hi Helix,

it tooks a time but now i tested the embargo and the different methods
to import that settings.
In short till now it wont work.
Now what have i done. I created an item and imported it in OAI Solr
search. Searched by the OAI and got the expected output.
I exported it as a simple archive by item exporter, as a csv with batch
metadata editor (-a option) and an AIP by packager.
In DSpace it self the item was searchable and browsable.
After checking that i put the item into private state via XMLUI.
Deleting the Solr search index, recreating the search index and clearing
the cache and the item was no longer browsable or getable by OAI.
I exported it again as a simple archive by item exporter, as a csv with
batch metadata editor (-a option) and an AIP by packager.
Now i did a diff between the two states for the three different ways of
export.
The simple archive and the csv file shows no difference between the two
states except of the dc.description.provenance.
The unpacked AIP shows a difference between the two mets.xml files.
That tells me, that only over AIP i can handle the embargo settings,
or?

So i had the idea now to restore/replace it with the AIP i created as
the item was in "not private" state and it have to be in that state
after replacing it.
Unfortunately it replaces the item but the "not private" state was not
applied to the item.
I set the "not private" state in XMLUI manually and imported the AIP i
created as the item was in the "private" state and this works in parts.
In parts means the item is withdrawn but in "not private" state after
that.
That means replacing an item that is in "not private" state with the
"private" state item works the other way round not.
To see the difference between the withdrawn item that is in "not
private" state i put it manually in "private" state, exported it as AIP,
unzipped, and diff the two mets.xml ... no changes that shows me what i
have to set so get the item in "private"state.

Importing an item with AIP and withdraw options will cause an error
like this.

Replacing DSpace object(s) with package located at aip24.zip
org.dspace.authorize.AuthorizeException: To withdraw item must be
COLLECTION_ADMIN or have REMOVE authorization on owning Collection
at
org.dspace.app.util.AuthorizeUtil.authorizeWithdrawItem(AuthorizeUtil.java:574)
at org.dspace.content.Item.withdraw(Item.java:1870)
at
org.dspace.content.crosswalk.AIPTechMDCrosswalk.ingest(AIPTechMDCrosswalk.java:452)
at
org.dspace.content.crosswalk.AIPTechMDCrosswalk.ingest(AIPTechMDCrosswalk.java:350)
at
org.dspace.content.packager.METSManifest.crosswalkXmd(METSManifest.java:1151)
at
org.dspace.content.packager.METSManifest.crosswalkObjectSourceMD(METSManifest.java:1093)
at
org.dspace.content.packager.AbstractMETSIngester.ingestObject(AbstractMETSIngester.java:456)
at
org.dspace.content.packager.AbstractMETSIngester.replace(AbstractMETSIngester.java:1134)
at org.dspace.app.packager.Packager.replace(Packager.java:766)
at org.dspace.app.packager.Packager.main(Packager.java:373)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:183)
org.dspace.authorize.AuthorizeException: To withdraw item must be
COLLECTION_ADMIN or have REMOVE authorization on owning Collection


Puh so, till now i have no way to import items in private state.
I can import it and, replace it afterwards to get it withdrawn but not
in state "private"

Can you please confirm if i'm right here or enlighten me ;)



Best regards, Marco

[helix84]

On Fri, Jan 18, 2013 at 11:36 AM, <marco...@kesslernetworks.de> wrote:
> Now what have i done. I created an item and imported it in OAI Solr search.

What do you mean OAI Solr search? I assume you mean the "oai" Solr
core? "search" is a different Solr core, used for Discovery.

> The simple archive and the csv file shows no difference between the two
> states except of the dc.description.provenance.
> The unpacked AIP shows a difference between the two mets.xml files.
> That tells me, that only over AIP i can handle the embargo settings, or?

Yes, as I wrote in my previous email.
Old embargo = bitstream authorizations + embargo terms in metadata (so
SAF import should recognize it)
3.0 embargo = bitstream resource policies (so probably only AIP would
recognize it - it's part of METS)

> So i had the idea now to restore/replace it with the AIP i created as the
> item was in "not private" state and it have to be in that state after
> replacing it.
> Unfortunately it replaces the item but the "not private" state was not
> applied to the item.
> I set the "not private" state in XMLUI manually and imported the AIP i
> created as the item was in the "private" state and this works in parts.
> In parts means the item is withdrawn but in "not private" state after that.
> That means replacing an item that is in "not private" state with the
> "private" state item works the other way round not.
> To see the difference between the withdrawn item that is in "not private"
> state i put it manually in "private" state, exported it as AIP, unzipped,
> and diff the two mets.xml ... no changes that shows me what i have to set so
> get the item in "private"state.

Thanks for the testing, you may have found a bug. I didn't try it
myself (exporting/importing items with embargo via AIP), so I can't
explain it. If noone else answers in the next few days, you should
file a Jira issue to make sure it's addressed (either explained if
it's supposed to work that way, or fixed if it's not).

> org.dspace.authorize.AuthorizeException: To withdraw item must be
> COLLECTION_ADMIN or have REMOVE authorization on owning Collection

This sounds pretty straightforward - did you use a correct value for
the -e flag? E.g. a site admin eperson?

[marco...@kesslernetworks.de]

Hi Helix,

thank you for that link. I now new which direction Joao and you are
thinking of but i can't find a documentation what i have to do.
Just with the configuration file and the note "OAI 2.0 allows you to
configure following advanced options ... filters" i can't understand
what to do.
Maybe i'm to stupid to get my gyrus around that, and googling the half
day doesn't solve it ;)

Maybe Joao can you explain it to me or give me a hint?


Regards, Marco

[João Melo]

Hi Marco,

sorry for the delay. Unfortunately i've misunderstood, as helix said, it requires code changes.

But, i would strongly disencourage you, OAI-PMH protocol is not meant to be used for custom query purposes, you could use some rest-api instead (we are using and it works fine).

If you have any further questions (about OAI/rest-api), please, feel free to question.

On 18 January 2013 15:12, <marco...@kesslernetworks.de> wrote:

Hi Helix,

thank you for that link. I now new which direction Joao and you are thinking of but i can't find a documentation what i have to do.
Just with the configuration file and the note "OAI 2.0 allows you to configure following advanced options ... filters" i can't understand what to do.
Maybe i'm to stupid to get my gyrus around that, and googling the half day doesn't solve it ;)

Maybe Joao can you explain it to me or give me a hint?


Regards, Marco


Am 17.01.2013 13:35, schrieb helix84:

On Thu, Jan 17, 2013 at 1:18 PM, <marco.weiss@kesslernetworks.de> wrote:

yes we are using DSpace 3.0. Can you provide me a hint where i can find a documentation of these filters?

The filters Joao meant are defined and used here:


https://github.com/DSpace/DSpace/blob/dspace-3.0/dspace/config/crosswalks/oai/xoai.xml

I don't know how they allow you to change what identifiers are
_accepted_ by oai without writing Java code, but of course Joao knows
better, he wrote the code.


Regards,
~~helix84

Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

[marco...@kesslernetworks.de]

Hi Joao,

thank you for answering! Ok REST would be nice but i see there is not documentation of such a implementation.

I found these instructions

https://jspace.atlassian.net/wiki/display/DSPACEAPI/DSpace+REST+API+Home

https://jspace.atlassian.net/wiki/display/DSPACEAPI/API+Documentation

https://wiki.duraspace.org/display/DSPACE/REST+API

 

It seems to be in beta status, or?

Other idea, is it possible to use the solr search server to get what i want?

 

 

Regards Marco

[helix84]

On Tue, Jan 22, 2013 at 9:01 AM, <marco...@kesslernetworks.de> wrote:
> Other idea, is it possible to use the solr search server to get what i
> want?

Yes! Please, read my first email.

[João Melo]

Hi Marco,

about the rest-api (we use https://github.com/wijiti/dspace-rest-api), it lacks of documentation, if you are willing to use it, i've uploaded a short manual, look at: 

http://lyncode.github.com/dspace-rest-api/

Any doubt, please, feel free to question, it may help improving it.

[marco...@kesslernetworks.de]

Hi Helix,

thank you for remembering! My brain lost that information somewhere...
I now using the solr and it works well.
I deactivated the localhost restriction in web.xml
and did my first search.

http://10.2.46.252:8080/solr/oai/select?q=metadata.dc.identifier.uuid:955a3c1d-921b-4522-a452-672b4fe86d09&fl=item.handle

@Joao thanks for the link to the REST information but i will use the
Solr.


Regards, Marco

[helix84]

On Mon, Jan 28, 2013 at 8:29 AM, <marco...@kesslernetworks.de> wrote:

> Am 18.01.2013 12:52, schrieb helix84:
>> On Fri, Jan 18, 2013 at 11:36 AM, <marco...@kesslernetworks.de> wrote:
>>> Now what have i done. I created an item and imported it in OAI Solr
>>> search.
>>
>> What do you mean OAI Solr search? I assume you mean the "oai" Solr
>> core? "search" is a different Solr core, used for Discovery.
>

> With OAI Solr search i meant i imported the items in Solr so that a query on
> OAI can deliver the data getting it from the Solr ... hope i'm right here.

Yes, that is the "oai" Solr core and you've done well.

>>> The simple archive and the csv file shows no difference between the two
>>> states except of the dc.description.provenance.
>>> The unpacked AIP shows a difference between the two mets.xml files.
>>> That tells me, that only over AIP i can handle the embargo settings, or?
>>
>> Yes, as I wrote in my previous email.
>> Old embargo = bitstream authorizations + embargo terms in metadata (so
>> SAF import should recognize it)
>> 3.0 embargo = bitstream resource policies (so probably only AIP would
>> recognize it - it's part of METS)
>

> Ok so the old embargo should work, but what is SAF ? Can't find something in
> the documentation.
> Unfortunately i don't have enough time to test it because it was just nice
> to have and not must have criteria.
> Maybe in a later step i will get back to this. The step is planed but i can
> not tell when.

Sorry, SAF is just a name for the format used in the original importer
([dspace]/bin/dspace import/export).

>>> So i had the idea now to restore/replace it with the AIP i created as the
>>> item was in "not private" state and it have to be in that state after
>>> replacing it.
>>> Unfortunately it replaces the item but the "not private" state was not
>>> applied to the item.
>>> I set the "not private" state in XMLUI manually and imported the AIP i
>>> created as the item was in the "private" state and this works in parts.
>>> In parts means the item is withdrawn but in "not private" state after
>>> that.
>>> That means replacing an item that is in "not private" state with the
>>> "private" state item works the other way round not.
>>> To see the difference between the withdrawn item that is in "not private"
>>> state i put it manually in "private" state, exported it as AIP, unzipped,
>>> and diff the two mets.xml ... no changes that shows me what i have to set
>>> so
>>> get the item in "private"state.
>>
>>
>> Thanks for the testing, you may have found a bug. I didn't try it
>> myself (exporting/importing items with embargo via AIP), so I can't
>> explain it. If noone else answers in the next few days, you should
>> file a Jira issue to make sure it's addressed (either explained if
>> it's supposed to work that way, or fixed if it's not).
>
>

> Ok no one had answered to this mail so i will file a Jira issue.

>
>
>>> org.dspace.authorize.AuthorizeException: To withdraw item must be
>>> COLLECTION_ADMIN or have REMOVE authorization on owning Collection
>>
>>
>> This sounds pretty straightforward - did you use a correct value for
>> the -e flag? E.g. a site admin eperson?
>
>

> Yes, sure i only have one account the site admin i created following the
> installation documentation.
> And yes i used the -e flag but in long version
> --eperson=my....@adress.de

Sorry, I can't think of anything else that could have gone wrong there.

> Ok, i hope i can find a bit spare time to place a Jira issue the next day
> for that and i'll post the link to that.

[marco...@kesslernetworks.de]

Hi Helix,

it's a bit late my answer but here is it.

Am 18.01.2013 12:52, schrieb helix84:

> On Fri, Jan 18, 2013 at 11:36 AM, <marco...@kesslernetworks.de>
> wrote:
>> Now what have i done. I created an item and imported it in OAI Solr
>> search.
>
> What do you mean OAI Solr search? I assume you mean the "oai" Solr
> core? "search" is a different Solr core, used for Discovery.

With OAI Solr search i meant i imported the items in Solr so that a
query on OAI can deliver the data getting it from the Solr ... hope i'm
right here.

>> The simple archive and the csv file shows no difference between the
>> two
>> states except of the dc.description.provenance.
>> The unpacked AIP shows a difference between the two mets.xml files.
>> That tells me, that only over AIP i can handle the embargo settings,
>> or?
>
> Yes, as I wrote in my previous email.
> Old embargo = bitstream authorizations + embargo terms in metadata
> (so
> SAF import should recognize it)
> 3.0 embargo = bitstream resource policies (so probably only AIP would
> recognize it - it's part of METS)

Ok so the old embargo should work, but what is SAF ? Can't find
something in the documentation.
Unfortunately i don't have enough time to test it because it was just
nice to have and not must have criteria.
Maybe in a later step i will get back to this. The step is planed but i
can not tell when.

Ok no one had answered to this mail so i will file a Jira issue.

>> org.dspace.authorize.AuthorizeException: To withdraw item must be
>> COLLECTION_ADMIN or have REMOVE authorization on owning Collection
>
> This sounds pretty straightforward - did you use a correct value for
> the -e flag? E.g. a site admin eperson?

Yes, sure i only have one account the site admin i created following
the installation documentation.
And yes i used the -e flag but in long version
--eperson=my....@adress.de

Ok, i hope i can find a bit spare time to place a Jira issue the next
day for that and i'll post the link to that.

Regards Marco

[marco...@kesslernetworks.de]

Hi all,

i now tested i again in 3.1 and its the same as in 3.0
I placed a Jira issue https://jira.duraspace.org/browse/DS-1514 like
helix advice a few mails ago.


Regards, Marco