backup restore testing.

[Francis Kayiwa]

Hey I have a sanity check question here. We are attempting to do this
and I'm hoping someone who has done this can tell us what dragons to
look for.

Our problem is.

System1 (currently running on Oracle)
We want to move it to
System2 (running PostgreSQL)

We've chosen this path

https://wiki.lyrasis.org/display/DSDOC6x/AIP+Backup+and+Restore

We exported successfully from System1 to A Google Cloud bucket.
(definition of success is no errors were reported)
We've manage to import some them into System2 with some errors that
need manual intervention (which isn't the worst fate given the scope)

Since this is mostly a proof of concept our questions are. If we
re-run the step above will it overwrite the items? I say this because
the import process has the replace flag. Or will we have to incur the
same Google Cloud Costs for network transfer when we do this "for
real"

Cheers,
./fxk

--
E Pluribus Unix

[Hardy Pottinger]

Hey, Francis. It has been more than three years since I have run a DSpace instance on Oracle, but I do remember doing research on this exit path, and running into "issues." My memory has faded a bit, I did not retain the notes, and I do not actively maintain a DSpace instance any more (and thus am an Emeritus Committer). However, with those caveats...

> If we re-run the step above will it overwrite the items? 

> I say this because the import process has the replace flag. Or will we have to incur the 

> same Google Cloud Costs for network transfer when we do this "for 
> real" 

I believe that the replace flag is for situations where you do not want to create a new item (i.e. "mint a new handle"). I do think that you would still be "pushing bits" if you re-run your AIP load. Looking at the code for the AIP restore process might give you an answer? I'll check in a bit and let you know what I find.

--Hardy

[Mark H. Wood]

On Fri, Dec 13, 2019 at 08:11:51AM -0800, Hardy Pottinger wrote:
> Hey, Francis. It has been more than three years since I have run a DSpace
> instance on Oracle, but I do remember doing research on this exit path, and
> running into "issues." My memory has faded a bit, I did not retain the
> notes, and I do not actively maintain a DSpace instance any more (and thus
> am an Emeritus Committer). However, with those caveats...
>
> > If we re-run the step above will it overwrite the items?
> > I say this because the import process has the replace flag. Or will we
> have to incur the
> > same Google Cloud Costs for network transfer when we do this "for
> > real"
>
> I believe that the replace flag is for situations where you do not want to
> create a new item (i.e. "mint a new handle"). I do think that you would
> still be "pushing bits" if you re-run your AIP load. Looking at the code
> for the AIP restore process might give you an answer? I'll check in a bit
> and let you know what I find.

The answer seems to be in AbstractMETSIngester#replaceObject. It
deletes all bitstreams and metadata from the object and then adds the
new ones. I don't see any check for identical content.

That might be a nice enhancement. Everything that you really need
should be in the database. The existing Bitstreams' checksums could
be compared to their replacements without touching remote storage.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

[Francis Kayiwa]

On Fri, Dec 13, 2019 at 11:43 AM Mark H. Wood <mwood...@gmail.com> wrote:
>
> On Fri, Dec 13, 2019 at 08:11:51AM -0800, Hardy Pottinger wrote:
> > Hey, Francis. It has been more than three years since I have run a DSpace
> > instance on Oracle, but I do remember doing research on this exit path, and
> > running into "issues." My memory has faded a bit, I did not retain the
> > notes, and I do not actively maintain a DSpace instance any more (and thus
> > am an Emeritus Committer). However, with those caveats...
> >
> > > If we re-run the step above will it overwrite the items?
> > > I say this because the import process has the replace flag. Or will we
> > have to incur the
> > > same Google Cloud Costs for network transfer when we do this "for
> > > real"
> >
> > I believe that the replace flag is for situations where you do not want to
> > create a new item (i.e. "mint a new handle"). I do think that you would
> > still be "pushing bits" if you re-run your AIP load. Looking at the code
> > for the AIP restore process might give you an answer? I'll check in a bit
> > and let you know what I find.

For our "interesting-ness" this is the answer we needed, confirmation
on. So no need to re-run this until we are indeed ready.

Thanks y'all!