Hi,
I noticed this week that we had duplicate bitstream rows in the "Bitstream" table in Dspace 1.2.2. Only one was marked "Deleted=false" however and I'm assuming that these duplicate rows got in there when multiple imports were done into DSpace way back when and no one ever noticed it. They all had different bitstream ids, however all had the same name, size, checksum, etc and there was only one item in the Item table that related to a row in the Bitstream table (the one with Deleted=false of course). I deleted over 6700 of these duplicate records this morning, but now I need to figure out a way to delete the corresponding documents in Assetstore. Does anyone have a nice script that will accomplish this, that has been tested and proven to work????? :-)
Thanks,
Sue
NASA Langley Research Center
Technical Library
ConITS Contract
![]() | |||
|
Hi,
We run index-all every night, but my understanding was that this job is what builds the search indices. I've also run cleanup and it doesn't do a thing as far as I can see. My row count in Bitstream is exactly the same before and after cleanup runs. I have 8452 rows in Bitstream that are marked as "Deleted" = true and they are still there after Cleanup runs. I read in the documentation that Cleanup looks in Bitstream and deletes all rows marked as "Deleted" = true if they are more than one hour old. So now I'm left with the dilemma of having to write something that deletes the rows from Bitstream that are marked as "Deleted" and then delete the corresponding record out of Assetstore.
Thanks,
Sue
-------Original Message------- |
Hi again Grant,
Well, lookie what I just found in dspace.log:
2006-04-18 14:43:53,086 INFO org.dspace.storage.bitstore.Cleanup @ Cleaning up asset store
2006-04-18 14:43:55,330 FATAL org.dspace.storage.bitstore.Cleanup @ Caught exception: java.lang.NullPointerException at org.dspace.storage.bitstore.BitstreamStorageManager.deleteParents(BitstreamStorageManager.java:740) at org.dspace.storage.bitstore.BitstreamStorageManager.cleanup(BitstreamStorageManager.java:673) at org.dspace.storage.bitstore.Cleanup.main(Cleanup.java:67) It's having some kind of problem on the highlighted line (if (files.length !=0)):
// Only delete empty directories
if (files.length != 0) { break; } directory.delete();
tmp = directory; I'm new at Java and have successfully found several errors and solutions in the past 6 months, however I'm stumped on this one right now.
Do you have any ideas??
Thanks much, |
Sue
-------Original Message-------
From: Grant Johnson
Date: 04/18/06 13:51:56
Subject: Re: [Dspace-tech] Does anyone have a script that gets rid of orphaned Assetstore files?? |
Hmmmmm...
Sounds like what I was fighting with.
Again - I hope this isn't too simplistic a suggestion.........
Is Tomcat starting as user dspace?
Tomcat needs to start as user dspace, I was starting Tomcat as root and
therefore .cleanup(as dspace) did nothing b/c of user rights.
As soon as Tomcat was configured to launch "dspace" ./cleanup runs
successfully.
Hope this helps. |
--
F. Grant Johnson
566-0630 / fgjo...@upei.ca
Systems/Web Coordinator
RM 285 - Robertson Library
University of Prince Edward Island
***************
Attitude is IT!
. |
Hi Scott,
Thanks very much for the suggestion. I am running a mass-record import right now and don't want to mess with Assetstore, but will try this after the import finishes. One question - what exactly is "Cleanup" supposed to do? Is it supposed to *both* delete the records in Bitstream marked as deleted AND delete the corresponding record in Assetstore?
Thanks again Scott - will let you know how it turns out!
Sue
-------Original Message------- |
I'm trying to make the same change to BitstreamStorageManager.java, but am getting compile errors with the following code:
import edu.sdsc.grid.io.FileFactory;
import edu.sdsc.grid.io.GeneralFile; import edu.sdsc.grid.io.GeneralFileOutputStream; import edu.sdsc.grid.io.local.LocalFile; import edu.sdsc.grid.io.srb.SRBAccount; import edu.sdsc.grid.io.srb.SRBFile; import edu.sdsc.grid.io.srb.SRBFileSystem; What do I need to get the program to compile??
Thanks,
Sue
-------Original Message------- |
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
_______________________________________________
DSpace-tech mailing list
. | |||
|
Thanks everyone who replied to this message. After I correctly added jargon.jar to the classpath, it compiled fine.
Thanks again!!
Sue
-------Original Message-------
|
From: Sue Walker-Thornton
Date: 04/25/06 16:23:12
|
Sorry - forgot to include my code....Here is the code I put in BitstreamStorageManager.java:
DatabaseManager.delete(context, "Bitstream", bid);
if (isRegisteredBitstream(row.getStringColumn("internal_id"))) {
continue; // do not delete registered bitstreams } boolean success = file.delete();
if (log.isDebugEnabled()) { log.debug("log is debug enabled"); <-- this does NOT display in log, but other displays show up in the log (see bottom of this code) |
log.debug("Deleted bitstream " + bid + " (file " + file.getAbsolutePath() + ") with result " + success); } |
// 04/25/2006 SMWT - Change begins
if (success) { deleteParents(file); } // deleteParents(file);
// 04/25/2006 - Change ends } context.complete();
} Here is what DOES display in the log:
2006-04-26 13:41:16,221 INFO org.dspace.storage.bitstore.Cleanup @ Cleaning up asset store 2006-04-26 13:41:16,505 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query "select * from Bitstream where deleted = 't'" 2006-04-26 13:41:18,128 DEBUG org.dspace.storage.bitstore.BitstreamStorageManager @ Local filename for 143932774548522411732039458334760354644 is /export/home/dspace/assetstore/14/39/32/143932774548522411732039458334760354644 2006-04-26 13:41:18,130 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query "delete from bitstream where bitstream_id = '21'" 2006-04-26 13:41:18,231 DEBUG org.dspace.storage.bitstore.BitstreamStorageManager @ Deleted bitstream 21 (file /export/home/dspace/assetstore/14/39/32/143932774548522411732039458334760354644) with result false 2006-04-26 13:41:18,250 FATAL org.dspace.storage.bitstore.Cleanup @ Caught exception: |
java.lang.NullPointerException at org.dspace.storage.bitstore.BitstreamStorageManager.deleteParents(BitstreamStorageManager.java:740) at org.dspace.storage.bitstore.BitstreamStorageManager.cleanup(BitstreamStorageManager.java:673) at org.dspace.storage.bitstore.Cleanup.main(Cleanup.java:67) |
I'm also attaching a copy of my BitstreamStorageManager.java to this email.
Thanks a bunch!
Sue
-------Original Message-------
|
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
_______________________________________________
DSpace-tech mailing list
. |
Hi,
I changed my log level to DEBUG, made the program change outlined below, got the program to compile, copied the .class file back to dspace/jsp/WIN-INF.....classes/storage... and also to dspace/WIN-INF...classes/storage... and also to dspace-1.3.2-source/build. For some reason, I am still getting the exact same error(s) as before with the exact same line numbers. I even put in some more log.debug statements to display things and none of these are displaying in the log, which indicates to me that my new version of the program is NOT being picked up. Is there something I'm missing?
Thanks,
Sue
-------Original Message-------
|
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
_______________________________________________
DSpace-tech mailing list
. |
Hi Scott - So the .class file in /dspace/jsp/WEB-INF/classes...... are not used?? Or rather, they are not the .class files that get executed??? Before, whenever I changed a java program, I compiled it myself in JCreator, then copied the source back to /dspace-source and copied the .class files to /dspace-source/build and somehow it's always worked until now......With BitstreamStorageManager, I couldn't get it to see my changes until I ran the ant....update command. Any ideas?
Thanks,
Sue
-------Original Message-------
|
Ah.....that's why my other changes worked.....because they were to "online" programs, not "batch" like "cleanup". (Can you tell I'm an old mainframe programmer??? :-)
Thanks again,
Sue
-------Original Message-------
|
From: Scott Yeadon
Date: 04/26/06 19:05:15
|
It worked!! I never knew about the ant....update command. Before, I always compiled my own java code and manually copied the .class files over. I honestly don't know how that one got by me. :-) THANK YOU very much for your help! I now have a nice clean Bitstream table and Assetstore directory.
Thanks again,
Sue
p.s. I also made another change to BitstreamStorageManager.java....In that same block of code that you changed, I also check for (success) before I check for log.isDebugEnabled so that you ONLY get that line of code in your log IF the delete was successful. Before, if DEBUG was enabled, it was displaying that line indicating it had done the delete when it actually had not. By the way, the problem it had with my application is that it was trying to delete 102534............ and there was a subdirectory 10 and there was a subdirectory 25, but there was NOT a subdirectory 34.
-------Original Message-------
|