org.modeshape.jcr.value.ValueFormatException: Error converting "ns009:Object" from String to a Name

[Steven Anderson]

Hiya,

I recently did an upgrade from Fedora 4.7.0 to Fedora 4.7.2 and things were going well for about two weeks. However, suddenly, it just suddenly stopped working with the above error message. It seems possibly related to: https://groups.google.com/d/topic/fedora-tech/Q1fTNYx8UF0/discussion

Full stack trace of the error is attached. The site that I help to maintain (digitaltransgenderarchive.net) is down due to service failing to start and I'm unable to find a workaround. Even attempting to revert back to Fedora 4.7.0 failed to help. Any suggestions on what I might be able to do? Thanks!

[Steven Anderson]

From looking through the logs around the time this occurred, I can provide some error messages that happened at that time. However, I don't know what might be significant and these only showed up around the time the Fedora 4 instance stopped working. Not sure if any of these help but thought I'd include them (compiled from various catalina log files):

ARN 18:38:36.251 (Errors) The following warnings have been detected: WARNING: HK2 failure has been detected in a code that does not run in an active Jersey Error scope.
WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 4
java.lang.OutOfMemoryError: Java heap space
MultiException stack 2 of 4
java.lang.IllegalArgumentException: While attempting to create a Proxy for javax.servlet.http.HttpServletRequest in proxiable scope org.glassfish.jersey.process.internal.RequestScoped an error occured while creating the proxy

(This last successfully loaded object had a dozen lines of of "@^@^" repeated weirdly after it)

INFO 17:38:15.561 (FedoraLdp) HEAD for: prod/9w/03/23/02/9w0323021/thumbnail

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600"

Exception in thread "TxCleanupService,FedoraRepository,local" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-149" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-150" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-151" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-152" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-153" java.lang.OutOfMemoryError: Java heap space

ERROR 00:00:04.836 (JcrRepository) Error during background garbage collection: Java heap space

java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-154" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-443-exec-7" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-443-Acceptor-0" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-443-exec-8" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-155" java.lang.OutOfMemoryError: Java heap space

Exception in thread "http-bio-80-exec-156" Exception in thread "http-bio-80-Acceptor-0" java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

(This one does continue to happen):

ERROR 23:09:28.350 (RepositoryLockManager) Error while cleaning up locks for the "repo" repository
java.lang.NullPointerException: null

Thanks for any assistance!

Sincerely,

Steven Anderson

[Andrew Woods]

Hello Steven,

There appear to be two issues at hand. One is the ValueFormatException (VFE) and the other is the OutOfMemoryError (OOM). Although they may be related, that is not necessarily the case.

Related to the VFE, 

1. Can you describe if there have been any customizations to CND files? 

2. Can you think of any repository activities that may have occurred two weeks after the upgrade that may have provoked the error?

3. Were there any other upgrades to the repository environment in addition to the Fedora version?

4. If you are using Hydra, which version?

As for the OOM error, this looks like the JVM options should be tuned:

https://wiki.duraspace.org/display/FEDORA47/Java+HotSpot+VM+Options+recommendations

BTW, Daniel, where has your issue landed?

Thanks,

Andrew

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Armintor]

Hi Steven,

I spent some time this morning looking over the commits to MODE since the version included in the FCREPO-4.7.0 release to see if anything jumped out, but I'm not seeing anything yet.  There were some changes to the configurations of both FCREPO and MODE in those releases, so anything you could share about the json or spring configurations would probably help, too.

- Ben

On Sat, May 13, 2017 at 5:48 AM, Steven Anderson <stevencar...@gmail.com> wrote:

--

[Steven Anderson]

Ben,

Where would I get the JSON and spring configurations? I believe those should be default as I haven't customized those to my knowledge. Thanks for taking a look at this issue!

Also:  this is most likely a data corruption issue of some kind. I've restored a previous months old backup that I had without issue (still waiting to hear from this client's server admin if he has backups based on the process I showed him how to do over a year ago). Additionally, if the current data store was valid, reverting back to Fedora 4.7.0 should have worked unless some change broke backward compatibility of the data store with that version.

From Google sources, it seems the current potential for Fedora 4 Commons to mess up its data store has been known about by the Hydra community since March 14th based on: https://github.com/projecthydra/active_fedora/pull/1213 . As the error message is the same as what I am encountering, I assume one can use a version of fcrepo_wrapper prior to that change to help debug what is going on. The comments there don't seem to have a clear resolution as to what was going on. I can fairly confident that it is the same issue as it references the following issue that lacks details but has the same exact overall stack trace that I am experiencing: https://github.com/fcrepo4/fcrepo4/issues/1178

In the meantime, while I wait to hear if there is a more recent backup of the repository, how do I fix a corrupted whatever might be wrong with the data store? For example, when Fedora 4 loads, how is it validating its repo? Is it simply the database and the objectStore? The database shows a last_update of prior to this error occurring and I can't see anything in the readable part of the blob content there that jumps out at me as being incorrect. I tried to modify the ActiveMQ state but nothing there seemed to make a difference. I haven't touched the objectStore since it is the most unknown to me... but would that be where any bad repository state errors on load might reside somehow?

Thanks for any help you might provide!

Sincerely,

Steven Anderson

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.

[Benjamin Armintor]

Steven,

I have a foot in the Hydra community as well, but as far as I can tell this is something strange at the Modeshape layer (and this is what I'm trying to track down). Modeshape is the storage management component that it relies on (http://modeshape.jboss.org/ https://modeshape.wordpress.com/). As far as I can tell MODE is validating the namespaces as some kind of property stored on the system node, which I presume is retrievable from the database - but I don't know how to reconstruct it, at least not yet.

- Ben

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

Ben,

How do I tell what the system node is in the MySQL database? Would it just have an id of "repo" or "prod"?

Andrew,

Has missed your message before. Apologies! 

1, 2, and 3: None

4: It is based on Sufia 6.6.1 which uses ActiveFedora 9.7.2 (with some modifications... for example, Fedora 4.7.2 changed some content-type header responses from "text/turtle" to "text/turtle;charset=utf-8" that causes errors with an unmodified ActiveFedora 9.7.2). Unsure of the hydra-head version of the top of my head but can get that if wanted.

Less worried about the errors that happened when Fedora stopped working than just getting the data back to a state that Fedora can understand and start with though.

Thanks!

Sincerely,

Steven Anderson

[Embry, Randall Paul]

I mentioned this earlier, but in case it was overlooked and is relevant, here was my experience in attempting to upgrade from 4.7.0 to 4.7.2 using a copy of production data. Things seem to work ok, but the log messages are too disconcerting to attempt this in the actual production environment. I’ve not made any notable changes from the default configurations.

ERROR 13:51:22.960 (RepositoryNodeTypeManager) Node types were read from the system content, and appear to be inconsistent or invalid: repo

java.lang.NullPointerException: null

        at org.modeshape.jcr.SystemContent.first(SystemContent.java:932) 

I’ve pasted the full stacktrace here: https://gist.github.com/rpembry/697aa35a3a3a568fc005c2e8cd7f35e8

Casual inspection through the REST GUI seems like things are running ok, but I’m also seeing a fair number of messages like these in catalina.out:

ERROR 13:57:18.532 (WebApplicationExceptionMapper) WebApplicationException intercepted by WebApplicationExceptionMapper: HTTP 304 Not Modified

Thanks,

--Randall

[dsanfo...@gmail.com]

We've had this problem with test batches of our production data when moving from 4.7.0 to 4.7.1. Something, we're not sure what, comes in and breaks the modeshape data. It works until the service is restarted at which time it can no longer boot up. There's a thread here (https://groups.google.com/forum/#!topic/fedora-tech/Q1fTNYx8UF0) about our problems with things so far. Right now our only solution has been to take a backup of the data from prior to the migration and rebuild it on a machine still running 4.7.0.

Daniel

[Benjamin Armintor]

Randall-

It's possible that this is related, but at least some of the effects look a little different. Here are the differences I see:

1) upgrading to a different version (with a different bundled MODE implementation)

2) NPE instead of the naming exception

2) your repo starts

3) You get a lot of messages about 304s.

That latter is a little surprising to me just as a matter of slf4j implementation, because that message is only logged at the debug level: https://github.com/fcrepo4/fcrepo4/blob/fcrepo-4.7.2/fcrepo-http-commons/src/main/java/org/fcrepo/http/commons/exceptionhandlers/ExceptionDebugLogging.java#L43

Are there actual requests that would elicit a 304 for your system (namely, conditional GETs from a cache or similar)? Do you have logging turned up to DEBUG?

The null pointer is worrying, but it seems like it might be a different problem- weirdly, for some reason, one of the core JCR CND properties is not configured (but some of them are) in your system:

https://github.com/ModeShape/modeshape/blob/modeshape-5.2.0.Final/modeshape-jcr/src/main/java/org/modeshape/jcr/SystemContent.java#L798

https://github.com/ModeShape/modeshape/blob/modeshape-5.2.0.Final/modeshape-jcr/src/main/resources/org/modeshape/jcr/jsr_283_builtins.cnd#L52

But at least one other is. This might be a database issue (worth raising with Modeshape), but it seems like it might be a problem locating that config file, or triggering its loading.

This is all to say: It looks like some kind of problem, but maybe also a different problem? It might be better to get this into its own ticket and/or thread. Maybe it's a hint about what's going on here, but it will help keep the subsequent conversation/investigation straight (so we know how to verify what).

- Ben

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

Unfortunately, I learned there are no backups of our data. According the server admin of the client, the automated backup had silently broken, and so the only recent copy of the data is from the Fedora 4.7.2 state. As I have a backup from several months ago up and running on production at the moment, they are currently building another VM for me to see what I can recover. Which, beyond the file contents, is quite a significant amount of metadata work.

So any help on what I can do to resolve this is appreciated! I'll be continuing to try to find a way to resolve this issue tonight.

I do also appreciate others speaking up about their experiences with the upgrade. It does seem like Daniel encountered the exact same issue based on his log (and do recognize that Randall's appears a bit different). With that combined with the same stacktrace showing up in those using fcrepo_wrapper for tests, would it be worth marking 4.7.1 and 4.7.2 release downloads with a warning and suggest staying on 4.7.0 until what is going on is figured out? Might save others from arriving in a bad state but I could be wrong about how to best handle this.

Thanks!

[Embry, Randall Paul]

Per Ben’s suggestion, I’m opening this up as a separate thread.

As a sanity check, I redeployed into a fresh install of Tomcat. I’m still seeing the:

ERROR 14:03:28.834 (WebApplicationExceptionMapper) WebApplicationException intercepted by WebApplicationExceptionMapper: HTTP 304 Not Modified

and I don’t have any extra DEBUG logging turned on. The 304s seem reasonable (for instance, they appear when I hit reload in the browser), but I’m surprised to see them logged as errors.

I’ve deployed using the released fcrepo-webapp-4.7.2.war file with a few modifications to configure my database, add container authentication and to turn off JMS.

Something else I stumbled into that I find interesting is that the MODESHAPE_REPOSITORY table contains a row with my production database server name in it. It has ID of ‘repository:info’ and contains the JDBC connection string jdbc:mysql://abc.def.iu.edu:3306/mydbname?createDatabaseIfNotExist=true

I decided to look at the 300 oldest rows in the database and came across something suspicious. The colons and slashes in some of the relevant URLs don’t seem like colons and slashes to me. See here: http://imgur.com/Lom0EuO But it seems to both be in the key and the data, so I’m not sure what to think.

I’m trying to compare this with my live production database, but the query is taking a long time.

--Randall

Begin forwarded message:

[Benjamin Armintor]

Just as regards the colon and slash characters: I assume that's the effect of encoding internally characters that aren't permitted to be part of a JCR name value:

https://github.com/ModeShape/modeshape/blob/modeshape-5.2.0.Final/modeshape-common/src/main/java/org/modeshape/common/text/Jsr283Encoder.java

In the final JCR 2.0 spec that's in section 3.2.5.4, not the section the comments indicate.

The escaped UTF8 characters would look like boxed question marks on a Linux system. But I'll check to see if that process is no longer undone on the MODE side (I guess also make sure that your db is handling the utf8 data correctly, since the connection URL looks like it's using the default?).

- Ben

[Steven Anderson]

About to call it for the night but some further research:

1. My database indeed wasn't set to UTF-8 which meant many things in it had actual "?" instead of the correct character encoding for their ID value. This has been manually fixed. Potentially this lead to the issue I'm experiencing?

2. However, this wasn't the issue I was experiencing. Rather, it seems to come from the following values in my system, I think (with object ids varying compared to others):

• 87a0a8c317f1e7jcr:nodeTypes -> This has a hard-entered reference to ns009 within it. I can edit the value to, say, ns008 to verify that this is where the error is coming from. Which one might just suggest removing it from that blob content... but it uses some kind of counter and children reference within that complains if I change the length of the namespace or delete the entry.
  
• The ns009 is also used within 87a0a8c317f1e7/jcr:system/jcr:nodeTypes/{httppcdm.orgmodels#}Object but I don't believe that to be affecting anything at this level of the error.
• The namespaces are all registered from 87a0a8c317f1e7mode:namespaces . In that file, it defines the PCDM namespace as just "ns" and again has other logic backed into the entry that stops me from just changing the value there. So... essentially the namespace entry loads the PCDM namespace as "ns" while the nodeTypes is looking for it to be "ns009".

Any suggestions from anyone on how to edit either of those two db entries to make them consistent? Thanks!

-Steven

[Embry, Randall Paul]

Recreating my database as UTF-8 has eliminated the NPE I was getting upon startup. Thanks for all the help. I am still seeing messages like these:

ERROR 09:32:04.271 (WebApplicationExceptionMapper) WebApplicationException intercepted by WebApplicationExceptionMapper: HTTP 304 Not Modified

--Randall

[Benjamin Armintor]

Randall, I apologize: That ticket's solution is not released yet (https://jira.duraspace.org/browse/FCREPO-2385), and the released code logs that message at both DEBUG and ERROR level.

Your only solution in 4.7.x for these log messages that I can see (short of running a locally-built version of the http-commons module) is to configure the WebApplicationMapper class not to log except at FATAL (which would basically turn it off).

That sounds more drastic than it is - any exception logging there is just happening at the point of mapping to a HTTP response, and should be logged elsewhere - but I appreciate reluctance to do that, too.

- Ben

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

So some final cleanup notes from my end. Should make more sense than my rambling in the past that was often only the current theory I had on what was going on:

• I realized this morning that changing the data wasn't the best approach so I instead learned how to build Modeshape and started deploying versions of Fedora 4 Commons with more debugging information with it. I eventually got it to boot by having it ignore the ns009:Object (ie. PCDM:Object) load attempt. I incorrectly figured that it might have been some legacy thing since a clean Fedora 4 install had no reference to PCDM for its jcr:nodeTypes... but many records in my system actually do refer to the URI despite me never having implemented it. So I really don't know why my system has this and a clean install does not.
  
  
• From further debugging, it turns out that <root_id>jcr:nodeTypes is special in its use of <root_id>/jcr:system/jcr:nodeTypes/{<pcdm_uri>}Object' to solidify its what namespace it will load for PCDM. In this case, it is hard-coded as ns009 from my system's use in the past when those records where created. Meanwhile, Fedora 4.7.2 changed my <root_id>mode:namespaces content... which means the generated namespace is assigns to PCDM by the time it reaches that entry changed. By adding in some logging, I could determine what namespace it became for me at runtime now and changed the hard-coded values in the two jcr:nodeTypes content successfully. With those two nodes changed to match the dynamic namespace, I can boot with an unmodified Fedora 4.7.x instance (even using the latin1 version of the mySQL database).
  
  
• However, while Fedora 4 is running, I believe the "prod" child of my root node may be messed up yet. In particular, It only lists 100 records as children for it in the UI but perhaps that was some design change to prevent the list from getting to large? Regardless, attempting to run ActiveFedora::reindex_everything doesn't work right. But I can access everything in the system directly and that information seems to be correct... so I can likely use my Solr index at this point to recover fully by this weekend. I had hoped to more directly fix things but about at the limit of what I can do with my limited understanding of how Fedora 4 Commons and Modeshape works and it will likely be better to transfer into a fresh database correctly set to utf-8.

If anyone else runs into this issue in the future and cannot recover like myself, let me know and I can expand upon the above. Should be the last message on my end unless something I didn't find from my data spot check pops up. Special thanks to Ben and everyone else on this thread for all of the help!

Sincerely,

Steven Anderon

[Esmé Cowles]

Steven-

I think the key thing here is that the changes to the namespaces defined in the builtin CND file was a breaking change that should have had a migration. So we should probably make a new release that removes that change so others who are migrating can avoid this problem.

-Esmé

[Andrew Woods]

Hello Steven,

Thank you for your diligence in seeing this issue through and for bringing your findings back to the community list.

Related to your second bullet and Esmé's comment, I am not sure how Fedora 4.7.2 may have changed your namespace declarations. Fedora 4.7.0 was released on October 31, 2016 and the last update to the Fedora file that defines those namespaces was made on October 26, 2016, prior to the 4.7.0 release:

https://github.com/fcrepo4/fcrepo4/commits/master/fcrepo-kernel-modeshape/src/main/resources/fedora-node-types.cnd

Is it possible that you have a local configuration with custom-defined namespaces that may have changed between your 4.7.0 and 4.7.2 deployments?

Related to your third bullet, yes, the HTML view of resources limits the number of returned children to 100. That limit does not exist for RDF representations of resources. If making a GET request for an RDF representation is not returning all of the resource's children, that would be a bug.

Thanks again.

Andrew

> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

Andrew,

The data model did not change from Fedora 4.7.0 and Fedora 4.7.2 so I don't believe there should have been new custom namespaces. Potentially it was a modeshape thing that cleaned itself up of namespaces it can't find on objects? For example, pcdm had gone from ns009 to ns006 in the generated namespace list and that list only goes up to ns007 now. I'm unsure if some of those removed namespaces came from Modshape itself that were changed when that was upgraded further.

I do know that with debug logging on, it writes to that namespaces database object on every start-up for some odd reason. I don't know if previous versions did this so it might be that 4.7.0 just had not overridden what was stored there from different namespaces updating behavior?

Good to know on the HTML view as it wasn't like that in the past. May be a different error with the reindexing of all of the content with my older version of ActiveFedora then.

Thanks!

Sincerely,

Steven Anderson

[dsanfo...@gmail.com]

Steven,

That was a lot of really useful information. I'm going to hold off that move until a new release comes out as Esme suggested, but this is quite interesting.

Daniel Sanford

[Andrew Woods]

Hello All,

As an update, on today's Fedora Tech call [1] we determined that indeed, relevant Fedora configuration changes were included in the release of 4.7.1. We will be creating a 4.7.3 release candidate today with those changes reverted. 

For those of you who are interested in this topic and available to engage in some quick turnaround testing, your participation would be appreciated by all.

Stay tuned.

Andrew

[1] https://wiki.duraspace.org/display/FF/2017-05-18+-+Fedora+Tech+Meeting

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Andrew Woods]

Hello Daniel,

Your feedback in testing the 4.7.3 release candidate would be greatly appreciated:

- https://groups.google.com/d/msg/fedora-tech/y0B9pzjJJmk/XVQEZ1-MAAAJ

- https://github.com/fcrepo4/fcrepo4/releases/tag/fcrepo-4.7.3-RC-1

Thanks,

Andrew

[Andrew Woods]

Hello Steven,

As we prepare to release Fedora 4.7.3 in response to the namespace issue that you helped surface, would you be able to share a link to a GitHub diff or the like that includes your local updates to ModeShape used for debugging, exception catching, etc? It would be helpful to have documented for anyone who runs into a similar issue in the future.

Thanks,

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

Andrew,

I'll try to provide something in the next day or two in regards to what code I changed. Hope Fedora 4.7.3 goes well!

Sincerely,

Steven Anderson

[Andrew Woods]

Hello Anna, Daniel and Steven,

We are ready to ship the 4.7.3 release, but are waiting on any feedback you may have regarding verification of the release candidate (Anna/Daniel) and documentation/code-diffs related to a post-namespace-corruption fix (Steven).

Thanks,

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[dsanfo...@gmail.com]

Andrew,

I've done a pair of test migrations without an errors and put them through a few paces. No namespace issues have cropped up so it looks like at least on our end 4.7.3 solves the problem without any issues.

Dan

[Andrew Woods]

Thank you, Dan.

That is very helpful.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Steven Anderson]

From my end regarding the debugging changes I had done:

1. I recommend setting up the log level to be as verbose as possible. In my case, I added the file from https://drive.google.com/open?id=0B398Bp3fT9A0RTd0MlhERzhSRm8 that sets the log level and outputs to /var/lib/tomcat7/logs/fcrepo.log. This configuration is linked by setting JAVA_OPTS and I did so inside of /etc/default/tomcat7 with the following:
1. JAVA_OPTS="${JAVA_OPTS} -Dlogback.configurationFile=/var/lib/tomcat7/logs/logback.xml"
   
   
2. I added very basic logging to the <modeshape_root>/src/main/java/org/modeshape/jcr/value/basic/NameValueFactory.java file to print out what namespaces in the "ns001" to "ns012" were assigned to. Additionally, I removed a test I could never get to compile and may have made worthwhile changes to the JcrNamespaceRegistry.java I forgot about. These are all in the following commit (which isn't meant to be "clean code"):
1.  https://github.com/scande3/modeshape/commit/fd16152855348263d80d5b1568c6b34b7be0f3ba
   
   
3. I compiled that into a Fedora 4.7.2 WAR at the following location. Note that I can't claim to have tested this exact version as I removed a small bit of hard-coding that would only execute the logging related to the PCDM namespace - but it should give you the output needed to fix the two database entries I mentioned in my previous posts:
1. https://drive.google.com/open?id=0B398Bp3fT9A0Z0ZvYXloeXdyOW8
1. To find the namespace to update in your MySQL, you should be able to do a query of:
   Select * from fcrepo.MODESHAPE_REPOSITORY where CONTENT LIKE '%ns00#%'
   
   
4. For whatever reason, backup and restore on my Fedora 4.7.2 instance doesn't work by default. This is due to a known issue that was supposed to have been fixed and has a sample issue thread located at: https://jira.duraspace.org/browse/FCREPO-2069 . While I get the same error, using the backup-fixer.jar file located there doesn't cut it for me as it resets its hash lookup on every compressed archive metadata file it checks. In my case, 'jcr:system/mode:metadata' duplication was spread among multiple files rather than them all being in the same one. So I have a corrected version of this jar located at the link following this bullet point. It essentially just makes the hash checking for duplicates static and once it has removed the duplicates from all the archive files, it can restore without apparent issue with the same overall triple count and with a complete reindex of solr not finding any errors. 
1. https://drive.google.com/open?id=0B398Bp3fT9A0WTZDREVqWWE0S2M
   
1. Run it with: java -jar backup-fixer.jar <path_to_restore_directory>

Let me know if there is more I need to provide or if this wasn't what you were looking for. 

Sincerely,

Steven Anderson

[Andrew Woods]

Hello Steven,

Thank you for pulling these updates together. 

These notes will likely serve as the starting point if we need to help anyone who has upgraded from 4.7.0 to either 4.7.1 or 4.7.2 and not performed an /fcr:backup before stopping their server.

Also, thanks again for helping surface this issue which as resulted in the forthcoming 4.7.3 patch release.

Regards,

Andrew

--