Null pointer exception - Exception intercepted by WildcardExceptionMapper: null

146 views
Skip to first unread message

Kevin Ford

unread,
Apr 24, 2017, 3:42:36 PM4/24/17
to fedor...@googlegroups.com
Dear All,

We have a problem with our repository, but we don't have enough to file
a bug report.

We recently completed a very large, multi-week ingest of images into
Fedora (4.7.1). By and large it went fine, but we have identified, thus
far, a handful of resources that appear to be corrupt. Performing a GET
on a specific resource results in a 500 Error (a null pointer
exception); performing a GET on two levels of PairTree parent resources
also results in a 500 Error (null pointer too); trying to PUT a new
resource under the same tree results in a 500; and trying to DELETE the
problem resource results in a 500 Error.

This gist shows TRACE level logging for each of the scenarios I just
described:

https://gist.github.com/kefo/154fc27d88fae80b526b333126ff5e8b

We do not know the extent of this issue presently (we're trying to
figure that out) and we cannot reproduce this.

We've not tried to delete a parent-level PairTree resource, in an
attempt to rectify this issue, because we cannot be certain that the
problem resource is the *only* resource under the PairTree path. This
is our production system.

From the TRACE level logs, can anyone shed any light on this or have
ideas for additional debugging? For example, is there a way to increase
Modeshape's logging level?

Yours,
Kevin


Esmé Cowles

unread,
Apr 24, 2017, 4:38:26 PM4/24/17
to fedor...@googlegroups.com
Kevin-

Have you set the cacheSize parameter in your repository.json? Because several of the stacktraces start in org.modeshape.jcr.JcrSession.cachedNode, I wonder if setting cacheSize: 0 might help, at least for the GETs that would let you figure out if the resource was the only member of the PairTree node.

-Esmé
> --
> You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

Andrew Woods

unread,
Apr 24, 2017, 4:50:58 PM4/24/17
to fedor...@googlegroups.com
Hello Kevin,
Additionally, you can get ModeShape logging by adding something like the following to your logback.xml file:
---
<logger name="org.modeshape.jcr" additivity="false" level="TRACE">
    <appender-ref ref="STDOUT"/>
</logger>
---

Andrew

On Mon, Apr 24, 2017 at 4:38 PM, Esmé Cowles <esco...@ticklefish.org> wrote:
Kevin-

Have you set the cacheSize parameter in your repository.json?  Because several of the stacktraces start in org.modeshape.jcr.JcrSession.cachedNode, I wonder if setting cacheSize: 0 might help, at least for the GETs that would let you figure out if the resource was the only member of the PairTree node.

-Esmé

> On Apr 24, 2017, at 12:42 PM, Kevin Ford <kfo...@artic.edu> wrote:
>
> Dear All,
>
> We have a problem with our repository, but we don't have enough to file a bug report.
>
> We recently completed a very large, multi-week ingest of images into Fedora (4.7.1).  By and large it went fine, but we have identified, thus far, a handful of resources that appear to be corrupt.  Performing a GET on a specific resource results in a 500 Error (a null pointer exception); performing a GET on two levels of PairTree parent resources also results in a 500 Error (null pointer too); trying to PUT a new resource under the same tree results in a 500; and trying to DELETE the problem resource results in a 500 Error.
>
> This gist shows TRACE level logging for each of the scenarios I just described:
>
> https://gist.github.com/kefo/154fc27d88fae80b526b333126ff5e8b
>
> We do not know the extent of this issue presently (we're trying to figure that out) and we cannot reproduce this.
>
> We've not tried to delete a parent-level PairTree resource, in an attempt to rectify this issue, because we cannot be certain that the problem resource is the *only* resource under the PairTree path.  This is our production system.
>
> From the TRACE level logs, can anyone shed any light on this or have ideas for additional debugging?  For example, is there a way to increase Modeshape's logging level?
>
> Yours,
> Kevin
>
>
> --
> You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

Kevin Ford

unread,
Apr 24, 2017, 5:15:11 PM4/24/17
to fedor...@googlegroups.com
Thanks, Esmé, for the pointer, but it didn't work. cacheSize must be a
positive integer.

Yours,
Kevin

Kevin Ford

unread,
Apr 24, 2017, 5:16:30 PM4/24/17
to fedor...@googlegroups.com
Thanks, Andrew.

I'm not having a lot of luck with the below. I've tried STDOUT and
having the output write to the daily logfile.

I'm not seeing any information that seems specific to modeshape. Is
there something I can do that would confirm that modeshape is logging at
TRACE level?

Yours,
Kevin



On 4/24/17 15:50, Andrew Woods wrote:
> Hello Kevin,
> Additionally, you can get ModeShape logging by adding something like the
> following to your logback.xml file:
> ---
> <logger name="org.modeshape.jcr" additivity="false" level="TRACE">
> <appender-ref ref="STDOUT"/>
> </logger>
> ---
>
> Andrew
>
> On Mon, Apr 24, 2017 at 4:38 PM, Esmé Cowles <esco...@ticklefish.org
> <mailto:esco...@ticklefish.org>> wrote:
>
> Kevin-
>
> Have you set the cacheSize parameter in your repository.json?
> Because several of the stacktraces start in
> org.modeshape.jcr.JcrSession.cachedNode, I wonder if setting
> cacheSize: 0 might help, at least for the GETs that would let you
> figure out if the resource was the only member of the PairTree node.
>
> -Esmé
>
> > On Apr 24, 2017, at 12:42 PM, Kevin Ford <kfo...@artic.edu
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> > To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> <https://groups.google.com/group/fedora-tech>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.

Andrew Woods

unread,
Apr 24, 2017, 7:23:43 PM4/24/17
to fedor...@googlegroups.com
Hello Kevin,
If your logging configuration is correctly setting ModeShape to TRACE, you would definitely know it.

When you update your logback.xml to include 'org.modeshape.jcr', for example, the servlet container will need to be restarted for the configuration to be picked up... unless you have auto-reloading configured, per:

Andrew


    > To post to this group, send email to fedor...@googlegroups.com
    <mailto:fedora-tech@googlegroups.com>.


    --
    You received this message because you are subscribed to the Google
    Groups "Fedora Tech" group.
    To unsubscribe from this group and stop receiving emails from it,

    To post to this group, send email to fedor...@googlegroups.com
    <mailto:fedora-tech@googlegroups.com>.



--
You received this message because you are subscribed to the Google
Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send

To post to this group, send email to fedor...@googlegroups.com
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.

Kevin Ford

unread,
Apr 25, 2017, 11:11:36 AM4/25/17
to fedor...@googlegroups.com
I got TRACE output to work for modeshape. I've updated the gist with
this newer information:

https://gist.github.com/kefo/154fc27d88fae80b526b333126ff5e8b

Yours,
Kevin

p.s. Pro-tip: Modify the correct logback.xml file. Works better that way.


On 4/24/17 15:50, Andrew Woods wrote:
> Hello Kevin,
> Additionally, you can get ModeShape logging by adding something like the
> following to your logback.xml file:
> ---
> <logger name="org.modeshape.jcr" additivity="false" level="TRACE">
> <appender-ref ref="STDOUT"/>
> </logger>
> ---
>
> Andrew
>
> On Mon, Apr 24, 2017 at 4:38 PM, Esmé Cowles <esco...@ticklefish.org
> <mailto:esco...@ticklefish.org>> wrote:
>
> Kevin-
>
> Have you set the cacheSize parameter in your repository.json?
> Because several of the stacktraces start in
> org.modeshape.jcr.JcrSession.cachedNode, I wonder if setting
> cacheSize: 0 might help, at least for the GETs that would let you
> figure out if the resource was the only member of the PairTree node.
>
> -Esmé
>
> > On Apr 24, 2017, at 12:42 PM, Kevin Ford <kfo...@artic.edu
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> > To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> <https://groups.google.com/group/fedora-tech>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.

Esmé Cowles

unread,
Apr 25, 2017, 1:38:09 PM4/25/17
to Fedora Tech
Kevin-

I believe this is the line where the null pointer is happening:

https://github.com/ModeShape/modeshape/blob/modeshape-5.2.0.Final/modeshape-jcr/src/main/java/org/modeshape/jcr/JcrSession.java#L630

I'm just speculating here, but given the last message logged:

> DEBUG 10:06:29.221 (PoolBase) HikariPool-0 - Reset (readOnly) on connection org.postgresql.jdbc.PgConnection@55a30686

it may be trying to load the item from the db and failing — though I'm not sure why that would be. Can you look in your database and see if the referenced item is in there?

e.g., when the log says:

DEBUG 10:06:29.221 (DefaultStatements) Searching for entry by id 87a0a8c7505d64da9b1758-7d8d-4b21-9eb4-da2bb39fc814 in MODESHAPE_REPOSITORY

Is there actually a row in the modeshape_repository table with the id = '87a0a8c7505d64da9b1758-7d8d-4b21-9eb4-da2bb39fc814'?

-Esmé
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
> To post to this group, send email to fedor...@googlegroups.com.

Kevin Ford

unread,
Apr 25, 2017, 2:32:42 PM4/25/17
to fedor...@googlegroups.com
Hello Esmé,

No. There is nothing in the database with that identifier.

Yours,
Kevin

Esmé Cowles

unread,
Apr 25, 2017, 2:47:39 PM4/25/17
to Fedora Tech
Kevin-

Maybe there was a problem saving that node to the database? Is it possible that Fedora ran out of database connections or something during the ingest? I would hope that would be logged — can you look for that, or share the logs so I could look for it?

-Esmé

Kevin Ford

unread,
Apr 25, 2017, 3:23:33 PM4/25/17
to fedor...@googlegroups.com
Thanks, Esmé, for your assistance.

I can't find anything of note in the logs. The postgres logs just
record vacuuming events; the catalina logs do not reveal anything
specific vis-a-vis these events (in some cases, there is no catalina log
on the days some of these events happened); and the fcrepo log details
little.

I excerpted a few lines from the fcrepo logs pertaining to three of
these events. I included a few extra lines for context - so you can see
that there was little usual happening, except for the ERROR line of
course. The lines with asterisks are connected in so far as HEAD
request is against a URI that now returns a 500 Error. I hesitate,
however, to conclude causation, that the HEAD requests resulted in the
ERROR, but I guess it's possible.

Yours,
Kevin


*INFO 01:41:48.788 (FedoraLdp) HEAD for:
prod/d8/73/f0/7c/d873f07c-d1da-a6be-1c4b-2e8e4ab201cd
INFO 01:41:48.804 (FedoraLdp) GET resource
'prod/nc/58/0x/24/nc580x246/files/04da588e-defe-406a-8ae7-4f4cd7cf2a0c/fcr:metadata'
*ERROR 01:41:48.848 (WildcardExceptionMapper) Exception intercepted by
WildcardExceptionMapper: null

INFO 01:41:48.850 (FedoraLdp) HEAD for:
prod/50/ef/b4/c2/50efb4c2-88b9-435a-a02e-4314e5172b44
INFO 01:41:48.855 (FedoraLdp) GET resource
'prod/50/ef/b4/c2/50efb4c2-88b9-435a-a02e-4314e5172b44'




INFO 17:58:20.207 (FedoraLdp) HEAD for:
prod/c5/e8/ab/7b/c5e8ab7b-6b0f-377c-3fbb-10bfb5c2426f
INFO 17:58:20.237 (FedoraLdp) GET resource
'prod/c5/e8/ab/7b/c5e8ab7b-6b0f-377c-3fbb-10bfb5c2426f'
*INFO 17:58:20.241 (FedoraLdp) HEAD for:
prod/a2/10/78/e0/a21078e0-d671-f889-09ee-010ae0357841
*ERROR 17:58:20.265 (WildcardExceptionMapper) Exception intercepted by
WildcardExceptionMapper: null

INFO 17:58:20.276 (FedoraLdp) HEAD for:
prod/58/f0/0e/ca/58f00eca-c398-02f0-f9bb-6b2b6105c0ef
INFO 17:58:20.292 (FedoraLdp) GET resource
'prod/58/f0/0e/ca/58f00eca-c398-02f0-f9bb-6b2b6105c0ef'




INFO 10:39:09.432 (FedoraLdp) HEAD for:
prod/j9/60/28/45/j96028454/files/ef05be1e-83f7-4039-b335-947da57ec01b/fcr:metadata
INFO 10:39:09.451 (FedoraLdp) GET resource
'prod/j9/60/28/45/j96028454/files/ef05be1e-83f7-4039-b335-947da57ec01b/fcr:metadata'
*INFO 10:39:09.509 (FedoraLdp) HEAD for:
prod/1b/0c/94/05/1b0c9405-5ba3-efc1-e3f3-822b444c1839
*ERROR 10:39:09.531 (WildcardExceptionMapper) Exception intercepted by
WildcardExceptionMapper: null

INFO 10:39:09.593 (FedoraLdp) PATCH for
'prod/j9/60/28/45/j96028454/files/ef05be1e-83f7-4039-b335-947da57ec01b/fcr:metadata'

Andrew Woods

unread,
Apr 25, 2017, 5:30:32 PM4/25/17
to fedor...@googlegroups.com
Hello Kevin,
Although there is not an immediate and obvious solution here...

1. Along the lines of Esmé's log inspection questions, are there any log entries that mention the problematic resource: f44e0e86-b3ab-db54-02b5-1546dc764f0d
2. Were there any system events (restarts, outages, etc) during ingest that you are aware of that may have interfered with the creation of the above resource?
3. How did you discover the problematic resource?
4. Do you know what that resource is, i.e. can you recreate the resource?
5. Do you have backups? system backups, /fcr:backups, messaging-based backups, etc?

Additionally, one way of resolving the issue that may be worth exploring is to perform a JCR backup (excluding binaries), followed by inspecting and removing the problematic resource. This would require a small amount of code, but not a problem if you are interested in such an exploration.

In any case, I have added this issue to Thursday's tech meeting agenda:

Regards,
Andrew



  > To post to this group, send email to fedor...@googlegroups.com

  > Visit this group at https://groups.google.com/group/fedora-tech
  <https://groups.google.com/group/fedora-tech>.
  > For more options, visit https://groups.google.com/d/optout
  <https://groups.google.com/d/optout>.

  --
  You received this message because you are subscribed to the Google
  Groups "Fedora Tech" group.
  To unsubscribe from this group and stop receiving emails from it,

  To post to this group, send email to fedor...@googlegroups.com

  Visit this group at https://groups.google.com/group/fedora-tech
  <https://groups.google.com/group/fedora-tech>.
  For more options, visit https://groups.google.com/d/optout
  <https://groups.google.com/d/optout>.


--
You received this message because you are subscribed to the Google
Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send

To post to this group, send email to fedor...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

Kevin Ford

unread,
Apr 26, 2017, 11:44:03 AM4/26/17
to fedor...@googlegroups.com
Hi Andrew,

Comments in-line.

One thing to add. There is some kind of pattern related to the NPE.
Every URI conforms to this pattern:

/fcrepo/rest/prod/ff/2a/3d/9b/ff2a3d9b-4eb2-180f-9cbf-2811cb8175ef

Every instance of this problem that we have seen conforms to this HTTP
Status Code response pattern:

GET /fcrepo/rest/prod/ff/2a/3d/9b/ff2a3d9b-4eb2-180f-9cbf-2811cb8175ef = 500
GET /fcrepo/rest/prod/ff/2a/3d/9b/ = 500
GET /fcrepo/rest/prod/ff/2a/3d/ = 500
GET /fcrepo/rest/prod/ff/2a/ = 200
GET /fcrepo/rest/prod/ff/ = 200
GET /fcrepo/rest/prod/ = 200

Point is this: the system returns a 500 when requesting the specific
resource, its parent, and its grandparent (for the lack of a better term).

After determining, with reasonable confidence, that the ONLY resource under

/fcrepo/rest/prod/ff/2a/

was

/fcrepo/rest/prod/ff/2a/3d/9b/ff2a3d9b-4eb2-180f-9cbf-2811cb8175ef

we tried to DELETE

/fcrepo/rest/prod/ff/2a/

This too failed to DELETE the resource (and its children), returning the
same 500 NPE Error.

Yours,
Kevin



On 4/25/17 16:30, Andrew Woods wrote:
> Hello Kevin,
> Although there is not an immediate and obvious solution here...
>
> 1. Along the lines of Esmé's log inspection questions, are there any log
> entries that mention the problematic resource:
> f44e0e86-b3ab-db54-02b5-1546dc764f0d
-- Yes, but the extent of these entries are represented below. There is
a HEAD request issued against the URI followed shortly by the ERROR.
Those requests come from Sufia, so I imagine the initial HEAD request
returned a 404 and the ERROR is the result of a PUT, but that is
speculation.

> 2. Were there any system events (restarts, outages, etc) during ingest
> that you are aware of that may have interfered with the creation of the
> above resource?
-- No. And I've looked at this question with a few other problem
resources in mind. While we had to stop and restart the system on
occasion, I have determined that some of the problem resources hit this
ERROR on days when the system was not touched, at least by us. If there
was some kind of network outage or hiccup, I can't say. It's unlikely,
but possible.

> 3. How did you discover the problematic resource?
-- We performed a very large, multi-week load. There were some FAILURES
and I was reviewing those.

> 4. Do you know what that resource is, i.e. can you recreate the resource?
-- We know what the resource is. When it comes to recreating the
resource the answer is "yes and no." "Yes" we *could* recreate the
resource, but its identifier is a hash of its existing identifier (from
a separate system) so "no" because we'd generate the same URI and we
cannot write to that URI presently (or DELETE, etc).

> 5. Do you have backups? system backups, /fcr:backups, messaging-based
> backups, etc?
-- Yes and no. We have filesystem backups, but since this was a
always-running, multi-week load we did not capture any fcr:backups
during. NO messaging-based ones either.

>
> Additionally, one way of resolving the issue that may be worth exploring
> is to perform a JCR backup (excluding binaries), followed by inspecting
> and removing the problematic resource. This would require a small amount
> of code, but not a problem if you are interested in such an exploration.
-- THis might be something to explore.
> <kfo...@artic.edu <mailto:kfo...@artic.edu>> wrote:
>
> I got TRACE output to work for modeshape. I've
> updated the gist with this newer information:
>
> https://gist.github.com/kefo/154fc27d88fae80b526b333126ff5e8b
> <https://gist.github.com/kefo/154fc27d88fae80b526b333126ff5e8b>
>
> Yours,
> Kevin
>
> p.s. Pro-tip: Modify the correct logback.xml
> file. Works better that way.
>
>
> On 4/24/17 15:50, Andrew Woods wrote:
>
> Hello Kevin,
> Additionally, you can get ModeShape logging by
> adding something like the
> following to your logback.xml file:
> ---
> <logger name="org.modeshape.jcr"
> additivity="false" level="TRACE">
> <appender-ref ref="STDOUT"/>
> </logger>
> ---
>
> Andrew
>
> On Mon, Apr 24, 2017 at 4:38 PM, Esmé Cowles
> <esco...@ticklefish.org
> <mailto:esco...@ticklefish.org>
> <mailto:esco...@ticklefish.org
> <mailto:esco...@ticklefish.org>>> wrote:
>
> Kevin-
>
> Have you set the cacheSize parameter in your
> repository.json?
> Because several of the stacktraces start in
> org.modeshape.jcr.JcrSession.cachedNode, I
> wonder if setting
> cacheSize: 0 might help, at least for the GETs
> that would let you
> figure out if the resource was the only member
> of the PairTree node.
>
> -Esmé
>
> > On Apr 24, 2017, at 12:42 PM, Kevin Ford
> <kfo...@artic.edu <mailto:kfo...@artic.edu>
> <mailto:kfo...@artic.edu
> fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>
>
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> > To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> > Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> > For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
> --
> You received this message because you are
> subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop
> receiving emails from it,
> send an email to
> fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>
>
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> You received this message because you are
> subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop
> receiving emails from it, send
> an email to
> fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>
> <mailto:fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed
> to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving
> emails from it, send an email to
> fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.

Andrew Woods

unread,
May 2, 2017, 11:02:10 AM5/2/17
to fedor...@googlegroups.com
Hello Kevin,
When you have a moment, it would be helpful to know if you can perform a "backup" of your repository (or a mirror thereof) with the following utility which performs a direct ModeShape backup, excluding binaries:

Here is the usage documentation:

If that "backup" is successful, we should look into removing the problematic resources from there.

Thanks,
Andrew

                        fedora-tech+unsubscribe@googlegroups.com
                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>

                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com
                        <mailto:fedora-tech%252Bunsubscr...@googlegroups.com>>.

                          > To post to this group, send email to
                        fedor...@googlegroups.com
                        <mailto:fedora-tech@googlegroups.com>
                          <mailto:fedora-tech@googlegroups.com
                        <mailto:fedora-tech@googlegroups.com>>.

                          > Visit this group at
                        https://groups.google.com/group/fedora-tech
                        <https://groups.google.com/group/fedora-tech>
                          <https://groups.google.com/group/fedora-tech
                        <https://groups.google.com/group/fedora-tech>>.
                          > For more options, visit
                        https://groups.google.com/d/optout
                        <https://groups.google.com/d/optout>
                          <https://groups.google.com/d/optout
                        <https://groups.google.com/d/optout>>.

                          --
                          You received this message because you are
                        subscribed to the Google
                          Groups "Fedora Tech" group.
                          To unsubscribe from this group and stop
                        receiving emails from it,
                          send an email to
                        fedora-tech+unsubscribe@googlegroups.com
                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>

                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com
                        <mailto:fedora-tech%252Bunsubscr...@googlegroups.com>>.

                          To post to this group, send email to
                        fedor...@googlegroups.com
                        <mailto:fedora-tech@googlegroups.com>
                          <mailto:fedora-tech@googlegroups.com
                        <mailto:fedora-tech@googlegroups.com>>.

                          Visit this group at
                        https://groups.google.com/group/fedora-tech
                        <https://groups.google.com/group/fedora-tech>
                          <https://groups.google.com/group/fedora-tech
                        <https://groups.google.com/group/fedora-tech>>.
                          For more options, visit
                        https://groups.google.com/d/optout
                        <https://groups.google.com/d/optout>
                          <https://groups.google.com/d/optout
                        <https://groups.google.com/d/optout>>.


                        --
                        You received this message because you are
                        subscribed to the Google
                        Groups "Fedora Tech" group.
                        To unsubscribe from this group and stop
                        receiving emails from it, send
                        an email to
                        fedora-tech+unsubscribe@googlegroups.com
                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>
                        <mailto:fedora-tech+unsubscribe...@googlegroups.com
                        <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>>.

                        To post to this group, send email to
                        fedor...@googlegroups.com
                        <mailto:fedora-tech@googlegroups.com>
                        <mailto:fedora-tech@googlegroups.com

                        <mailto:fedora-tech@googlegroups.com>>.

                        Visit this group at
                        https://groups.google.com/group/fedora-tech
                        <https://groups.google.com/group/fedora-tech>.
                        For more options, visit
                        https://groups.google.com/d/optout
                        <https://groups.google.com/d/optout>.


                    --
                    You received this message because you are subscribed
                    to the Google Groups "Fedora Tech" group.
                    To unsubscribe from this group and stop receiving
                    emails from it, send an email to
                    fedora-tech+unsubscribe@googlegroups.com
                    <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>.

                    To post to this group, send email to
                    fedor...@googlegroups.com
                    <mailto:fedora-tech@googlegroups.com>.

                    Visit this group at
                    https://groups.google.com/group/fedora-tech
                    <https://groups.google.com/group/fedora-tech>.
                    For more options, visit
                    https://groups.google.com/d/optout
                    <https://groups.google.com/d/optout>.



            --
            You received this message because you are subscribed to the
            Google Groups "Fedora Tech" group.
            To unsubscribe from this group and stop receiving emails
            from it, send an email to
            fedora-tech+unsubscribe@googlegroups.com
            <mailto:fedora-tech%2Bunsubscri...@googlegroups.com>.

            To post to this group, send email to
            fedor...@googlegroups.com
            <mailto:fedora-tech@googlegroups.com>.

            Visit this group at
            https://groups.google.com/group/fedora-tech
            <https://groups.google.com/group/fedora-tech>.
            For more options, visit https://groups.google.com/d/optout
            <https://groups.google.com/d/optout>.



    --
    You received this message because you are subscribed to the Google
    Groups "Fedora Tech" group.
    To unsubscribe from this group and stop receiving emails from it,

    To post to this group, send email to fedor...@googlegroups.com
    <mailto:fedora-tech@googlegroups.com>.

    Visit this group at https://groups.google.com/group/fedora-tech
    <https://groups.google.com/group/fedora-tech>.
    For more options, visit https://groups.google.com/d/optout
    <https://groups.google.com/d/optout>.


--
You received this message because you are subscribed to the Google
Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send

To post to this group, send email to fedor...@googlegroups.com
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.

Kevin Ford

unread,
May 4, 2017, 3:32:16 PM5/4/17
to fedor...@googlegroups.com
Thanks, Andrew, for this. For a few reasons I've not had time to get to
this but I will as soon as I can.

Yours,
Kevin


On 5/2/17 10:02, Andrew Woods wrote:
> Hello Kevin,
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>
>
> <mailto:fedora-tech%252Buns...@googlegroups.com
> <mailto:fedora-tech%25252Bun...@googlegroups.com>>>.
> > To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>
>
> <mailto:fedora-tech%252Buns...@googlegroups.com
> <mailto:fedora-tech%25252Bun...@googlegroups.com>>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> <mailto:fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>
>
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> Visit this group at
>
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> You received this message because you are
> subscribed
> to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop
> receiving
> emails from it, send an email to
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed
> to the
> Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the
> Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails
> from it,
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.

Kevin Ford

unread,
May 9, 2017, 4:58:33 PM5/9/17
to fedor...@googlegroups.com
Hi Andrew,

I was able to generate the backup this weekend and I spent yesterday
evaluating it on my side.

I had identified 19 problem paths based on pinging paths that we create
(i.e. URIs we *know* about). Evaluating the backup, I can see that this
issue is more widespread than previously understood.

After some massaging, I loaded the backup data into MongoDb. There are
roughly 20 million documents!

After figuring out how modeshape does its thing, I was able to write a
script that looks for individuals with non-existent children. Using
that information, I can work my way to a full fedora URI. I then tested
them.

I found 2,021 affected paths. I had written before that they all seemed
to conform to a pattern whereby the 500 NPE would occur at a third-level
resource, such as:

/prod/a/b/c/ = 500

Although that was all I had seen, I can amend my report with the
following counts:

/prod/a/b/c/: 651 NPE
/prod/a/b/c/d/: 688 NPE
/prod/a/b/c/d/e/: 682 NPE

Anyways, follow the below link to a representation of a modeshape
resource that contains a non-existent child, and a few additional notes:

https://gist.github.com/kefo/85ada964060b4398d78ae9f35635d8c3#file-prod-8c-88

If my understanding of this is correct (and my knowledge of Modeshape is
pretty, pretty thin), it seems likely to remove the non-existent child
references at the modeshape layer, especially if I can provide a list of
modeshape identifiers.

Moving on to a different matter but on this same topic...

In a past tech call meeting I was asked for a log snippet showing the
traffic at the time one of these errors. It is included as part of the
gist, but the actual (initial) 500 error can be seen here:

https://gist.github.com/kefo/85ada964060b4398d78ae9f35635d8c3#file-access_log_sample-log-L977

The snippet represents 10 seconds before the 500 and 10 seconds
afterwards.

Note - that appears to be a 500 as a result of performing a HEAD request
*before* issuing any PUT. Is it possible that modeshape is writing
something at the time of a HEAD request?

Let me know what else I can share.

Yours,
Kevin




On 5/2/17 10:02, Andrew Woods wrote:
> Hello Kevin,
> <mailto:fedora-tech%25252Bun...@googlegroups.com>>>.
> > To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> <mailto:fedora-tech%25252Bun...@googlegroups.com>>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> <mailto:fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>
>
> <mailto:fedora-tech%2Bunsu...@googlegroups.com
> <mailto:fedora-tech%252Buns...@googlegroups.com>>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com>>>.
> Visit this group at
>
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> You received this message because you are
> subscribed
> to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop
> receiving
> emails from it, send an email to
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed
> to the
> Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
>
> --
> You received this message because you are subscribed to the
> Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails
> from it,
> <mailto:fedora-tech%252Buns...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at
> https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>
> <https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to
> fedor...@googlegroups.com <mailto:fedor...@googlegroups.com>
> <mailto:fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>>.
> Visit this group at https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech%2Bunsu...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/fedora-tech
> <https://groups.google.com/group/fedora-tech>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
Reply all
Reply to author
Forward
0 new messages