discrepancies in retrievability of individual mementos vs. their timemaps

[Nicholas Taylor]

Hello All,

I've noticed that for some resources indexed in our OpenWayback instance, individual mementos can be accessed, e.g., https://swap.stanford.edu/20170628171729/https://www.waseda.jp/fcom/soc/ but trying to access the timemap for all versions of the same memento gives "Resource not in archive": https://swap.stanford.edu/*/https://www.waseda.jp/fcom/soc/.

Any suggestions on where to troubleshoot would be welcome.

Thanks!

~Nicholas

[Sawood Alam]

Here are my quick findings:

Memento (works):

$ curl -I https://swap.stanford.edu/20170628171729/https://www.waseda.jp/fcom/soc/

HTTP/1.1 200 OK

TimeGate (works):

$ curl -I https://swap.stanford.edu/https://www.waseda.jp/fcom/soc/

HTTP/1.1 302 Found

Location: https://swap.stanford.edu/20170628171729/https://www.waseda.jp/fcom/soc/

TimeMap (fails):

$ curl -I https://swap.stanford.edu/timemap/link/https://www.waseda.jp/fcom/soc/

HTTP/1.1 404 Not Found

Calendar (fails):

$ curl -I https://swap.stanford.edu/*/https://www.waseda.jp/fcom/soc/

HTTP/1.1 404 Not Found

With this experiment in hand, I would think there might be some issue in how binary search is being performed in loaded CDX files when searching for a specific line vs. a range of lines. I would perhaps guess there might be some issue in how CDX files are sorted. It might be worth trying to sort them again with "LC_ALL=C" environment variable set and see if any of the CDX files are different from what they should be. Alternatively, I would manually locate the WARC file(s) with the help of the CDX file(s) that contain record of this URI. Then I would index those WARC files again and run a test instance of the replay system (with as much of the settings replicated from the production as possible) just on that subset to see how it behaves.

Best,

--

Sawood Alam

Department of Computer Science

Old Dominion University

Norfolk VA 23529

--
You received this message because you are subscribed to the Google Groups "openwayback-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwayback-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/openwayback-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/openwayback-dev/39c0633c-ed6e-43b9-96c6-b20a12202284%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicholas Taylor]

Thanks for the troubleshooting tips, Sawood. It does seem like a
content indexing issue.

I double-checked our indexing workflows for the environment variable
being set for the sort to work properly, and it's there. Will try your
other strategies and report back.

~Nicholas

[Andy Jackson]

Weirdly this works:

https://swap.stanford.edu/2018*/https://www.waseda.jp/fcom/soc/

Are you accessing OWB via a proxy server? Is it mucking about with the URLs? If you go directly to the back-end service does that work okay?

Eg http://OWB:8080/*/https...

Cheers,

Andy

--

Sent via a tiny keyboard, so apologies for any tipos.

Cheers,

Andy

--

Sent via a tiny keyboard, so apologies for any tipos.

--
You received this message because you are subscribed to the Google Groups "openwayback-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwayback-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/openwayback-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/openwayback-dev/CAAXk%2BeXkDRfXRr%3DKi%2BgqEmBoObnUs-gi_V9QPwjg%3DK%3DC3hfUew%40mail.gmail.com.

[Andy Jackson]

Oh no this is much weirder….

Note that this works:

https://swap.stanford.edu/201706*/https://www.waseda.jp/fcom/soc/

And this doesn’t:

https://swap.stanford.edu/201705*/https://www.waseda.jp/fcom/soc/

i.e. target dates that are since the first date are not working!

Hm, actually this reminds me of an oddity we came across. Are you using RemoteResourceIndex? If so:

https://github.com/ukwa/waybacks/issues/3#issuecomment-337565793

https://github.com/ukwa/waybacks/commit/0820f3827a07e006db0d6cd91c69c3636576855f

If not, it’s possible that the logic in your BubbleCalendar.jsp has clipped things to the most recent year. You could try comparing if with the current version.

HTH,

Andy

To view this discussion on the web visit https://groups.google.com/d/msgid/openwayback-dev/8967A1EB-95F7-4B29-A751-DA72C028BC2E%40gmail.com.