*URGENT* testing and debugging required

[Gabriel Bodard]

Dear EFES Users,

You'll all be aware of the long-standing issue with EFES requiring a legacy version of Java JDK, which sometimes caused fatal errors for users who were unable to get it working (even after following the Kiln recommendations for installing Java).

After some recently activity in the Cocoon open source community, Jamie has updated several of the jar files in the EFES repo so that everything should work with more recent versions of JDK. He reports that these work for him (with Java 11), and has pushed a branch of EFES containing these upgrades to <https://github.com/EpiDoc/EFES/tree/commons-lang-patch>.

Others (including myself) have however reported that this patch causes fatal errors in EFES for them (in Java 16, 11 or 8)—in particular, EpiDoc files do not render using the Reference XSLT. We urgently need more testing and feedback on this issue. If you merge the commons-lang-patch into your fork of EFES, you should get the updates you need; or if you prefer not to risk that, a new export or fork of this branch should be sufficient to test it on your computer, especially if you are able to install newer versions of Java. Any report of what does or doesn't work for you would be useful for Jamie to try to debug this.

(Any help in actual debugging would also be useful. There may be specific templates in the XSLT that trigger the bug, as speculated by Irene.)

Jamie, Irene and others may be better able to help explain some of the technical points I have stumbled on above. If anyone has any questions or other suggestions, please reply onlist!

Many thanks in advance,

Gabby

==
Dr Gabriel BODARD (he/him)
Reader in Digital Classics

Institute of Classical Studies / Digital Humanities Research Hub

University of London
Senate House
Malet Street
London WC1E 7HU
 
E: Gabriel...@sas.ac.uk
T: +44 (0)20 78628752 

 
Especially at the moment, I may email at odd hours of the day and night/days of the week. I do not ever expect a reply outside of your working hours.

[Irene Vagionakis]

I have tried with JDK 11 and:

- without the patch I am having troubles with the handling of images, but not with Sesame (which was an issue with JDK 16)

- with the patch the handling of the images is fine and apparently also the relations with Sesame, but when opening some inscription pages (among which almost all the sample files included in EFES repo), I get an EDF:f-wwrap error message, which looks similar to the one related to symbols https://github.com/EpiDoc/EFES/issues/57. This error message, by the way, seems to appear only in the inscriptions that have at least a <supplied>, whose stylesheet teisupplied.xsl calls the EDF:f-wwrap function declared in teig.xsl.

[Jamie Norrish]

On Tue, 2021-06-29 at 11:19 -0700, Irene Vagionakis wrote:


> - with the patch the handling of the images is fine and apparently
> also the relations with Sesame

Great.

> but when opening some inscription pages (among which almost all the
> sample files included in EFES repo), I get an EDF:f-wwrap
> error message, which looks similar to the one related to symbols
> https://github.com/EpiDoc/EFES/issues/57.

From the discussion of that issue, it sounds as though the recent merge
of the glyph-branch into the mainline of the Stylesheets repo might
have an effect. Are you able to retry after updating the kiln/epidoc
directory with the latest version of the Epidoc XSLT?

Jamie

[Irene Vagionakis]

I have tried again with JDK 11 with the most recent stylesheets, but with the patch I got the same error messages (and without the patch the EDF:f-wwrap is gone, but the images are not displayed and there are some glitches in the display of the interpretive/diplomatic edition tabs, plus the sample file iospe-5.238 is not displayed because of how the symbols are encoded in it - but all this has to do with bugs in the new stylesheets; see https://lsv.uky.edu/scripts/wa.exe?A2=MARKUP;d2e8367b.2105&S=).

---

Da: efes-...@googlegroups.com <efes-...@googlegroups.com> per conto di Jamie Norrish <ja...@artefact.org.nz>
Inviato: giovedì 1 luglio 2021 05:13
A: efes-...@googlegroups.com <efes-...@googlegroups.com>
Oggetto: Re: [efes-users] Re: *URGENT* testing and debugging required

 

--
You received this message because you are subscribed to a topic in the Google Groups "EFES users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/efes-users/-xt_ZdAnDXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to efes-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/efes-users/fcea939efa46c646a3a269983e79150b6315cac4.camel%40artefact.org.nz.

[Jamie Norrish]

Hi all,

I have updated the (now poorly named) commons-lang-patch branch with a
newer RDF server (RDF4J version 2, as used in Kiln). I've confirmed
that I can run it with JDK 11 and it seems to behave correctly, but of
course I haven't been running into any of these problems so that may
mean nothing.

If anyone can test it and report back, that would be great. Note that
there is no migration from the old RDF server to the new - you'll have
to recreate the RDF repository as per the instructions and reharvest
everything.

I also bumped the version of Jetty (the webserver used for running EFES
locally) to version 9.2, but that shouldn't have an effect on any of
the problems people have been seeing.

Jamie

[Gabriel Bodard]

Thanks, Jamie. For what it's worth, I'm still seeing the same problems with your new version with Java 1.8 (haven't had time to update that for testing yet). Viz: all example inscriptions in the epidoc folder except those with no `<supplied>` elements throw a "Failed to process pipeline" error.

Is anyone else seeing improvement with this new version of the branch?

Thanks,

Gabby

==
Dr Gabriel BODARD (he/him)
Reader in Digital Classics

Institute of Classical Studies / Digital Humanities Research Hub

University of London
Senate House
Malet Street
London WC1E 7HU
 
E: Gabriel...@sas.ac.uk
T: +44 (0)20 78628752 

 
Especially at the moment, I may email at odd hours of the day and night/days of the week. I do not ever expect a reply outside of your working hours.

---

From: efes-...@googlegroups.com <efes-...@googlegroups.com> on behalf of Jamie Norrish <ja...@artefact.org.nz>
Sent: 16 July 2021 04:25
To: efes-...@googlegroups.com <efes-...@googlegroups.com>
Subject: Re: R: [efes-users] Re: *URGENT* testing and debugging required

 

--
You received this message because you are subscribed to the Google Groups "EFES users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to efes-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/efes-users/e65650d44cb1a07da673c1031a2a2009e7288baf.camel%40artefact.org.nz.

[Jamie Norrish]

On Fri, 2021-07-16 at 14:09 +0000, Gabriel Bodard wrote:


> For what it's worth, I'm still seeing the same problems with your new
> version with Java 1.8 (haven't had time to update that for testing
> yet). Viz: all example inscriptions in the epidoc folder except those
> with no `<supplied>` elements throw a "Failed to process pipeline"
> error.

Yeah, this latest change would only potentially address the problem
with later versions of Java interacting with the RDF store. And, more
broadly, it brings EFES more in line with Kiln. It does seem that I
will need to upgrade RDF4J to version 3, however.

The EDF:f-wwrap and potentially related XSL troubles are a different
matter, and another one that I'm not seeing with JDK 8 (1.8) or JDK 11
on igcyr002200.xml, which has a supplied element.

Jamie

[Irene Vagionakis]

Same for me: I am getting the same error (with JDK 8, 11 and 16).

---

Da: efes-...@googlegroups.com <efes-...@googlegroups.com> per conto di Gabriel Bodard <gabriel...@sas.ac.uk>
Inviato: venerdì 16 luglio 2021 16:09
A: efes-...@googlegroups.com <efes-...@googlegroups.com>
Oggetto: Re: R: [efes-users] Re: *URGENT* testing and debugging required

 

To view this discussion on the web visit https://groups.google.com/d/msgid/efes-users/LO0P123MB41403FA309F227A4985D80CAC1119%40LO0P123MB4140.GBRP123.PROD.OUTLOOK.COM.

[Jamie Norrish]

To be clear, is the error you (Gabby and Irene) are seeing the function
error reported in https://github.com/EpiDoc/EFES/issues/57 ? And how
does this relate to the errors mentioned at
https://lsv.uky.edu/scripts/wa.exe?A2=MARKUP;d2e8367b.2105&S= that seem
to be in reference to a version of the XSLT that is not committed to
EFES? Is the problem seen in EFES using the XSLT from the EFES
repository, or the latest XSLT from the Epidoc stylesheets repository?
Or does it occur with both?

Does the XSLT (either or both sets) work correctly when applied outside
Cocoon on the same Epidoc files?

Jamie

[Gabriel Bodard]

Apologies, for some reason this reply never made it to my inbox, so I only by chance saw it now on the web version of the forum.

I think the problem does look like the one in issue 57, yes. This problem however does not occur when running stylesheets with Oxygen/Saxonica, which suggests that it is specific to the version of Java and/or Saxon that are being used in EFES?

The error occurs for me whether I use the version of the stylesheets currently bundled with EFES, or if I copy the latest version (in which another problem with XSLT 2 vs 3 has been fixed).

G

[Gabriel Bodard]

Dear Jamie, Irene and others,

I think I have made the current small problem with my EFES go away, just by moving the function EDF:f-wwrap() from teig.xsl into the functions.xsl stylesheet where it belongs. (If you temporarily copy the stylesheets from <https://github.com/EpiDoc/Stylesheets.git> into the "commons-lang-patch" branch it should disappear for everyone else too.) Apologies if I've broken anything else in the process, either in EFES or the EpiDoc Stylesheets.

I still need to test the rest of the behaviour of EFES, repository, etc., with different versions of JDK, but I hope we've got this distracting issue out of the way and can focus on larger scale problems with EFES under recent versions of Java.

[Jamie Norrish]

On Wed, 2021-07-21 at 16:58 +0000, Gabriel Bodard wrote:

> I think I have made the current small problem with my EFES go away,
> just by moving the function EDF:f-wwrap() from teig.xsl into the
> functions.xsl stylesheet where it belongs. (If you temporarily copy
> the stylesheets from <https://github.com/EpiDoc/Stylesheets.git> into
> the "commons-lang-patch" branch it should disappear for everyone else
> too.) Apologies if I've broken anything else in the process, either
> in EFES or the EpiDoc Stylesheets.

I have updated the commons-lang-patch branch to ease testing this and
other things. It now includes:

* Latest EpiDoc XSLT (including the change mentioned above)
* Version 10 of Saxon HE XSLT engine
* RDF server is now version 2 of RDF4J
* Latest Cocoon 2.1.13 with patch to use newer commons-lang JAR

An update to RDF4J 3 is a possible next step if problems there are
still occurring.

Jamie

[Tom Elliott]

Dear colleagues:

I'm new to EFES. Gabby asked me if I could do some troubleshooting and provide feedback as I'm a Mac user (currently still on Mojave). I've just successfully installed EFES (first from the master and then from the commons-lang-patch branches), and successfully run build.sh for each (transcripts below). My setup is likely not typical: default Java on my machine is openjdk 14.0.2 2020-07-14.

If someone can post specific steps for testing a particular problem, I'm happy to try them and report any errors I see as I have time around other tasks.

Tom

New build with "master":

$ git clone g...@github.com:EpiDoc/EFES.git
Cloning into 'EFES'...
remote: Enumerating objects: 11403, done.
remote: Counting objects: 100% (267/267), done.
remote: Compressing objects: 100% (189/189), done.
remote: Total 11403 (delta 126), reused 168 (delta 77), pack-reused 11136
Receiving objects: 100% (11403/11403), 243.91 MiB | 12.50 MiB/s, done.
Resolving deltas: 100% (6940/6940), done.
$ cd EFES
$ ./build.sh
Buildfile: /Users/paregorios/Documents/files/E/EFES/local.build.xml
Development server is running at http://127.0.0.1:9999
Quit the server with CONTROL-C.
2021-07-22 06:49:32.444:INFO:/openrdf-sesame:main: Initializing Spring FrameworkServlet 'openrdf-http-server'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$2 (file:/Users/paregorios/Documents/files/E/EFES/webapps/openrdf-sesame/WEB-INF/lib/cglib-2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
^C
2021-07-22 06:53:52.330:INFO:/openrdf-sesame:Thread-1: Destroying Spring FrameworkServlet 'openrdf-http-server'

New build with “commons-lang-patch”

⌘ cd ..
$ rm -rf EFES
$ git clone g...@github.com:EpiDoc/EFES.git
Cloning into 'EFES'...
remote: Enumerating objects: 11403, done.
remote: Counting objects: 100% (267/267), done.
remote: Compressing objects: 100% (189/189), done.
remote: Total 11403 (delta 126), reused 168 (delta 77), pack-reused 11136
Receiving objects: 100% (11403/11403), 243.91 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (6940/6940), done.
$ cd EFES
$ git fetch origin commons-lang-patch
From github.com:EpiDoc/EFES
 * branch            commons-lang-patch -> FETCH_HEAD
$ git checkout commons-lang-patch
Branch 'commons-lang-patch' set up to track remote branch 'commons-lang-patch' from 'origin'.
Switched to a new branch 'commons-lang-patch'
$ ./build.sh
Buildfile: /Users/paregorios/Documents/files/E/EFES/local.build.xml
Development server is running at http://127.0.0.1:9999
Quit the server with CONTROL-C.
2021-07-22 06:58:30.391:INFO::main: Logging initialized @770ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/paregorios/Documents/files/E/EFES/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/paregorios/Documents/files/E/EFES/webapps/ROOT/WEB-INF/lib/slf4j-nop-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:WARN No appenders could be found for logger (net.sf.ehcache.CacheManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2021-07-22 06:59:23.301:INFO:/rdf4j-server:main: No Spring WebApplicationInitializer types detected on classpath
2021-07-22 06:59:23.560:INFO:/rdf4j-server:main: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: status display disabled
2021-07-22 06:59:23.577:INFO:/rdf4j-server:main: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: loaded (conf ok)
2021-07-22 06:59:23.724:INFO:/rdf4j-server:main: Initializing Spring FrameworkServlet 'rdf4j-http-server'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$2 (file:/Users/paregorios/Documents/files/E/EFES/webapps/rdf4j-server/WEB-INF/lib/spring-core-4.2.1.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Warning at xsl:stylesheet on line 1 column 183 of text-file-list-to-html.xsl?pipelinehash=2042831463259620319:
   Stylesheet module cocoon://_internal/template/xsl/stylesheets/text-file-list-to-html.xsl
  is included or imported more than once. This is permitted, but may lead to errors or
  unexpected behavior
^C
2021-07-22 07:12:59.370:INFO:/rdf4j-server:Thread-0: Destroying Spring FrameworkServlet 'rdf4j-http-server'
2021-07-22 07:12:59.372:INFO:/rdf4j-server:Thread-0: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called

Tom Elliott, Ph.D.
Associate Director for Digital Programs
Senior Research Scholar
Institute for the Study of the Ancient World, NYU
https://paregorios.org/about
https://isaw.nyu.edu/people/staff/tom-elliott

tom.e...@nyu.edu

pronouns: he/they, him/them

On 21 Jul 2021, at 17:21, Jamie Norrish wrote:

On Wed, 2021-07-21 at 16:58 +0000, Gabriel Bodard wrote:

I think I have made the current small problem with my EFES go away,
just by moving the function EDF:f-wwrap() from teig.xsl into the
functions.xsl stylesheet where it belongs. (If you temporarily copy

the stylesheets from <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EpiDoc_Stylesheets.git&d=DwIBaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=0nSVSkUoUKuuCBq1Egoepw&m=Fov7KnoFNi1m-srwiLbHWchGeNVgIWLvJwxLV_lr7AI&s=9__99_zrNEvL1biMkioL1EXy4yAlY_-ILccXTOQ-R28&e= > into

the "commons-lang-patch" branch it should disappear for everyone else
too.) Apologies if I've broken anything else in the process, either
in EFES or the EpiDoc Stylesheets.

I have updated the commons-lang-patch branch to ease testing this and
other things. It now includes:

* Latest EpiDoc XSLT (including the change mentioned above)
* Version 10 of Saxon HE XSLT engine
* RDF server is now version 2 of RDF4J
* Latest Cocoon 2.1.13 with patch to use newer commons-lang JAR

An update to RDF4J 3 is a possible next step if problems there are
still occurring.

Jamie

--
You received this message because you are subscribed to the Google Groups "EFES users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to efes-users+...@googlegroups.com.

To view this discussion on the web visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_msgid_efes-2Dusers_eed953a988b16986e3deef419c32d7d4969c32cb.camel-2540artefact.org.nz&d=DwIBaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=0nSVSkUoUKuuCBq1Egoepw&m=Fov7KnoFNi1m-srwiLbHWchGeNVgIWLvJwxLV_lr7AI&s=ipYLgcbk5dbN9-zs9-KkqiRxc0VSZBOswCMcQpsLgyo&e= .

[Jamie Norrish]

On Thu, 2021-07-22 at 07:16 -0500, Tom Elliott wrote:

> If someone can post specific steps for testing a particular problem,
> I'm happy to try them and report any errors I see as I have time
> around other tasks.

Thank you for this Tom!

The two things I'm aware of that I know how to trigger are:

1. The f-wwrap problem, which when present should cause an error on the
page http://localhost:9999/en/inscriptions/igcyr002200.html

2. The RDF server not running. If encountered I believe this prevents
the server from running at all (and so you are not encountering it),
but this can be further checked by creating a repository and harvesting
information into it, as described
at https://github.com/EpiDoc/EFES/wiki/Creating-an-RDF-repository [1]
and https://github.com/EpiDoc/EFES/wiki/Authority-lists


[1] Except that the URL is http://127.0.0.1:9999/rdf4j-workbench/


Jamie

[Gabriel Bodard]

Hi all,

I am now running the commons-lang-patch branch of EFES with Oracle JDK 11 (on Mac OSX), and not having a problem with creating the triplestore (although just got feedback from another user that with JDK 16 on Windows the triplestore won't build).

However whenever I try to open an inscription page now, despite having the new XSLT included, I get the following error:

Character reference "&#55296" is an invalid XML character.

This does *not* happen with commons-lang-patch and JDK 8 *nor* with the Home branch and JDK 11. (So it seems my hack was a temporary fix at best.) I am hopeful that we'll remove these Unicode references from the XSLT altogether at some point, but this is nonetheless a bug somewhere in the EFES stack, because this is   n o t    an invalid Unicode or XML character…

Is anyone else having the same problem, or is the triplestore issue still blocking this for some?

Gabby

[Jamie Norrish]

On Mon, 2021-08-02 at 14:39 +0000, Gabriel Bodard wrote:

> However whenever I try to open an inscription page now, despite
> having the new XSLT included, I get the following error:
>
> Character reference "&#55296" is an invalid XML character.
>
> This does *not* happen with commons-lang-patch and JDK 8 *nor* with
> the Home branch and JDK 11. (So it seems my hack was a temporary fix
> at best.)

Well, that's an odd set of conditions under which the problem triggers!
I have no idea what's going on there to cause the problem.


The RDF problem under JDK 16 is expected. To get that working requires
upgrading to use RDF4J 3, which itself requires an upgrade to Jetty.
I've done this locally, and it appears to work, but does prompt a lot
of warnings (of code at different versions between Jetty and RDF4J)
that I am having difficulty finding a way to remove.

Jamie