race condition while reading file from vertx cache?

[Asher Tarnopolski]

hi folks,

we have a verticle that creates vertx-web routes and then starts an http server. on some routes we add a JWTAuthHandler to provide jwt based security.

the keystore,jceks file used for that matter is in the classpath. 

all worked fine while we had 2 verticle instances, but once we increased the number of instances our verticle started to randomly (but frequently :) ) crash during deployment:

the two exceptions we've seen sa the reason for crash are:

java.lang.RuntimeException: java.io.FileNotFoundException: .vertx\file-cache-812c2a19-b5b6-4339-8a80-fabdfa90cb4e\jwt-keystore.jceks (The process cannot access the file because it is being used by another process)  

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:61)

at io.vertx.ext.auth.jwt.JWTAuth.create(JWTAuth.java:41)

at com.zzz.rest.verticles.RestServerVerticle.addJwtSecurity(RestServerVerticle.java:144)

at com.zzz.rest.verticles.RestServerVerticle.start(RestServerVerticle.java:78)

at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$3(ContextImpl.java:359)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)

at java.lang.Thread.run(Thread.java:745)

and

java.io.EOFException

at java.io.DataInputStream.readInt(DataInputStream.java:392)

at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:698)

at java.security.KeyStore.load(KeyStore.java:1445)

at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:62)

at io.vertx.ext.auth.jwt.JWTAuth.create(JWTAuth.java:41)

at com.zzz.rest.verticles.RestServerVerticle.addJwtSecurity(RestServerVerticle.java:144)

at com.zzz.rest.verticles.RestServerVerticle.start(RestServerVerticle.java:78)

at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8( .java:434)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$3(ContextImpl.java:359)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)

at java.lang.Thread.run(Thread.java:745)

as you can see, in both cases the problem occurs while JWTAuth tries to load a keystore file from vertx cache.

the exception is thrown from the following code in JWTAuthProviderImpl:

try (InputStream in = new FileInputStream(vertxInternal.resolveFile(keyStore.getString("path")))) ...

it seems the reading from the fileinputstream should be synchronized to prevent this from happen.

share your wisdom.

 

[Julien Viet]

Hi,

indeed, can you open an issue in Vert.x Web project for this please ?

thanks

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/bdeef25f-c3fa-4af6-90d7-f8acad12c783%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Asher Tarnopolski]

thanks julien,

issue opened in vertx-auth, since JWTAuthProviderImpl with a non synced code comes actually from there. 

https://github.com/vert-x3/vertx-auth/issues/71

[Paulo Lopes]

Should be fixed, I've added a test but it might be tricky to reproduce since it is a racing issue... can you verify that it works for you?

[Asher Tarnopolski]

thanks paulo,

i'll let you know if the problem returns :)

just to make sure - to use a fixed version now i should rollback to 3.3.0-SNAPSHOT?

[Paulo Lopes]

yes.

Or checkout the repo locally and build a local copy.

[mathias]

I keep getting a  NoSuchFileException when running multiple instances of my webserver verticle. Sometimes it throws a AccessDeniedException which is wired as well.

When configuring only one instance of the webserver the program starts fine.

Somehow vertx can't find the file inside the file-cache although I file-cache has been disabled at the beginning of the main method. So there must be an issue inside vertx.

[mathias]

After searching for the cause of this issue with the windows monitor "procmon" it turns out that windows begins a memory protection process which might cause this issue. This protection only occurs when there are too many file reads in a very short time. So my current workaround is to make vertx startup a  bit slower with Tread.sleep(1) and it works ;-)

Funny thing is that vertx runs super fast which is in this case faster than the windows policy allows :-)