Browser proxy for subset of host URLs

Nikolas Alvelo

unread,

Jun 10, 2021, 9:58:53 AM6/10/21

to wiremock-user

Hello!

I am curious if there is a way to set up WireMock so that it is running in browser proxy mode but only processing traffic from a subset of URLs, specifically. We are seeing an issue where we have a crash occurring within the wiremock-jre8 library while snapshot-recording browser proxy traffic.

java.lang.NullPointerException at com.github.tomakehurst.wiremock.recording.LoggedResponseDefinitionTransformer.bodyDecompressedIfRequired(LoggedResponseDefinitionTransformer.java:69)at com.github.tomakehurst.wiremock.recording.LoggedResponseDefinitionTransformer.apply(LoggedResponseDefinitionTransformer.java:51)at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingGenerator.apply(SnapshotStubMappingGenerator.java:53)at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingGenerator.apply(SnapshotStubMappingGenerator.java:31)at com.google.common.collect.Iterators$6.transform(Iterators.java:783)at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingPostProcessor.process(SnapshotStubMappingPostProcessor.java:55)at com.github.tomakehurst.wiremock.recording.Recorder.serveEventsToStubMappings(Recorder.java:132)at com.github.tomakehurst.wiremock.recording.Recorder.takeSnapshot(Recorder.java:105)at com.github.tomakehurst.wiremock.core.WireMockApp.snapshotRecord(WireMockApp.java:446)at com.github.tomakehurst.wiremock.core.WireMockApp.snapshotRecord(WireMockApp.java:442)at com.github.tomakehurst.wiremock.WireMockServer.snapshotRecord(WireMockServer.java:461)

We think this is due to some problematic response coming from an endpoint that cannot map successfully to a ResponseDefinition, leading us to be unable to make snapshot recording without altering our clients to bypass the proxy for problematic hosts.

Ideally, the recording code would fail silently and only return the recorded mocks that were successfully loaded, but as that is not the current functionality, we are curious if there is a way to set some subset of hosts to be accepted by the browser proxy, with all other URLs bypassing the proxy and not being recorded for snapshot recording.

Thank you for any help you can provide here!
Nikolas Alvelo

Tom Akehurst

unread,

Jun 10, 2021, 11:57:20 AM6/10/21

to wiremock-user

You can add a stub matching on the hostname you don't want proxied and just return a dummy value.

However that NPE shouldn't happen so I'll fix that in the next release.

Nikolas Alvelo

unread,

Jun 10, 2021, 1:09:50 PM6/10/21

to wiremock-user

Thanks for the quick reply, Tom!

Ideally, we'd want these hostnames to be passed through to their destinations, as they would with the browser proxying. Adding a stub for them and returning a dummy value could cause unknown behavior. Do you think there's an opportunity in the browser proxying or in-memory snapshotting feature for providing a "set" of hostnames that should be included in the snapshots and serialized to StubMappings? We'd essentially want everything to go through the browser proxy but only a subset to be processed as stub mappings. Similar to if we could have more than 1 proxyBaseUrl at once, rather than enabling browser proxying entirely.

Here's another flavor of that issue that I described above:

com.github.tomakehurst.wiremock.common.JsonException: {

"errors" : [ {

"code" : 10,

"source" : { },

"title" : "Error parsing JSON",

"detail" : "Unrecognized token 'x': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (String)\"x��R]k�0\u0010�+Eϲ...\"; line: 1, column: 2]"

} ]

}

at com.github.tomakehurst.wiremock.common.JsonException.fromJackson(JsonException.java:53)

at com.github.tomakehurst.wiremock.common.Json.read(Json.java:55)

at com.github.tomakehurst.wiremock.matching.EqualToJsonPattern.<init>(EqualToJsonPattern.java:24)

at com.github.tomakehurst.wiremock.recording.RequestBodyAutomaticPatternFactory.forRequest(RequestBodyAutomaticPatternFactory.java:68)

at com.github.tomakehurst.wiremock.recording.RequestPatternTransformer.apply(RequestPatternTransformer.java:60)

at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingGenerator.apply(SnapshotStubMappingGenerator.java:52)

at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingGenerator.apply(SnapshotStubMappingGenerator.java:31)

at com.google.common.collect.Iterators$6.transform(Iterators.java:783)

at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)

at com.github.tomakehurst.wiremock.recording.SnapshotStubMappingPostProcessor.process(SnapshotStubMappingPostProcessor.java:55)

at com.github.tomakehurst.wiremock.recording.Recorder.serveEventsToStubMappings(Recorder.java:132)

at com.github.tomakehurst.wiremock.recording.Recorder.takeSnapshot(Recorder.java:105)

at com.github.tomakehurst.wiremock.core.WireMockApp.snapshotRecord(WireMockApp.java:446)

at com.github.tomakehurst.wiremock.core.WireMockApp.snapshotRecord(WireMockApp.java:442)

at com.github.tomakehurst.wiremock.WireMockServer.snapshotRecord(WireMockServer.java:461)

at app.transformer.RecordMocks.transform(RecordMocks.kt:49)

at app.transformer.RequestTransformer.transform(RequestTransformer.kt:38)

at com.github.tomakehurst.wiremock.stubbing.InMemoryStubMappings.applyTransformations(InMemoryStubMappings.java:101)

at com.github.tomakehurst.wiremock.stubbing.InMemoryStubMappings.serveFor(InMemoryStubMappings.java:80)

at com.github.tomakehurst.wiremock.core.WireMockApp.serveStubFor(WireMockApp.java:209)

at com.github.tomakehurst.wiremock.http.StubRequestHandler.handleRequest(StubRequestHandler.java:56)

at com.github.tomakehurst.wiremock.http.AbstractRequestHandler.handle(AbstractRequestHandler.java:64)

at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.service(WireMockHandlerDispatchingServlet.java:121)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)

at java.lang.Thread.run(Thread.java:748)

If possible, we could solve some of this with options to fail silently when attempting to read request bodies as JSON or ignore "bad" JSON when snapshotting stubs (either by moving on and not generating a stub for this proxied request or producing one with an empty body).

In our use case, we have a variety of traffic from our automated mobile applications going through WireMock, but only a few hostnames that we will actually want to be stubbing, allowing the rest of our request to behave normally through the browser proxy function.

Appreciate you taking a look at this,

Nik

Tom Akehurst

unread,

Jun 11, 2021, 8:36:18 AM6/11/21

to wiremock-user

Would you be able to share a bit more context so that I can replicate these errors myself?

It seems like you're trying to work around one or more WireMock bugs, so I'd like to try and fix these (I'm about to start working on the first one, so that'll be shipped early next week with any luck).

BTW, I've never tried this but you could try using a proxy stub rather than a dummy response in the workaround I described in my previous reply.

Nikolas Alvelo

unread,

Jun 11, 2021, 3:29:31 PM6/11/21

to wiremock-user

No problem!
Here is the full error message from the JsonException above:

com.github.tomakehurst.wiremock.common.JsonException: {

"errors" : [ {

"code" : 10,

"source" : { },

"title" : "Error parsing JSON",

"detail" : "Unrecognized token 'x': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (String)\"x�u��J�0\u0010�_Er�C�$MһEŽQ\u0004ћR�ئk�m�4]<�wwZW\u0014�!��\u0017&�\t�\\\u0010!��)�P�JR\u00049��M�\u0002D\b\\�͛i/(x'a�7��G��08u�(.��\u001BY��*T%��.9ǵQ\\�%�}ۍ��֠�\u0013\r͇A\u0019��t\u000Bʡվ�\u0000��o��ۓqCc{�NS�\"�\u0011\u000Er\u0000ΡUi\u000Fs/��u7{\u000Ff�\u000B��A/��`biG �H��03\u0013\u0011V\u0012�a�+�S�\u0003�5�\u0019A��\u000B��l�\u001B^G�̀2�#*I��T)&8��Bޚκ��q0�:��i�C�'�Y~��\u0004R\u0012\t\fT�%��p��m�{�,vkt�_�K۶��[�f�뀰�$��4e�HLO�g�\u0019|]��4\u0012KR\u0002�`J��ME��|��\t��\u000B��k\"; line: 1, column: 2]"