How to configure WireMock to load/save StubMappings to AWS S3?

[Dalton Fury]

I am trying to use WireMock's Record and Playback feature but with the difference that I want all the Stub Mappings to be saved to S3 and then load it back. I wrote a custom S3MappingSource that implements MappingsSource:

public class S3MappingSource implements MappingsSource {
  @Override
  public void save(StubMapping stubMapping) {
    storeToS3(stubMapping.getId().toString(), StubMapping.buildJsonStringFor(stubMapping));
  }


  @Override
  public void loadMappingsInto(StubMappings stubMappings) {
    var s3Objects = getAllFromS3();
    s3Objects.stream()
        .map(value -> StubMapping.buildFrom(value.getValueBytes().toStringUtf8()))
        .filter(Objects::nonNull)
        .forEach(stubMappings::addMapping);
  }


  @Override
  public void remove(StubMapping stubMapping) {
    throw new IllegalStateException("Not allowed");
  }


  @Override
  public void removeAll() {
    throw new IllegalStateException("Not allowed");
  }
}

Then I start the Wiremock server like this:

new WireMockServer(
    WireMockConfiguration.wireMockConfig()
        .port(mockConfiguration.getPort())
        .mappingSource(new S3MappingSource());

Even though mappings are getting stored in S3, and they are being read back, and everything is working right now, I don't feel that this is the right way. What is the recommended right way to do this?

[Tom Akehurst]

I'd say this is exactly the right way to do it. In fact this is how MockLab stores and loads stubs (not in S3, but exactly the same approach).

[Dalton Fury]

Hi Tom,

Thanks for the reply. This is nice. Since I am implementing a custom MappingSource which doesn't use a FileSource, should I do something about it? I see that WireMock is loading a SingleRootFileSource by default, will it cause any problem?

Thanks,

--
You received this message because you are subscribed to a topic in the Google Groups "wiremock-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wiremock-user/mPj6YlrdVkM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wiremock-use...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/wiremock-user/90af2490-73be-436f-97f6-3a4e60139c6f%40googlegroups.com.

[Tom Akehurst]

If you don't create any stubs that use the bodyFileName attribute then you'll be fine. But if you do want to use that you could also implement a FileSource that backs on to S3. The main benefit of that would be that you'd save on heap space since you wouldn't be holding bodies in memory, but I'd only recommend doing that if you're serving large responses (> a few hundred KB). 

On Wednesday, 3 June 2020 12:51:39 UTC+1, Dalton Fury wrote:

Hi Tom,

Thanks for the reply. This is nice. Since I am implementing a custom MappingSource which doesn't use a FileSource, should I do something about it? I see that WireMock is loading a SingleRootFileSource by default, will it cause any problem?

Thanks,

To unsubscribe from this group and all its topics, send an email to wiremock-user+unsubscribe@googlegroups.com.

[Dalton Fury]

Hi,

This is working very well for us. Now I am wondering if WireMock can sit in between all the communications that happen with an external API and send in to S3, in production.

We have a micro-service A that connects to an external service E. There is also a async call that I can make to send the objects to S3 (Kafka). Wiremock is set up as a microservice on it's own:

+--------+       +--------------+    +---------------+

|        |       |              |    |               |

|    A   +------>+   WireMock   +--->+ External API  |

|        |       |              |    |               |

+--------+       +------+-------+    +---------------+

                        |

                        |

                        |

                        |

                        v

                      Kafka

So for this to scale, either WireMock should not store stubs in memory. It should instantly put it into the Kafka queue and forget about it. 

Do you think wiremock is a good choice for this, or are we better of doing something else, like directly putting it into Kafka from the microservice?

On Thursday, June 4, 2020 at 12:04:06 AM UTC+5:30, Tom Akehurst wrote:

If you don't create any stubs that use the bodyFileName attribute then you'll be fine. But if you do want to use that you could also implement a FileSource that backs on to S3. The main benefit of that would be that you'd save on heap space since you wouldn't be holding bodies in memory, but I'd only recommend doing that if you're serving large responses (> a few hundred KB). 

On Wednesday, 3 June 2020 12:51:39 UTC+1, Dalton Fury wrote:

Hi Tom,

Thanks for the reply. This is nice. Since I am implementing a custom MappingSource which doesn't use a FileSource, should I do something about it? I see that WireMock is loading a SingleRootFileSource by default, will it cause any problem?

Thanks,

To unsubscribe from this group and all its topics, send an email to wiremo...@googlegroups.com.

[Dalton Fury]

This happens if I disable should record repreats as scenarios.

On Thursday, June 4, 2020 at 12:04:06 AM UTC+5:30, Tom Akehurst wrote:

If you don't create any stubs that use the bodyFileName attribute then you'll be fine. But if you do want to use that you could also implement a FileSource that backs on to S3. The main benefit of that would be that you'd save on heap space since you wouldn't be holding bodies in memory, but I'd only recommend doing that if you're serving large responses (> a few hundred KB). 

On Wednesday, 3 June 2020 12:51:39 UTC+1, Dalton Fury wrote:

Hi Tom,

Thanks for the reply. This is nice. Since I am implementing a custom MappingSource which doesn't use a FileSource, should I do something about it? I see that WireMock is loading a SingleRootFileSource by default, will it cause any problem?

Thanks,

To unsubscribe from this group and all its topics, send an email to wiremo...@googlegroups.com.

[Dalton Fury]

Sorry, that was a hurried email out of excitement on finding a clue to why wiremock is crashing after I implemented a S3 Mapping source.

I had implemented a S3MappingSource, and left the file source untouched, but later got this exception:

On debugging, I found that here, https://github.com/tomakehurst/wiremock/blob/master/src/main/java/com/github/tomakehurst/wiremock/recording/SnapshotStubMappingPostProcessor.java#L67

we call extractInPlace which results in a call to a FileSource, which breaks my code. I am planning to ditch the S3MappingSource and attempt to write a S3FileSource, unless I figure out a way to not trigger this call

[Dalton Fury]

So the problem was I assumed:

wireMockServer.startRecording(mockServerConfiguration.getTargetUrl())

and 

wireMockServer.startRecording(

WireMock.recordSpec()

.forTarget(mockServerConfiguration.getTargetUrl()));

to have the same effect. I'm wondering why that assumption was wrong. I've seen similar things in other parts of wiremock too. What's the rationale?

Thanks,