Trouble Getting Started

[A L]

Hi,

Thank you for making Wiremock. Sorry, but I am having issues using it with Testng inside a Docker container.

It could very well be that I am new to Java as well.

But here are my stack trace and source code:

Running my.adapter.HttpSinkTest
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@2db7a79b
start
setting up mock server...
log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
finished setting up mock server...
stubbing...
stubbed...
sending...
process
defaultValue
[Event headers = {}, body.length = 7 ]
Test 11
response code:500
sent...
stop
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.968 sec <<< FAILURE!
testHttpSink(my.adapter.HttpSinkTest)  Time elapsed: 4.469 sec  <<< FAILURE!
com.github.tomakehurst.wiremock.client.VerificationException: Expected status 200 for http://localhost:8080/__admin/near-misses/request-pattern but was 500
        at com.github.tomakehurst.wiremock.client.HttpAdminClient.safelyExecuteRequest(HttpAdminClient.java:269)
        at com.github.tomakehurst.wiremock.client.HttpAdminClient.postJsonAssertOkAndReturnBody(HttpAdminClient.java:249)
        at com.github.tomakehurst.wiremock.client.HttpAdminClient.findTopNearMissesFor(HttpAdminClient.java:217)
        at com.github.tomakehurst.wiremock.client.WireMock.findAllNearMissesFor(WireMock.java:523)
        at com.github.tomakehurst.wiremock.client.WireMock.findNearMissesFor(WireMock.java:519)
        at com.github.tomakehurst.wiremock.client.WireMock.verificationExceptionForNearMisses(WireMock.java:387)
        at com.github.tomakehurst.wiremock.client.WireMock.verifyThat(WireMock.java:367)
        at com.github.tomakehurst.wiremock.WireMockServer.verify(WireMockServer.java:279)
        at rsae.adapter.HttpSinkTest.testHttpSink(HttpSinkTest.java:82)


Results :

Failed tests:   testHttpSink(my.adapter.HttpSinkTest): Expected status 200 for http://localhost:8080/__admin/near-misses/request-pattern but was 500

package my.adapter;

import org.testng.annotations.BeforeTest;
import org.testng.annotations.AfterTest;
import org.testng.annotations.Test;
import org.testng.Assert;

import com.github.tomakehurst.wiremock.WireMockServer;
import static com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig;
import static com.github.tomakehurst.wiremock.client.WireMock.*;
import com.github.tomakehurst.wiremock.client.WireMock;

import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.channel.PseudoTxnMemoryChannel;
import org.apache.flume.conf.Configurables;
import org.apache.flume.event.EventBuilder;
import org.apache.flume.lifecycle.LifecycleException;

import com.github.tomakehurst.wiremock.verification.NearMiss;
import java.util.List;

import java.io.*;
import java.net.*;
import java.util.concurrent.TimeUnit;

public class HttpSinkTest {

  private Channel channel;
  private HttpSink sink;
  private WireMockServer wireMockServer;
  private WireMock wireMock;

  @BeforeTest
  public void setUp() {
    Context context = new Context();
    context.put("endpoint", "http://localhost:8080/endpoint");

    sink = new HttpSink();
    channel = new PseudoTxnMemoryChannel();
    sink.setChannel(channel);
    sink.start();
    Configurables.configure(channel, context);
    Configurables.configure(sink, context);

    System.out.println("setting up mock server...");
    //wireMockServer = new WireMockServer(wireMockConfig().port(8080));
    wireMockServer = new WireMockServer();
    wireMockServer.start();
    System.out.println("finished setting up mock server...");

    //WireMock.configureFor("localhost", 8081);
    //wireMock = new WireMock("localhost", 8081);
  }

  @AfterTest
  public void tearDown() {
    sink.stop();
    wireMockServer.stop();
  }

  @Test
  public void testHttpSink() throws
      InterruptedException,
      LifecycleException,
      EventDeliveryException {

    System.out.println("stubbing...");
    wireMockServer.stubFor(post(urlEqualTo("/endpoint"))
      //.withRequestBody(equalToJson(event("SUCCESS")))
      .willReturn(aResponse().withStatus(200)));
    System.out.println("stubbed...");

    System.out.println("sending...");
    Event event = EventBuilder.withBody(("Test 11").getBytes());
    channel.put(event);
    sink.process();
    System.out.println("sent...");

    wireMockServer.verify(
      1,
      postRequestedFor(
        urlEqualTo("/endpoint")
      )
    );
    System.out.println("verified...");
  }
}

package my.adapter;

import org.apache.flume.sink.AbstractSink;
import org.apache.flume.conf.Configurable;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Context;
import org.apache.flume.Channel;
import org.apache.flume.Event;
import org.apache.flume.Transaction;

import java.net.HttpURLConnection;
import java.io.IOException;
import java.net.URL;
import java.io.OutputStream;
import java.net.MalformedURLException;

public class HttpSink extends AbstractSink implements Configurable {
  private String myProp;
  private HttpURLConnection httpClient;

  private static final String DEFAULT_CONTENT_TYPE = "text/plain";
  private static final String DEFAULT_ACCEPT_HEADER = "text/plain";
  private static final int DEFAULT_CONNECT_TIMEOUT = 5000;
  private static final int DEFAULT_REQUEST_TIMEOUT = 5000;

  private URL endpointUrl;
  private String contentTypeHeader = DEFAULT_CONTENT_TYPE;
  private String acceptHeader = DEFAULT_ACCEPT_HEADER;
  private int connectTimeout = DEFAULT_CONNECT_TIMEOUT;
  private int requestTimeout = DEFAULT_REQUEST_TIMEOUT;

  @Override
  public void configure(Context context) {
    String myProp = context.getString("myProp", "defaultValue");

    try {
      endpointUrl = new URL(context.getString("endpoint", ""));
    } catch (MalformedURLException e) {
      throw new IllegalArgumentException("Endpoint URL invalid", e);
    }

    // Process the myProp value (e.g. validation)

    // Store myProp for later retrieval by process() method
    this.myProp = myProp;
  }

  @Override
  public void start() {
    System.out.println("start");
    // Initialize the connection to the external repository (e.g. HDFS) that
    // this Sink will forward Events to ..
  }

  @Override
  public void stop () {
    System.out.println("stop");
    // Disconnect from the external respository and do any
    // additional cleanup (e.g. releasing resources or nulling-out
    // field values) ..
  }

  @Override
  public Status process() throws EventDeliveryException {
    Status status = null;
    OutputStream outputStream = null;

    // Start transaction
    Channel ch = getChannel();
    Transaction txn = ch.getTransaction();
    txn.begin();
    System.out.println("process");
    System.out.println(myProp);
    try {
      // This try clause includes whatever Channel operations you want to do

      Event event = ch.take();

      System.out.println(event);
      System.out.println(new String(event.getBody()));

      httpClient = getConnection();

      outputStream = httpClient.getOutputStream();
      outputStream.write(event.getBody());
      outputStream.flush();
      outputStream.close();

      int httpStatusCode = httpClient.getResponseCode();
      System.out.println("response code:" + httpStatusCode);

      httpClient.getInputStream().close();

      // Send the Event to the external repository.
      // storeSomeData(e);

      txn.commit();
      status = Status.READY;
    } catch (Throwable t) {
      txn.rollback();

      // Log exception, handle individual exceptions as needed

      status = Status.BACKOFF;

      // re-throw all Errors
      if (t instanceof Error) {
        throw (Error)t;
      }
    } finally {
      txn.close();
    }
    return status;
  }

  HttpURLConnection getConnection() throws IOException {
    HttpURLConnection connection = (HttpURLConnection) endpointUrl.openConnection();
    connection.setRequestMethod("POST");
    connection.setRequestProperty("Content-Type", contentTypeHeader);
    connection.setRequestProperty("Accept", acceptHeader);
    connection.setConnectTimeout(connectTimeout);
    connection.setReadTimeout(requestTimeout);
    connection.setDoOutput(true);
    connection.setDoInput(true);
    connection.connect();
    return connection;
  }
}

If you can point me to anything to try, that would be great cause I'm just stuck and don't know what else to do.

Thank you for your time!

[A L]

I've tried testing to see if the wireMockServer is up:

    try {
      TimeUnit.SECONDS.sleep(10);
      System.out.println("###1");
      StringBuilder result = new StringBuilder();
      System.out.println("###2");
      URL url = new URL("http://localhost:8080/__admin");
      System.out.println("###3");
      HttpURLConnection conn = (HttpURLConnection) url.openConnection();
      System.out.println("###4");
      conn.setRequestMethod("GET");
      System.out.println("###5");
      System.out.println(conn.getResponseCode());
      BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
      System.out.println("###6");
      String line;
      System.out.println("###7");
      while ((line = rd.readLine()) != null) {
        System.out.println("###8");
       result.append(line);
      }
      System.out.println("###9");
      rd.close();
      System.out.println("###10");
      System.out.println(result.toString());
    } catch(MalformedURLException ex) {
    } catch(IOException ex) {
    }

But it seems like the server just never comes up. But I would assume that the request will time out instead of returning 500.

Sorry, not sure if that helps or not, but that's the only clue I've got.

[A L]

Sorry, I'm not sure how to edit the posted code to separate out the two files....

[Tom Akehurst]

I suspect one of two things is happening:

1) WireMock is running, your client is connecting to it but something about how/where you're running it is causing it to throw a 500 response. In this case you'll need to turn on verbose logging on the WireMock server and see what's happening.

2) Your WireMock client is pointed at the wrong port and that 500 error is coming from another service.

[A L]

Thanks for the suggestions.  Sorry, but how do I turn on verbose logging?  I'm unfamiliar with Java and slf4j so this doesn't make sense to me:

Notification (logging)

WireMock wraps all logging in its own Notifier interface. It ships with no-op, Slf4j and console (stdout) implementations.

// Provide an alternative notifier. The default logs to slf4j.
.notifier(new ConsoleNotifier(true))

Thank you again for your help.

[A L]

I moved the ports around and I think it's case 1:

1) WireMock is running, your client is connecting to it but something about how/where you're running it is causing it to throw a 500 response. In this case you'll need to turn on verbose logging on the WireMock server and see what's happening.

When the ports are mismatched, I don't get a 500.

I guess, I just need to get a more verbose logging.  Sorry, for another dumb question, where are the logs stored now?  Nothing gets printed to the screen from the wiremock server.

Thank you again for your time.

[A L]

Also, I am running everything inside a docker container, so maybe that is doing something funky...

[A L]

Okay, got Log4j working....just needed a log4j.properties file under /src/main/resources

Any ideas on what could be causing these errors?:
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@6950e31

start
setting up mock server...

2016-08-11 20:31:48 INFO  log:186 - Logging initialized @10315ms
2016-08-11 20:31:48 INFO  Server:327 - jetty-9.2.13.v20150730
2016-08-11 20:31:48 INFO  ContextHandler:744 - Started o.e.j.s.ServletContextHandler@305a0c5f{/__admin,null,AVAILABLE}
2016-08-11 20:31:48 INFO  ContextHandler:744 - Started o.e.j.s.ServletContextHandler@373ebf74{/,null,AVAILABLE}
2016-08-11 20:31:48 INFO  ServerConnector:266 - Started ServerConnector@4c550889{HTTP/1.1}{0.0.0.0:8080}
2016-08-11 20:31:48 INFO  Server:379 - Started @10666ms

finished setting up mock server...
stubbing...
stubbed...
sending...
process
defaultValue
[Event headers = {}, body.length = 7 ]
Test 11

2016-08-11 20:31:49 INFO  /:2052 - RequestHandlerClass from context returned com.github.tomakehurst.wiremock.http.StubRequestHandler. Normalized mapped under returned 'null'
2016-08-11 20:31:49 WARN  ServletHandler:667 - Error for /endpoint
java.lang.NoClassDefFoundError: com/google/common/base/MoreObjects
        at com.github.tomakehurst.wiremock.servlet.WireMockHttpServletRequestAdapter.getCookies(WireMockHttpServletRequestAdapter.java:189)
        at com.github.tomakehurst.wiremock.verification.LoggedRequest.createFrom(LoggedRequest.java:54)
        at com.github.tomakehurst.wiremock.stubbing.InMemoryStubMappings.serveFor(InMemoryStubMappings.java:80)
        at com.github.tomakehurst.wiremock.core.WireMockApp.serveStubFor(WireMockApp.java:104)
        at com.github.tomakehurst.wiremock.http.StubRequestHandler.handleRequest(StubRequestHandler.java:32)
        at com.github.tomakehurst.wiremock.http.AbstractRequestHandler.handle(AbstractRequestHandler.java:41)
        at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.service(WireMockHandlerDispatchingServlet.java:99)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(java.base@9-internal/Thread.java:843)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.MoreObjects
        at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:366)
        at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:185)
        at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419)
        ... 27 more
response code:500
sent...
verifying...
verifying2...
verified...
2016-08-11 20:31:49 WARN  HttpChannel:383 - /endpoint
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:684)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(java.base@9-internal/Thread.java:843)
2016-08-11 20:31:49 WARN  HttpChannel:481 - Could not send response error 500: java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
stop
2016-08-11 20:31:49 INFO  ServerConnector:306 - Stopped ServerConnector@4c550889{HTTP/1.1}{0.0.0.0:8080}
2016-08-11 20:31:49 INFO  ContextHandler:865 - Stopped o.e.j.s.ServletContextHandler@373ebf74{/,null,UNAVAILABLE}
2016-08-11 20:31:49 INFO  ContextHandler:865 - Stopped o.e.j.s.ServletContextHandler@305a0c5f{/__admin,null,UNAVAILABLE}
2016-08-11 20:31:49 WARN  QueuedThreadPool:178 - qtp503938393{STOPPING,8<=8<=10,i=6,q=6} Couldn't stop Thread[qtp503938393-53,5,main]
2016-08-11 20:31:49 WARN  QueuedThreadPool:178 - qtp503938393{STOPPING,8<=8<=10,i=4,q=6} Couldn't stop Thread[qtp503938393-54,5,]
2016-08-11 20:31:49 WARN  QueuedThreadPool:178 - qtp503938393{STOPPING,8<=8<=10,i=0,q=3} Couldn't stop Thread[qtp503938393-56,5,]

[A L]

Posting this in case it helps anyone else.

It turn out that Docker was not routing the requests to the wire mock server correctly.  I know Docker treats 0.0.0.0 different from localhost, but not sure if that's where the problem lies.

The fix was to run the mock server in a separate container, so something like this in the docker-compose.yml:
version: '2'
services:
  adapter:
    extends:
      file: docker/docker-compose.common.yml
      service: adapter
    volumes:
      - .:/usr/src/app/
    ports:
      - 44444:44444
    links:
      - mockserver

  mockserver:
    image: rodolpheche/wiremock:2.1.7

[Tom Akehurst]

Judging by the stack traces you've posted the problem is classpath related. Somehow you've either ended up excluding some of WireMock's dependencies or doubling them up with incompatible versions (Guava is the one failing in the stack trace).

Are you using the standalone artifact inside Docker?

[A L]

Hi Tom,

Thanks again for helping me out with this.

I thought Maven handled all of the dependencies, but here is my POM (I don't see anything about Guava):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>my.adapter</groupId>
  <artifactId>my-adapter</artifactId>
  <version>0.0.1</version>
 
  <repositories>
    <repository>
      <id>central</id>
      <name>Maven Central</name>
      <url>http://repo1.maven.org/maven2/</url>
    </repository>
    <repository>
      <id>jcenter</id>
      <name>bintray</name>
      <url>http://jcenter.bintray.com</url>
    </repository>
  </repositories>
 
  <properties>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>
 
  <dependencies>
    <dependency>
      <groupId>org.apache.flume</groupId>
      <artifactId>flume-ng-core</artifactId>
      <version>1.6.0</version>
    </dependency>
    <dependency>
      <groupId>org.json</groupId>
      <artifactId>json</artifactId>
      <version>20160212</version>
    </dependency>
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>6.9.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.flume</groupId>
      <artifactId>flume-ng-embedded-agent</artifactId>
      <version>1.4.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.github.tomakehurst</groupId>
      <artifactId>wiremock</artifactId>
      <version>2.1.7</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpcore</artifactId>
      <version>4.4</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.19.1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

How do you know it is related to Guava?  I've only been Googling:

RequestHandlerClass from context returned com.github.tomakehurst.wiremock.http.StubRequestHandler. Normalized mapped under returned 'null'

and

java.lang.NoClassDefFoundError: com/google/common/base/MoreObjects

Do you know how I might be able to figure out the dependency issue?  It would be nice to not have to pull in another Docker image just to run tests.

Currently, to get past the issue, I'm using an image that is using the standalone artifact (I'm guessing):
https://github.com/rodolpheche/wiremock-docker/blob/master/Dockerfile#L5

Prior, I was not using the standalone.  So the stack trace is from when I was not using the standalone.

Thanks again for your time and sharing your knowledge!

[Tom Akehurst]

That MoreObjects class is part of Guava and although you're not explicitly referencing it you're probably pulling another version transitively.

I'd definitely suggest using the standalone version if you have this kind of problem.

[A L]

Thanks Tom.  I feel better that you are recommending using the standalone approach and that I'm not just skirting the problem.  Thanks again for your help and take care!