Transform response to read json compressed data

[Alain Szalajski]

Hi, 

I need help to acheive my case. I try to read json data in the response but I am facing a problem because the json data are returned into a tar compress format (XZ).

My strategy is to add a transform step to uncompress the boby response, and read the json data into the tar file in order to use jsonPath as if the json data was not compressed.

My code is  : 

def getPorteFeuille = http("REFCLIENT_PORTEFEUILLE")

      .post("/mypath/")

      .headers(Config.headers_11)

      .body(RawFileBody("SYNC/SYNC_PORTE_FEUILLE.json")).asJSON

      .check(status is 200)

      .transformResponse { case response if (response.statusCode.get == 200) =>

            logger.info("Transfomr step start)

            new ResponseWrapper(response) {

                logger.debug("Load of body compressed data")

                val bodyCompressed = new ByteArrayInputStream(response.body.bytes)

                logger.debug("Uncompressing the data")

                val uncompressedInputStream = new XZCompressorInputStream(bodyCompressed)

                logger.debug("TAR archive intialization")

                val tarArchiveInputStream = new TarArchiveInputStream(uncompressedInputStream)

                val decompressedByteArray = null

                val outputStreamDataJson =  new ByteArrayOutputStream();

                object foundJsonException extends Throwable { }

                object notfoundJsonException extends Throwable { }

                try {

                  logger.debug("Loop over TAR entries")

                  while (true){

                    val currentEntry = tarArchiveInputStream.getNextEntry()

                    logger.info("Current entry : " + currentEntry.getName())

                    if (currentEntry != null) {

                      if (currentEntry.getName() == "data.json"){

                        logger.info("Entry data.json found")

                        IOUtils.copy(tarArchiveInputStream, outputStreamDataJson)

                        outputStreamDataJson.close()

                        throw foundJsonException

                      }

                      else {

                        logger.info("Entry ignored" + currentEntry.getName())

                      }

                    }

                    else {

                      throw notfoundJsonException

                    }

                  }

                }

                catch {

                  case foundJsonException : Throwable => logger.info("end loop : json data found")

                  case notfoundJsonException : Throwable => logger.error("end loop : json data not found")

                }

                logger.info("Tranforming the body into json format")

//                logger.info("Json data size : " + outputStreamDataJson.size())

//                val fileJson = new File("gatling.data.json")

//                val fileJsonOutputStream = new FileOutputStream(fileJson)

//                val bw = new BufferedWriter(new FileWriter(fileJson))

//                fileJsonOutputStream.write(outputStreamDataJson.toByteArray())

//                fileJsonOutputStream.flush()

//                logger.info("Writing data into gatling.data.json");

                outputStreamDataJson.flush()

                override val body = new ByteArrayResponseBody(outputStreamDataJson.toByteArray, UTF_8)

            }

         

        }

        .check(jsonPath("$").findAll.saveAs("all_clients"))

When I uncomment the lignes to write json data into a local file, the content is good (json format). So the uncompress process and the tar loop are fine but I got the following error : java.lang.IndexOutOfBoundsException: index: 0, length: 18432 (expected: range(0, 1180)). I think the problem is about streaming. Gatling tries to execute jsonPath whereas all the data have not been processed entirely to have a complet boby.

The stacktrace is : 

        java.lang.IndexOutOfBoundsException: index: 0, length: 18432 (expected: range(0, 1180))

at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)

at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)

at io.netty.buffer.AbstractByteBuf.checkDstIndex(AbstractByteBuf.java:1139)

at io.netty.buffer.PooledHeapByteBuf.getBytes(PooledHeapByteBuf.java:92)

at io.netty.buffer.AbstractByteBuf.getBytes(AbstractByteBuf.java:440)

at io.gatling.commons.util.ByteBufs$.io$gatling$commons$util$ByteBufs$$$anonfun$2(ByteBufs.scala:32)

at io.gatling.commons.util.ByteBufs$.io$gatling$commons$util$ByteBufs$$$anonfun$2$adapted(ByteBufs.scala:31)

at scala.collection.immutable.List.foreach(List.scala:381)

at io.gatling.commons.util.ByteBufs$.byteBufsToByteArray(ByteBufs.scala:31)

at io.gatling.http.response.ByteArrayResponseBody$.apply(ResponseBody.scala:90)

at io.gatling.http.response.ResponseBuilder.build(ResponseBuilder.scala:198)

at io.gatling.http.ahc.AsyncHandler.withResponse(AsyncHandler.scala:126)

at io.gatling.http.ahc.AsyncHandler.onCompleted(AsyncHandler.scala:134)

at io.gatling.http.ahc.AsyncHandler.onCompleted(AsyncHandler.scala:47)

at org.asynchttpclient.netty.NettyResponseFuture.getContent(NettyResponseFuture.java:181)

at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:215)

at org.asynchttpclient.netty.handler.HttpHandler.finishUpdate(HttpHandler.java:58)

at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:159)

at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:187)

at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:76)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)

at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)

at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)

at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)

at java.lang.Thread.run(Thread.java:745)

Thank you for any help

Regards.

[Alain Szalajski]

I complete my version of gatling : 2.2.3