How to transform response body before extraction with JSONPath?

[Willem Alexander Hajenius]

I've run into a little data transformation problem, perhaps someone could point me in the right direction?

I need to call a REST API that returns JSON, extract an item from the result, and save its ID for future reference. Normally, this would be easily done with .check, .jsonPath and .saveAs but there's a snag: the JSON is preceded by an AngularJS JSON vulnerability protection token )]}', so it is not actually valid JSON.

Let's say my API is /api/gizmos/ and it returns the following:

)]}',
{"gizmos":[{"id": "ABC"},{"id": "XYZ"}]}


I cannot use the following because the response body is not valid JSON:

    http("all gizmos")
        .get("/api/gizmos/")
        .headers(my_custom_headers)
        .check(jsonPath("$.gizmos[0].id").saveAs("first_id"))

       
Gatling will then complain as follows:

    all gizmos: KO jsonPath($.gizmos[0].id).find(0).exists failed, could not prepare: Could not parse response into a JSON object: Unable to determine the current character, it is not a string, number, array, or object

    The current character read is ')' with an int value of 41
    Unable to determine the current character, it is not a string, number, array, or object
    line number 1
    index number 0
    )]}',
    ^
       

I've tried adding a .transformResponse to delete the token before doing the check, but I get the same error:

    http("all gizmos")
        .get("/api/gizmos/")
        .headers(my_custom_headers)
        .transformResponse {
            case response if response.isReceived =>
              new ResponseWrapper(response) {
                override val body = StringResponseBody(response.body.string.replace(")]}',", ""), UTF_8)
              }
        }
        .check(jsonPath("$.gizmos[0].id").saveAs("first_id"))


I cannot do .check(bodyString.transform(...) because jsonPath cannot be used then (it can only be used as a starting point)

I'm using Gatling 2.1.5. What am I doing wrong?

[Stéphane LANDELLE]

Using a transformResponse should work, are you sure you properly tested?

Note that you should:

• use stripPrefix instead of replace
• use response.charset instead of hardcoding UTF_8

Stéphane Landelle

Lead developer

slan...@gatling.io

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Willem Alexander Hajenius]

The flow doesn't seem to get into the body of the transformResponse. I experimented with disabling the guard condition after the case response and adding some println's but to no avail. None of these println's show up in the output:

  .transformResponse {
  case response /*if response.isReceived*/ =>
    println("*** creating ResponseWrapper")
    new ResponseWrapper(response) {
      val stripped = response.body.string.stripPrefix(")]}',")
      println("*** stripped=" + stripped)
      override val body = StringResponseBody(stripped, response.charset)
    }
  case _ =>
    println("*** something else happened")
    null
}

The following println just before calling the API does show up in the console, so the println itself works fine and assures me that my changes indeed affect the log output (i.e. I'm not looking at stale output).

.exec(session => {
  println("*** going to call API!")
  session
})

[Stéphane LANDELLE]

I have no idea what happens on your side. The following example works just fine for me with Gatling 2.1.5:

import io.gatling.core.Predef._

import io.gatling.http.Predef._

import io.gatling.jdbc.Predef._

import io.gatling.http.response.ResponseWrapper

import io.gatling.http.response.StringResponseBody

class PerformanceTest extends Simulation {

  val httpProtocol = http

    .baseURL("http://gatling.io/")

  val scn = scenario("Basic")

    .exec(http("Home").get("/")

      .transformResponse {

        case response if response.isReceived =>

          println("*** creating ResponseWrapper")

          new ResponseWrapper(response) {

            val prefixed = "FOOBAR" + response.body.string

            println("*** prefixed=" + prefixed)

            override val body = StringResponseBody(prefixed, response.charset)

          }

      }.check(substring("FOOBAR").count.is(1)))

  setUp(scn.inject(atOnceUsers(1))).protocols(httpProtocol)

}

Please provide a reproducer.

Stéphane Landelle

Lead developer

slan...@gatling.io

[Willem Alexander Hajenius]

Found the culprit! My API was being called within a .resources() and then it apparently does not do the transformResponse correctly. Which is odd, because there are no compile errors that indicate I shouldn't be calling it there. When I moved the API call outside the .resources it is processed correctly as expected.

This code reproduces the problem:

import io.gatling.core.Predef._
import io.gatling.http.Predef._

import io.gatling.http.response.ResponseWrapper
import io.gatling.http.response.StringResponseBody

class PerformanceTest extends Simulation {

  val httpProtocol = http

    .baseURL("http://whois.icann.org/en")

  val scn = scenario("Basic")
    .exec(

      http("Regular").get("/about-whois")

        .transformResponse {
        case response if response.isReceived =>

          println("*** creating regular ResponseWrapper")

          new ResponseWrapper(response) {
            val prefixed = "FOOBAR" + response.body.string

            println("*** prefixed (regular)=" + prefixed.substring(0, 100))

            override val body = StringResponseBody(prefixed, response.charset)
          }
      }.check(substring("FOOBAR").count.is(1))

        .resources(
          http("Resources").get("/policies")

            .transformResponse {
            case response if response.isReceived =>

              println("*** creating ResponseWrapper inside resources")
              new ResponseWrapper(response) {
                val prefixed = "BAZQUX" + response.body.string
                println("*** prefixed inside resources=" + prefixed.substring(0, 100))

                override val body = StringResponseBody(prefixed, response.charset)
              }

          }.check(substring("BAZQUX").count.is(1)))
    )

  setUp(scn.inject(atOnceUsers(1))).protocols(httpProtocol)
}

Because gatling.io is a single-page app, fetching a different page was apparently hitting the cache, which I didn't want, so I used whois.icann.org instead.
Relevant part of the console output:
2015-07-17 10:55:09,250 INFO  Sending request=Regular uri=http://whois.icann.org/en/about-whois: scenario=Basic, userId=4170040969285647757-0

================================================================================
2015-07-17 10:55:09                                           0s elapsed
---- Basic ---------------------------------------------------------------------
[                                                                          ]  0%
          waiting: 1      / active: 0      / done:0    
---- Requests ------------------------------------------------------------------
> Global                                                   (OK=0      KO=0     )

================================================================================

*** creating regular ResponseWrapper
*** prefixed (regular)=FOOBAR
<!DOCTYPE html>
<!--[if lt IE 9 ]>    <html class="lt-ie9 no-js"  lang="en" dir="ltr"> <![end
2015-07-17 10:55:10,372 INFO  Sending request=Resources uri=http://whois.icann.org/en/policies: scenario=Basic, userId=4170040969285647757-0
2015-07-17 10:55:10,597 WARN  Request 'Resources' failed: substring(BAZQUX).count.is(1), but actually found 0

So transformResponse in the regular call works as expected, but not when nested inside a resources. Is this the expected behaviour?

[Stéphane LANDELLE]

Fixed, thanks for reporting!

https://github.com/gatling/gatling/issues/2772

Stéphane Landelle

Lead developer

slan...@gatling.io

--

[Nol de Wit]

Willem Alexender, 

Did this ever work for you? And if so, which Gatling/jdk version? 

See my thread here; I'm trying to do the same as you, but there is not way I get it to work. On Gatling 2.2.2 I get a compilation error; on Gatling 2.1.7 the same code compiles, but at runtime the body isn't modified.

I would appreciate it very much if you could give your thoughts about this.

Thanks,

Nol

Op vrijdag 17 juli 2015 11:09:50 UTC+2 schreef Willem Alexander Hajenius:

[Willem Alexander Hajenius]

Hello Nol,

I think I haven't retested it with the fixed version. I checked my e-mails from that time; when I discovered that my problem only occurred within .resources, I moved the calls outside and then everything worked fine. After those particular Gatling tests were successfully completed, I didn't follow up on this issue anymore.

Kind regards,

Willem