[Haskell-cafe] Strange encoding issue with Text and Servant

[Jan von Löwenstein]

Hi,

I have got a two places with a `QueryParam "q" Text` and call it with a Text that contains a `=` literal. In one place the `=` is correctly encoded as %3D, in the other I see a `!D(MISSING)`.

This has to happen somewhere in Servant or the lower layers. Both Texts print out nicely with an `=` sign if I just print them to stdout.

Google does not find `!D(MISSING)` anywhere.

Any idea what could possibly be the problem here?

Best

Jan

[Nigel Rantor]

It would be really useful to see exactly how you are 'calling' this.

Do you have a curl command line or are you using some other client?

n

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

[Jan von Löwenstein]

Oh, it happens with servant on the client side.

I use a swagger-codegen generated servant based client to access kubernetes.

Calling https://github.com/soundcloud/haskell-kubernetes/blob/master/lib/Kubernetes/Api/ApivApi.hs#L475 works

Calling https://github.com/soundcloud/haskell-kubernetes/blob/master/lib/Kubernetes/Api/ApivApi.hs#L473 doesn't.

From what I see the `QueryParam "labelSelector" Text` is the same in both.

Logging the value gives the literal `=` as expected. Logging the (http-client) requests shows the difference.

Best

Jan

[Nigel Rantor]

On 14/01/17 17:37, Jan von Löwenstein wrote:
> Oh, it happens with servant on the client side.
>
> I use a swagger-codegen generated servant based client to access kubernetes.
> Calling https://github.com/soundcloud/haskell-kubernetes/blob/master/lib/Kubernetes/Api/ApivApi.hs#L475 works
> Calling https://github.com/soundcloud/haskell-kubernetes/blob/master/lib/Kubernetes/Api/ApivApi.hs#L473 doesn't.
>
> From what I see the `QueryParam"labelSelector" Text`is the same in both.
>
> Logging the value gives the literal `=` as expected. Logging the
> (http-client) requests shows the difference.

So I went ahead and wrote a simple client by hand and when I call the
two functions on a local endpoint that shows me the request URL I get
the following.

Same arguments for 'pretty' and 'labelSelector' in both cases.

Code:

runListSecret :: Manager -> BaseUrl -> ClientM SecretList

runListSecret manager baseUrl = listSecret (Just "prettyArg") (Just "=")
Nothing Nothing Nothing Nothing manager baseUrl

runListService :: Manager -> BaseUrl -> ClientM ServiceList

runListService manager baseUrl = listService (Just "prettyArg") (Just
"=") Nothing Nothing Nothing Nothing manager baseUrl

Server log:

127.0.0.1 - - [14/Jan/2017:18:39:49 +0000] "GET
/api/v1/secrets?pretty=prettyArg&labelSelector=%3D HTTP/1.1" 404 - 0.0003
127.0.0.1 - - [14/Jan/2017:18:39:49 +0000] "GET
/api/v1/services?pretty=prettyArg&labelSelector=%3D HTTP/1.1" 404 - 0.0003

I haven't used swagger-codegen but the servant-client code works for me
so I would look more into that.

Swagger codegen looks like it might take me a while to get my head into
so maybe you could look at the generated code, or sling it to me here or
directly.

The two calls are so similar that I can't see why the generated client
would behave differently in those cases.

Regards,

n

> Best
> Jan
>
> Nigel Rantor <wig...@wiggly.org <mailto:wig...@wiggly.org>> schrieb am

[David Turner]

Grasping at straws a little bit here, but can you (a) do a packet capture (e.g. `tcpdump -X`) to see what's going back and forth on the wire, just to make absolutely sure all of the oddness is on the client's end?

A string like `!D(MISSING)` sorta looks like a decoding error so perhaps it will be instructive to look at the bytes on the wire it's trying to decode, although I can't think what could be confused with an equals sign. It's not like that blooming Greek question mark (http://www.fileformat.info/info/unicode/char/037e/index.htm).

Cheers,

[Jan von Löwenstein]

Actually the code I pointed at _is_ the swagger-codegen generated client.

I have got a very small wrapper around it:

```

listService :: (MonadIO m, MonadCatch m, MonadReader Config m, MonadLog Text m) =>

     Text

  -> m ServiceList.ServiceList

listService labelSelector = namespacedF $ \namespace -> Kube.listNamespacedService namespace Nothing (Just labelSelector) Nothing Nothing Nothing Nothing

listSecret :: (MonadIO m, MonadThrow m, MonadReader Config m, MonadLog Text m) =>

     Text

  -> m SecretList.SecretList

listSecret labelSelector = namespacedF $ \namespace -> Kube.listNamespacedSecret namespace Nothing (Just labelSelector) Nothing Nothing Nothing Nothing

namespacedF :: (MonadIO m, MonadThrow m, MonadReader Config m, MonadLog Text m) =>

             NamespacedF model

          -> m model

namespacedF f = do

  config <- ask

  result <- let

    baseUrl = apiEndpoint config

    tlsSettings = Base.tlsSettings baseUrl (credentials config)

    kubeNamespace = namespace config

    in do

      manager <- liftIO $ newManager $ (mkManagerSettings tlsSettings Nothing)

                                        {

                                          managerModifyRequest = \req ->

                                            return req {

                                              checkResponse = \req res -> do

                                                Text.IO.hPutStrLn stderr (Text.pack (show req))

                                                responseMessage <- showResponse res

                                                Text.IO.hPutStrLn stderr responseMessage

                                                return ()

                                            }

                                        }

      liftIO $ runExceptT $ f kubeNamespace manager baseUrl

  either throwM return result

```

I call them like:

```

do

        let Just agentId = ...

            selector = "agentId" <> "=" <> agentId

        logMessage $ "listSecret selector: '" <> selector <> "'"

        secretList <- listSecret selector

```

`logMessage` gives a literal equals sign in both cases.

`Text.IO.hPutStrLn stderr (Text.pack (show req))` from namespacedF gives `%3D` and `%!D(MISSING)` depending on which function I call.

From what I understand this is definitely happening on the client side. Do you still think a tcpdump might be worth it?

```

listSecret selector: 'agentId=b49d4406-8020-44e6-7788-6f92d5c0e732'

Request {

  host                 = "192.168.99.100"

  port                 = 8443

  secure               = True

  requestHeaders       = [("Accept","application/json")]

  path                 = "/api/v1/namespaces/default/secrets"

  queryString          = "?labelSelector=agentId%!D(MISSING)b49d4406-8020-44e6-7788-6f92d5c0e732"

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

```

```

listService selector: 'agentId=24f99a4b-1682-44da-7ba4-dba935d107d2'

Request {

  host                 = "192.168.99.100"

  port                 = 8443

  secure               = True

  requestHeaders       = [("Accept","application/json")]

  path                 = "/api/v1/namespaces/default/services"

  queryString          = "?labelSelector=agentId%3D24f99a4b-1682-44da-7ba4-dba935d107d2"

  method               = "GET"

  proxy                = Nothing

  rawBody              = False

  redirectCount        = 10

  responseTimeout      = ResponseTimeoutDefault

  requestVersion       = HTTP/1.1

}

```

Thanks for the time you already invested. I am completely puzzled because I see no chance how the code could behave as it does. (From my experience that points out I am looking at the wrong place ;) )

Best

Jan

[David Turner]

Ok, so there's definitely a difference of some kind client-side. It'd be useful to see the network traffic nonetheless, as the source of the '!D(MISSING)' is still a bit of a mystery and you may see something suggestive in the raw bytes.

I can't find the string 'MISSING' anywhere in servant or its dependencies, even looking in libs and binaries. Any chance of tracking its source down using `find` and `grep` and `strings`?

[Nigel Rantor]

So, yes, I think it is in your client code

Could you call both of them with *exactly* the same input?

i.e. create a single selector and call both of them with exactly that input?

I just feel like this might be a really silly typo or something somewhere.

Alternatively can you give us the code you're using so we can play with it?

n

> <mailto:dct25...@mythic-beasts.com>> schrieb am Sa., 14. Jan. 2017 um

> 20:08 Uhr:
>
> Grasping at straws a little bit here, but can you (a) do a packet
> capture (e.g. `tcpdump -X`) to see what's going back and forth on
> the wire, just to make absolutely sure all of the oddness is on the
> client's end?
>
> A string like `!D(MISSING)` sorta looks like a decoding error so
> perhaps it will be instructive to look at the bytes on the wire it's
> trying to decode, although I can't think what could be confused with
> an equals sign. It's not like that blooming Greek question mark
> (http://www.fileformat.info/info/unicode/char/037e/index.htm).
>
> Cheers,
>
>
>
>
> On 14 January 2017 at 18:47, Nigel Rantor <wig...@wiggly.org

> <mailto:wig...@wiggly.org <mailto:wig...@wiggly.org>>>

[Brandon Allbery]

On Sat, Jan 14, 2017 at 8:34 PM, Nigel Rantor <wig...@wiggly.org> wrote:

So, yes, I think it is in your client code

On very little evidence (mostly the ! prefix) I am wondering if the client is using an XML library and it's glitching for some reason.

--

brandon s allbery kf8nh                               sine nomine associates

allb...@gmail.com                                  ball...@sinenomine.net

unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

[Nigel Rantor]

On 14/01/17 21:03, Brandon Allbery wrote:
>
> On Sat, Jan 14, 2017 at 8:34 PM, Nigel Rantor <wig...@wiggly.org
> <mailto:wig...@wiggly.org>> wrote:
>
> So, yes, I think it is in your client code
>
>
> On very little evidence (mostly the ! prefix) I am wondering if the
> client is using an XML library and it's glitching for some reason.

These are query parameters within the URL that is being requested, I'm
not sure why any particular content-type being used would have an effect
here.

Regards,

[Nigel Rantor]

Just wondering if you figured out what was going weird here.