How do you CORS with elm-reactor? (0.15 + elm-http)

[Hassan Hayat]

Before I ask my question, allow me to avoid the XY-problem by stating what I want to do.

I'm trying to implement a little autocompleter that gets data from wikipedia like this one: http://swannodette.github.io/2013/08/17/comparative/

If you follow what is done in the blog post, you get to the point where you make the url to get your data from wikipedia:

(def base-url
  "http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=")

So, all you need is send a request with this url, appended with whatever it is you wanna search (for example "cat").

That's when Google Chrome starts to scold me like I was some criminal or something:

XMLHttpRequest cannot load http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=cat. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access.

I then looked at the docs for elm-http, and there seemed to be examples for cross origin stuff (as you can see here), but then I tried it and I get the same result except with an added

Refused to set unsafe header "Origin"

I don't know much about how Http works and what's this whole CORS stuff. I sorta get that it's because getting things from other websites is bad... yet you can totally import scripts and images and html and json from other places and that's somehow ok.... I'm confused.  


TL;DR: How do I get the JSON from this site: http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=cat? I'm using elm-reactor.

[John Orford]

The host website's server has to send back an 'access control allow origin' header.

It looks like Wikipedia does not do this.

No idea why browsers should care about that header, but that's just the way it is. Must be some non-obvious logic behind it.

--
You received this message because you are subscribed to the Google Groups "Elm Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elm-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Mayer]

Basically, the Wikipedia API does not explicitly tell your browser that it is safe to access the Wikipedia API from other sites. Your browser (being conservatively security-minded) decides that it's probably not a good idea to contact the Wikipedia API just because this other website told it to. This is to prevent myevilsite.com from making a request to the Wikipedia API that, for instance, replaces your Wikipedia user profile with a picture of a cat.

--

[Max Goldstein]

This is not an Elm or Task thing but a part of how the web works. Here is Mozilla's page.

It's unfortunate, since you're stuck getting data from the server and sites that specifically allow cross-origin access.

The long-term solution is to have the runtime proxy all HTTP requests through elm-reactor, which is not restricted by the cross-origin policy. Of course, this will break when you don't use the reactor.

For your particular project, Wikipedia may well have an API with CORS enabled. But if the main site does not, there's nothing you can do. Sorry.

[Hassan Hayat]

Cool. Thanks guys. That makes sense.

Ok, so I'm reading the mediawiki (wikipedia) API docs, specifically the part about CORS: http://www.mediawiki.org/wiki/API:Cross-site_requests

For a CORS request to be allowed by the remote wiki, $wgCrossSiteAJAXdomains must be set appropriately to allow the origin site. The MediaWiki API also requires that the origin be supplied as a request parameter, appropriately named "origin", which is matched against the Origin header required by the CORS protocol. Note that this header must be included in any pre-flight request, and so should be included in the query string portion of the request URI even for POST requests.

If the CORS origin check passes, MediaWiki will include the Access-Control-Allow-Credentials: true header in the response, so authentication cookies may be sent.

On Wikimedia wikis CORS is enabled since September 2012; as of October 2013 CORS pre-flight requests are also supported.

I think that $wgCrossSiteAJAXdomains is a PHP thing.  Maybe all this makes sense to someone here.

I get the whole part that they'll include a header at the end to say "yay, you're we both recognized each other as legit", but I still don't get how to actually get the data from that site to my Elm program. 

[Max Goldstein]

That wg variable is server-side; you're not configuring a MediaWiki server so don't worry about it. Poking around, I found this: https://www.mediawiki.org/wiki/Manual:CORS

It has an example that you can copy from. Hopefully elm-http gives you access to all the things you need. Be sure to change "origin" to the name of your site (possibly your IP, or try "*"?). I would suggest getting their JS example to work before porting to Elm.

[Hassan Hayat]

I tried messing with that "origin" thing but I always get the same answer

Refused to set unsafe header "Origin"

(regardless how i spell it, i get that it's unsafe).

I also tried setting it to my IP and I even found that I can alias localhost to have some other name and same thing...

[Dobes Vandermeer]

You can't set the header yourself, the browser will do that.  However you should be able to add that query string parameter yourself.

--

[Hassan Hayat]

Ok, for clarification, here's all of my current code:

As you can imagine, I've been messing around a lot with the localhost. I even tried putting my IP address instead of localhost.

import Html                   exposing (Html, div, text, input)

import Html.Attributes

import Html.Events            exposing (on, targetValue)

import Http                   exposing (Error, get)

import Task                   exposing (Task, succeed, fail, andThen, mapError)

import Json.Decode as Decode  exposing (Decoder, tuple4, list, string)

import Signal                 exposing (Signal, Mailbox, mailbox, message, send)

import List

----

crossOriginGet : String -> String -> Task Http.RawError Http.Response

crossOriginGet origin url =

  Http.send Http.defaultSettings

    { verb = "GET"

    , headers =

      [ ("Origin", origin)

      , ("Content-Type", "application/json; charset=UTF-8")

      ]

    , url = url

    , body = Http.empty

    }

corsGet : String -> String -> Decoder value -> Task Error value

corsGet origin url decoder =

  Http.fromJson decoder (crossOriginGet origin url)

----

type alias WikipediaRequestJson =

  ( String

  , List String

  , List String

  , List String

  )

type alias WikipediaSearchTermRequest =

  { term    : String

  , results : List WikipediaSearchTermResult

  }

type alias WikipediaSearchTermResult =

  { term        : String

  , description : String

  , url         : String

  }

emptyRequest : WikipediaSearchTermRequest

emptyRequest =

  WikipediaSearchTermRequest "" []

fromJson : WikipediaRequestJson -> WikipediaSearchTermRequest

fromJson (term, results, descriptions, urls) =

  WikipediaSearchTermRequest term

    (List.map3 WikipediaSearchTermResult results descriptions urls)

wikipediaRequest : Decoder WikipediaRequestJson

wikipediaRequest =

  tuple4 (,,,) string (list string) (list string) (list string)

getWikipediaSearchTerm : String -> Task Error ()

getWikipediaSearchTerm term =

  let

      wikiurl =

        "http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=" ++ term

  in corsGet "localhost:8000" wikiurl wikipediaRequest

    `andThen` (fromJson >> send wikipediaTermMailbox.address)

termSearchMailbox : Mailbox (Task Error ())

termSearchMailbox = mailbox (succeed ())

wikipediaTermMailbox : Mailbox WikipediaSearchTermRequest

wikipediaTermMailbox = mailbox emptyRequest

model : Signal WikipediaSearchTermRequest

model = wikipediaTermMailbox.signal

view model =

  div []

    [ input

        [ on "input" targetValue (message termSearchMailbox.address << getWikipediaSearchTerm)]

        []

    , text (toString model)

    ]

port termSearch : Signal (Task Error ())

port termSearch = termSearchMailbox.signal

main : Signal Html

main = Signal.map view model

[John Mayer]

You can set an origin field in the querystring

https://en.wikipedia.org/w/api.php

https://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=cat&origin=http://client.cors-api.appspot.com

However, your origin might not be whitelisted by wikipedia (it most likely is not), so this would not work either. In that case, what you are trying to do is not possible.

[Hassan Hayat]

Oh well, at least I tried. Thanks.

[ma...@murrayh.id.au]

Hello,

One other alternative is to use JSONP (http://en.wikipedia.org/wiki/JSONP), which the wikipedia API supports.

For example:
http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=example&callback=handle_wikipedia_response

I would be interested to see how that would be implemented using Elm, considering that you need to:

  1. Define a global level function to handle the response data; and
  2. Write a <script> element to the document in order to initiate the request

Cheers,
Murray

[Laszlo Pandy]

JSONP it requires having global functions and so it doesn't scale well
in large applications. It's pretty awful to use even in JavaScript so
I don't think Elm could or should ever support it.

CORS was invented so we don't have to do use JSONP.

[Hassan Hayat]

I looked into JSONP and you can tell that it's terrible for security from a mile away. You give it a callback and it gives you back a script tag you have to load dynamically.... scary....I'd rather just get pure JSON. I also give my vote for never officially supporting JSONP. 

[Dobes Vandermeer]

Well, I would give a counterpoint that if you (and your users) can trust the API you are consuming, JSONP is acceptable and gets the job done.  Keep in mind that you may already be "injecting" scripts like Google Analytics, jQuery, and so on from a third-party site you trust, and that's exposing you to basically the attack vector where someone can compromise that site or launch a man in the middle attack to inject javascript code.

As for global callbacks, you can reuse a single callback using techniques such putting an array index into the callback name, for example:

http://en.wikipedia.org/w/api.php?action=opensearch&format=json&search=example&callback=handle_wikipedia_response[1]

So, it's not so much an issue of scalability or global callbacks.  It is about security to some degree - if your app has private information in it then you should restrict the use of JSONP to sites whose security you can trust.  If you feel a higher level of security is needed and CORS isn't available you'll have to use a proxy of some sort, or use Flash to bypass the CORS restriction.

--