Dart HttpClient and JSON

[Nikolay Botev]

Hi,

I am trying to write a simple script in dart to retrieve and parse a simple JSON object from an HTTP server. Here is the script to do that in Python:

#!/usr/bin/python

import urllib2
import json

for n in range(11):
        contents = urllib2.urlopen("https://api.github.com/legacy/repos/search/:java?language=java&start_page=%s" % n).read()

        j = json.loads(contents)

        for item in j['repositories']:
                print item['url']

When I started trying to get this translated to Dart, I was pleasantly surprised that Dart came with batteries included - both HTTP and JSON are supported out-of-the-box. I also knew that Dart I/O was exclusively asynchronous using E-like futures. No biggie - been there, done that.

It has been almost an hour, however, and I still have not been able to get this working. Here is what I got so far. The code looks horrendous, at least compared to the Python verison. I hope I am doing something wrong and I sure hope I do not need all these error handlers, which I added in the hope of trapping any potential issues.

#!/bin/env dart

import 'dart:io';
import 'dart:uri';
import 'dart:json';

main() {

  for (int i = 1; i <= 1; i++) {
    var url = "https://api.github.com/legacy/repos/search/:java?language=java&start_page=${i}";
    print("Getting ${url}");
    new HttpClient().getUrl(Uri.parse(url)) .then((HttpClientRequest request) {
        // Prepare the request then call close on it to send it.
        return request.close();
    }, onError: (e) => print("ERROR ${e}"))
    .then((HttpClientResponse response) {
        // Process the response.
        print("Got repsponse ${response}.");
        StringBuffer buf = new StringBuffer();
        response.transform(new StringDecoder()).listen((d) { print("data IN"); buf.append(d); },
            onError: (e) => print("ERROR ${e}"), onDone: () {
                print("Got response ${buf.length} chars long.");
                var j = parse(buf);
                for (var item in j['repositories']) {
                    print(item['url']);
                }
            });
        new StringDecoder().bind(response).fold("", (a,b) { print("${a}..."); a + b; });
    }, onError: (e) => print("ERROR ${e}"));
    //.then((String s) {
    //    print("Got string ${s}");
    //    var j = parse(s);
    //    for (var item in j['repositories']) {
    //        print(item['url']);
    //    }
    //}, onError: (e) => print("ERROR ${e}"));

  }

}

Running the above I get the following exception:

$ dart repos.dart
Getting https://api.github.com/legacy/repos/search/:java?language=java&start_page=1
Got repsponse Instance of '_HttpClientResponse@0x1da10ec4'.
#0      _SingleStreamImpl._addListener (dart:async/stream_impl.dart:301:7)
#1      _StreamImpl.listen (dart:async/stream_impl.dart:39:17)
#2      _HttpIncoming.listen (http_impl.dart:17:26)
#3      _HttpClientResponse.listen (http_impl.dart:154:25)
#4      _EventTransformStreamSubscription._EventTransformStreamSubscription (dart:async/stream.dart:477:34)
#5      EventTransformStream.listen (dart:async/stream.dart:465:12)
#6      StreamEventTransformer.bind.<anonymous closure> (dart:async/stream.dart:442:47)
#7      _SingleControllerStream._runGuarded (dart:async/stream_controller.dart:31:26)
#8      _SingleControllerStream._onSubscriptionStateChange (dart:async/stream_controller.dart:39:16)
#9      _StreamImpl._checkCallbacks (dart:async/stream_impl.dart:211:35)
#10     _SingleStreamImpl._addListener (dart:async/stream_impl.dart:308:22)
#11     _StreamImpl.listen (dart:async/stream_impl.dart:39:17)
#12     Stream.fold (dart:async/stream.dart:111:31)
#13     main.<anonymous closure> (file:///C:/Users/Nikolay/Downloads/repos.dart:28:48)
#14     _ThenFuture._sendValue (dart:async/future_impl.dart:265:24)
#15     _FutureImpl._setValue (dart:async/future_impl.dart:149:26)
#16     _FutureListenerWrapper._sendValue (dart:async/future_impl.dart:65:21)
#17     _FutureImpl._setValue (dart:async/future_impl.dart:149:26)
#18     _FutureImpl._setOrChainValue (dart:async/future_impl.dart:239:16)
#19     _ThenFuture._sendValue (dart:async/future_impl.dart:271:21)
#20     _FutureImpl._setValue (dart:async/future_impl.dart:149:26)
#21     _AsyncCompleter._setFutureValue.<anonymous closure> (dart:async/future_impl.dart:29:23)
#22     _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:15:17)
#23     Timer.run.<anonymous closure> (dart:async/timer.dart:17:21)
#24     Timer.Timer.<anonymous closure> (dart:async-patch/timer_patch.dart:9:15)

Unhandled exception:
Bad state: Stream already has subscriber.
#0      _throwDelayed.<anonymous closure> (dart:async/stream_impl.dart:22:5)
#1      _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:15:17)
#2      _asyncRunCallback._asyncRunCallback (dart:async/event_loop.dart:25:9)
#3      Timer.run.<anonymous closure> (dart:async/timer.dart:17:21)
#4      Timer.run.<anonymous closure> (dart:async/timer.dart:25:13)
#5      Timer.Timer.<anonymous closure> (dart:async-patch/timer_patch.dart:9:15)
#6      _Timer._createTimerHandler._handleTimeout (timer_impl.dart:99:28)
#7      _Timer._createTimerHandler._handleTimeout (timer_impl.dart:107:7)
#8      _Timer._createTimerHandler.<anonymous closure> (timer_impl.dart:115:23)
#9      _ReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:81:92)


The commented out code at the end was my first attempt at this, however that listener always got null values:

$ dart repos.dart
Getting https://api.github.com/legacy/repos/search/:java?language=java&start_page=1
Got repsponse Instance of '_HttpClientResponse@0x1da10ec4'.
Got string null
Uncaught Error: The null object does not have a getter 'length'.

Please help.

[Søren Gjesse]

The error you are getting is caused by listening on the HttpClientResponse stream twice. The response.transform(...) and the new StringDecoder().bind(response) both listens on the stream. You should do this:

  response.transform(new StringDecoder()).listen(

      (d) { buf.write(d); },

      onError: (e) => print("ERROR ${e}"),

       onDone: () {

      print("Got response ${buf.length} chars long.");

  });

or use fold like this:

  response.transform(new StringDecoder()).fold(new StringBuffer(), (a, b) => a..write(b)).then((buf) { print("Got response ${buf.length} chars long."); });

This is the same as:

  new StringDecoder().bind(response).fold(new StringBuffer(), (a, b) => a..write(b)).then((buf) { print("Got response ${buf.length} chars long."); });

Note that we now have the HttpBodyHandler to do most of the work - like this:

  .then(HttpBodyHandler.processResponse)

  .then((HttpClientResponseBody body) {

    print("Received {body.body}");

  });

See http://api.dartlang.org/docs/releases/latest/dart_io/HttpBodyHandler.html.

Regards,

Søren

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new
 
 

[James Ots]

I would do something like this, using the http package:

import 'package:http/http.dart' as http;

import 'dart:json';

main() {

  for (var i = 1; i <= 11; i++) {

    http.get("https://api.github.com/legacy/repos/search/:java?language=java&start_page=$i",

        headers: {'User-Agent': 'Super-Duper-User-Agent'}).then((response) {

      var j = parse(response.body);

      for (var item in j['repositories']) {

        print(item['url']);

      }

    });

  }

}

It doesn't quite work for me - I get a message about not being able to get more than 1000 results, but I think it's in the right kind of direction.

[Lukas Renggli]

You get the future of HttpClientResponse response from
"request.close()", not from the request.

Maybe not the simplest way, but this works for me:

import 'dart:io';
import 'dart:uri';
import 'dart:json';

void main() {
for (int i = 0; i < 11; i++) {
var uri = Uri.parse('https://api.github.com/legacy/repos/search/:java?language=java&start_page=$i');
new HttpClient().getUrl(uri).then((HttpClientRequest request) {
request.headers.add("User-Agent", "Dart");
request.close().then((HttpClientResponse response) {
response.transform(new
StringDecoder()).toList().then((List<String> data) {
var json = parse(data.join());
for (var repo in json['repositories']) {
print(repo['url']);
}
});
});
});
}
}

Lukas

> --
> For other discussions, see https://groups.google.com/a/dartlang.org/
>
> For HOWTO questions, visit http://stackoverflow.com/tags/dart
>
> To file a bug report or feature request, go to http://www.dartbug.com/new
>
>



--

Lukas Renggli
www.lukas-renggli.ch

[chl]

Well the way i see it, the io package still need much improvement.

Heres a helper function that returns the response as as string:

Future<String> getAsString(Uri uri) {

 HttpClient client = new HttpClient();

 return client.getUrl(uri)

     .then((HttpClientRequest request) => request.close())

     .then((HttpClientResponse resp) => resp.transform(new StringDecoder()).fold(new StringBuffer(), (buf, next) {

       buf.write(next);

       return buf;

     }))

     .then((StringBuffer buf) => buf.toString());

}

with this helper function your code would look something like this:

 

import 'dart:io';
import 'dart:uri';
import 'dart:json';

main(){

  for(var i = 0; i <= 1; i++) {
    var url = "https://api.github.com/legacy/repose/search/:java?langauge=java&start_page=${i}";
    getAsString(Uri.parse(url))
      .then(parse)
      .then(process)
      .catchError((err) => print("Error $err"));    
  }
}

void process(jsonData){
  jsonData["repositories"].map((item) => item["url"]).forEach(print);
}

[Zdeslav Vojkovic]

this could be simplified with:

client.get(...)
    .then((HttpClientRequest response) => response.close())
    .then(HttpBodyHandler.processResponse)
    .then((HttpClientResponseBody body) {
      ...
    });

--

[Shailen Tuli]

Nikolay, would you please ask this question on StackOverflow? We are trying to use this forum for miscellaneous Dart

discussions.  'How-to' questions should be asked on StackOverflow.  Thanks.

--

For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new
 
 

--
Developer Programs Engineer (Dart)

Developer Relations

Google

developers.google.com

[Nikolay Botev]

Shalien, sure. I will repost this on stackoverflow and use that channel for how-to questions in the future.

Sent from my iPhone

[Seth Ladd]

Hi James,

Thanks, the http package is useful for exactly this reason. :)

The message you're seeing is from Github.  @Nikolay, do you have a URL form the API that works? I'd love to see this work. :)

Thanks,

Seth