How to deal with Optional Paths or how to use $[*]?

[Anderson Vaz]

Hi,

I make this question on GitHub: https://github.com/jayway/JsonPath/issues/78 and on the answer that I've received I was oriented to post here this kind of questions.

In the answer on Github I was recommend to use $[*] and Option.DEFAULT_PATH_LEAF_TO_NULL but it didn't solve my problem, I think that I'm doing something wrong when using this suggestion and so, if, please, someone could help me I appreciate.

I already had tried the suggestion in your answer, even with configuration, and I still getting `com.jayway.jsonpath.PathNotFoundException: No results for path: $['location']['crossStreet']`, but I think that I've not posted all the code that I'm using, and so, maybe, it have others misuse of JsonPath. What I'm trying to accomplish is parse just a small part of FoursquareAPI Venues Search JSON response, that is published is this URL https://api.foursquare.com/v2/venues/search?ll=40.7,-74, and have this response (just a part):

{

    meta: {
        code: 200
    }
    notifications: [
        {
            type: "notificationTray"
            item: {
                unreadCount: 0
            }
        }
    ]
    response: {
        venues: [
            {
                id: "430d0a00f964a5203e271fe3"
                name: "Brooklyn Bridge Park"
                contact: {
                    phone: "+12128033822"
                    formattedPhone: "+1 212-803-3822"
                    twitter: "nycparks"
                    facebook: "104475634308"
                    facebookUsername: "BartowPell"
                    facebookName: "Bartow-Pell Mansion Museum"
                }
                location: {
                    address: "Main St"
                    crossStreet: "Plymouth St"
                    lat: 40.70227697066692
                    lng: -73.9965033531189
                    distance: 389
                    postalCode: "11201"
                    cc: "US"
                    city: "Brooklyn"
                    state: "NY"
                    country: "United States"
                    formattedAddress: [
                        "Main St (Plymouth St)"
                        "Brooklyn, NY 11201"
                        "United States"
                    ]
                }
                categories: [
                    {
                        id: "4bf58dd8d48988d163941735"
                        name: "Park"
                        pluralName: "Parks"
                        shortName: "Park"
                        icon: {
                            prefix: "https://ss3.4sqi.net/img/categories_v2/parks_outdoors/park_"
                            suffix: ".png"
                        }
                        primary: true
                    }
                ]
                verified: true
                stats: {
                    checkinsCount: 28108
                    usersCount: 17545
                    tipCount: 171
                }
                url: "http://nyc.gov/parks"
                specials: {
                    count: 0
                    items: [ ]
                }
                hereNow: {
                    count: 13
                    summary: "13 people are checked in here"
                    groups: [
                        {
                            type: "others"
                            name: "Other people here"
                            count: 13
                            items: [ ]
                        }
                    ]
                }
                referralId: "v-1427653325"
            }
            {
                id: "51eabef6498e10cf3aea7942"
                name: "Brooklyn Bridge Park - Pier 2"
                contact: { }
                location: {
                    address: "Furman St"
                    crossStreet: "Brooklyn Bridge Park Greenway"
                    lat: 40.69956454780675
                    lng: -73.99835740533105
                    distance: 146
                    cc: "US"
                    city: "Brooklyn"
                    state: "NY"
                    country: "United States"
                    formattedAddress: [
                        "Furman St (Brooklyn Bridge Park Greenway)"
                        "Brooklyn, NY"
                        "United States"
                    ]
                }
                categories: [
                    {
                        id: "4bf58dd8d48988d163941735"
                        name: "Park"
                        pluralName: "Parks"
                        shortName: "Park"
                        icon: {
                            prefix: "https://ss3.4sqi.net/img/categories_v2/parks_outdoors/park_"
                            suffix: ".png"
                        }
                        primary: true
                    }
                ]
                verified: false
                stats: {
                    checkinsCount: 1478
                    usersCount: 1138
                    tipCount: 10
                }
                specials: {
                    count: 0
                    items: [ ]
                }
                hereNow: {
                    count: 1
                    summary: "One person is checked in here"
                    groups: [
                        {
                            type: "others"
                            name: "Other people here"
                            count: 1
                            items: [ ]
                        }
                    ]
                }
                referralId: "v-1427653325"
            }        ]
        confident: true
    }
}

I want to get only some Venues information from this JSON, like `id`, `name`, `location` and the `name` of first item of `categories` array. I'm trying to use the approach of read the json once and work with an Object Document in memory. As result, I want a `List<Venue>`, so my code is like:

    URI uri = new URI( String.format( "https://api.foursquare.com/v2/venues/search" +
                    "?ll=%s&v=20140806&m=foursquare&categoryId=4d4b7105d754a06374d81259",
            String.format( "%s,%s", latitude, longitude ) ) );
    final Configuration build = Configuration.builder().jsonProvider( new JacksonJsonProvider() )
            .mappingProvider( new JacksonMappingProvider() ).build();
    final Object document = build.jsonProvider().parse( uri.toURL().openConnection().getInputStream(),
            Charset.defaultCharset().name() );
    final List<Object> list = JsonPath.read( document, "$.response.venues" );
    final List<Venue> venues = list.stream().collect( mapping( ( Object i ) -> {
      Venue venue = new Venue();
      venue.setFoursquareId( JsonPath.read( i, "$.id" ).toString() );
      venue.setName( JsonPath.read( i, "$.name" ).toString() );
      Location location = new Location();
      location.setStreet( JsonPath.read( i, "$.location.address" ).toString() );
      location.setCrossStreet( JsonPath.read( i, "$.location.crossStreet" )  );
      location.setPostalCode( JsonPath.read( i, "$.location.postalCode" ).toString() );
      location.setCity( JsonPath.read( i, "$.location.city" ).toString() );
      location.setState( JsonPath.read( i, "$.location.state" ).toString() );
      location.setCountry( JsonPath.read( i, "$.location.country" ).toString() );
      location.setGeoCode( new GeoCode( "point", new double[]{
              JsonPath.read( i, "$.location.lat" ),
              JsonPath.read( i, "$.location.lng" )
      } ) );
      final List<Object> categories = JsonPath.read( i, "$.categories" );
      final Optional<Object> categoryJson = categories.stream().findFirst();
      if ( categoryJson.isPresent() ) {
        venue.setType( JsonPath.read( categoryJson.get(), "$.name" ) );
      }
      return venue;
    }, toList() ) );

When `crossStreet` is return in response everything works and when is not returned `PathNotFound` is raised. I already tried the following variations of above code:

    final Configuration build = Configuration.builder().options(
Option.DEFAULT_PATH_LEAF_TO_NULL ).jsonProvider( new
JacksonJsonProvider() )
            .mappingProvider( new JacksonMappingProvider() ).build();

Raise `PathNotFound`
And with `Option.DEFAULT_PATH_LEAF_TO_NULL` and:

location.setCrossStreet( JsonPath.read( i, "$[*].location.crossStreet" ) );

Raise `com.jayway.jsonpath.PathNotFoundException: Property ['location'] not found in path $['id']`.

I'm new to this library and can't figure out where I'm misusing it.

Can someone help?
Thanks for attention.

[kalle stenflo]

Hi,

You don't need Option.DEFAULT_PATH_LEAF_TO_NULL in the code above. 

Your sample JSON is not valid and that makes it hard to help. As long as location is not null I don't see why your code would fail. 

       

URI uri = URI.create("https://api.foursquare.com/...");

List<Object> jsonVenues = JsonPath.parse(uri.toURL().openStream()).read("$.response.venues");

List<Venue> venues = jsonVenues.stream().map(o -> {

   DocumentContext ctx = JsonPath.parse(o);
   String id = ctx.read("$.id", String.class);
   String name = ctx.read("$.name", String.class);

   return new Venue(id, name, ......);
}).collect(Collectors.toList());

You should consider using the ObjectMapping features of Jackson if you are in control of your POJO:s. Maybe you could end up with

URI uri = URI.create("https://api.foursquare.com/...");

List<Venue> venues = JsonPath.parse(uri.toURL().openStream())
   .read("$.response.venues", List.class)
   .stream()
   .map(o -> JsonPath.parse(o).read("$", Venue.class))
   .collect(Collectors.toList());

[Anderson Vaz]

Hi kalle,

Thanks for you answer and I’m really sorry for the late response. I couldn’t do this earlier.

I restructured my code as your suggestion and maybe I’m still missing something, because I’m still receiving an error. The error that I’m receiving now is:

com.jayway.jsonpath.PathNotFoundException: Property ['location'] not found in path $['id']

Unfortunately, I modify my POJO because it represent a Venue concept on my App that the information will be consumed from  several Rest Services, I now that I can create some POJO’s for just consume foursquare and parse them to my main POJO’s but I’ve an intention to do the foursquare parse in one trip.

I’ve published a simple project on github showing the issue. As I said, I need to parse the path ‘$.location.crossStreet’ that is a path that exists for some documents and don’t exist for others.

The client id and secret published on the github is from a test app and represent no risk.

https://github.com/avaz/json-path-test

Thanks for your help.

Best regards,

Anderson Vaz

Anderson Vaz

http://twitter.com/aavaz

--
You received this message because you are subscribed to a topic in the Google Groups "JsonPath" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jsonpath/yPurMLMbB2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jsonpath+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kalle Stenflo]

This works:

package json;

import com.jayway.jsonpath.Configuration;

import com.jayway.jsonpath.DocumentContext;

import com.jayway.jsonpath.JsonPath;

import com.jayway.jsonpath.Option;

import com.jayway.jsonpath.spi.json.JacksonJsonProvider;

import com.jayway.jsonpath.spi.json.JsonProvider;

import com.jayway.jsonpath.spi.mapper.JacksonMappingProvider;

import com.jayway.jsonpath.spi.mapper.MappingProvider;

import org.junit.Test;

import java.io.IOException;

import java.net.URI;

import java.net.URISyntaxException;

import java.util.EnumSet;

import java.util.List;

import java.util.Map;

import java.util.Set;

import static java.util.stream.Collectors.mapping;

import static java.util.stream.Collectors.toList;

import static org.hamcrest.CoreMatchers.is;

import static org.junit.Assert.assertThat;

/**

 * Created by Anderson on 2/4/15.

 */

public class JsonPathFoursquareTest {

    static {

        Configuration.setDefaults(new Configuration.Defaults() {

            private final JsonProvider jsonProvider = new JacksonJsonProvider();

            private final MappingProvider mappingProvider = new JacksonMappingProvider();

            private final Set<Option> options = EnumSet.of(Option.DEFAULT_PATH_LEAF_TO_NULL);

            public JsonProvider jsonProvider() {

                return jsonProvider;

            }

            @Override

            public MappingProvider mappingProvider() {

                return mappingProvider;

            }

            @Override

            public Set<Option> options() {

                return options;

            }

        });

    }

    @Test

    public void testParseFoursquareApi() throws IOException, URISyntaxException {

        URI uri = new URI(String.format("https://api.foursquare.com/v2/venues/search" +

                        "?client_id=%s&client_secret=%s&ll=%s&v=20140806&m=foursquare&categoryId=4d4b7105d754a06374d81259",

                "ZB1PWXEJF2ZO3LNDHQEHVU11BHDEX4QTE4ZTZHBQMT3JFZD0",

                "V4SDDITEI54LRYG3C3JTVM3AOEII5GU4HT4DVTHKITFEKYBJ",

                String.format("%s,%s", -23.588529, -46.680510)));

        final List<Object> list = JsonPath.parse(uri.toURL().openStream()).read("$.response.venues");

        final List<Map<String, Object>> venues = list.stream().collect(mapping((o) -> {

            DocumentContext ctx = JsonPath.parse(o);

            Map<String, Object> venue = ctx.read("$['id', 'name']");

            Map<String, String> location = ctx.read("$.location['address', 'crossStreet', 'postalCode', 'city', 'state', 'country']");

            venue.put("location", location);

            return venue;

        }, toList()));

        assertThat(venues.isEmpty(), is(false));

    }

}

Sent from my iPhone

You received this message because you are subscribed to the Google Groups "JsonPath" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonpath+u...@googlegroups.com.

[Anderson Vaz]

Hi Kalle,

Thanks! In this way the parse works but unfortunately it don’t fits to my needs.

The simple project that I’d shared using Map’s in the parse was just to demonstrate the error that I’m facing when using ‘$[*].location.crossStreet’ syntax. In my real code I’m not parsing using Map’s, I’m parsing straight to my POJO’s, and I can’t change my POJO’s because they are supposed to be homogenous POJO’s to be used to parse others API’s like Foursquare (i.e Yelp, Facebook, Google, etc), so with your suggestion I will need to parse to Map’s and after parse the Map’s to my POJO’s, witch is not what I’m looking for, because by this way I will need to do 'two trips’ to do the parse, JSON -> MAP’s, MAP’s -> POJO’s. I’m looking to do this in ‘one trip’, that’s the reason of why I start to use JsonPath.

There is another way to do this parse, a way that fits the code bellow?

Location location = new Location();

location.setStreet( ctx.read( "$.location.address", String.class ) );
location.setCrossStreet( ctx.read( "$[*].location.crossStreet", String.class ) );
location.setPostalCode( ctx.read( "$.location.postalCode", String.class ) );
location.setCity( ctx.read( "$.location.city", String.class ) );
location.setState( ctx.read( "$.location.state", String.class ) );
location.setCountry( ctx.read( "$.location.country", String.class ) );

 Thanks!

Best regards,

Anderson Vaz

Anderson Vaz

http://twitter.com/aavaz

[kalle stenflo]

I'm not sure what you are trying to read with the path "$[*].location.crossStreet". You are applying this to a single Venue. It does not make sense.

/Kalle

[Anderson Vaz]

Kalle,

In fact I’m not trying to read with the path ‘$[*].location.crossStreet’ at least I not started my code with it, I started to use ‘$[*]’ after your recommendation. But maybe it’s because my problem wasn’t clear.

My real problem is that I want to parse the Foursquare Venue API json result via JsonPath straight to my POJO (Venue), but the Foursquare Venue API returns the path ‘venue.location.crossStreet’ for Venues that have crossStreet and doesn’t return this path when the Venue doesn’t have crossStreet and because of that if I put this path on my code, as show bellow, JsonPath raises ‘PathNotFoundException’ for json’s that doesn’t have ‘crossStreet’.

So my real question is: There is a way to ignore paths not found and consequently avoid ‘PathNotFoundException’ with JsonPath?

Here is my currently code, if I uncomment the line with ‘$.location.crossStreet’ expression ‘PathNotFoundException’ raises:

    final List<Object> list = jsonPath.parse( uri.toURL().openStream() ).read( "$.response.venues" );
    final List<Venue> venues = list.stream().collect( mapping( ( o ) -> {
      DocumentContext ctx = jsonPath.parse( o );

      Venue venue = new Venue();

      venue.setFoursquareId( ctx.read( "$.id", String.class ) );
      venue.setName( ctx.read( "$.name", String.class ) );

      Location location = new Location();

      location.setStreet( ctx.read( "$.location.address", String.class ) );

//      location.setCrossStreet( ctx.read( "$[*].location.crossStreet", String.class ) );

      location.setPostalCode( ctx.read( "$.location.postalCode", String.class ) );
      location.setCity( ctx.read( "$.location.city", String.class ) );
      location.setState( ctx.read( "$.location.state", String.class ) );
      location.setCountry( ctx.read( "$.location.country", String.class ) );

      location.setGeoCode( new GeoCode( "Point", new double[]{
              ctx.read( "$.location.lng", Double.class ),
              ctx.read( "$.location.lat", Double.class )
      } ) );
      venue.setLocation( location );
      final List categories = ctx.read( "$.categories", List.class );
      final Optional categoryJson = categories.stream().findFirst();
      if ( categoryJson.isPresent() ) {
        DocumentContext categoryCtx = jsonPath.parse( categoryJson.get() );
        venue.setType( categoryCtx.read( "$.name", String.class ) );

      }
      return venue;
    }, toList() ) );

Thanks for your attention!

Anderson Vaz

Anderson Vaz

http://twitter.com/aavaz