issue with json parsing, how to investigate exception

[Michel Daviot]

Thanks for this excellent library, but I'm a bit in trouble with this not so clear exception...

Here is my sample code and exception, any idea how I can make some progress ? This was working with Jackson-scala ... (second version of the source file)

Exception in thread "main" java.util.NoSuchElementException: None.get

at scala.None$.get(Option.scala:322)

at scala.None$.get(Option.scala:320)

at rapture.json.jsonBackends.jackson.internal.JacksonAst$.dereferenceObject(ast.scala:79)

at rapture.data.DataType$class.$accessInnerMap(data.scala:93)

at rapture.json.Json.$accessInnerMap(json.scala:131)

at rapture.data.DataType$class.$accessWith(data.scala:99)

at rapture.json.Json.$accessWith(json.scala:131)

at net.hearthstats.core.CardData$$anon$1.construct(CardData.scala:34)

at net.hearthstats.core.CardData$$anon$1.construct(CardData.scala:34)

at rapture.data.Extractor.safeConstruct(extractors.scala:96)

at rapture.data.Extractor$$anon$3$$anonfun$construct$1.apply(extractors.scala:59)

at rapture.data.Extractor$$anon$3$$anonfun$construct$1.apply(extractors.scala:58)

at scala.collection.immutable.List.map(List.scala:274)

at rapture.data.Extractor$$anon$3.construct(extractors.scala:58)

at rapture.data.Extractor$$anon$3.construct(extractors.scala:56)

at rapture.data.Extractor.safeConstruct(extractors.scala:96)

at rapture.data.Extractor$$anon$6$$anonfun$construct$4.apply(extractors.scala:86)

at rapture.data.Extractor$$anon$6$$anonfun$construct$4.apply(extractors.scala:85)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)

at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)

at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.AbstractTraversable.map(Traversable.scala:104)

at rapture.data.Extractor$$anon$6.construct(extractors.scala:85)

at rapture.data.Extractor$$anon$6.construct(extractors.scala:83)

at rapture.data.Extractor.safeConstruct(extractors.scala:96)

at rapture.data.DataType$$anonfun$as$1.apply(data.scala:127)

at rapture.core.modes$ThrowExceptions.wrap(modes.scala:105)

at rapture.data.DataType$class.as(data.scala:125)

at rapture.json.Json.as(json.scala:131)

at net.hearthstats.core.CardData$.delayedEndpoint$net$hearthstats$core$CardData$1(CardData.scala:34)

at net.hearthstats.core.CardData$delayedInit$body.apply(CardData.scala:28)

at scala.Function0$class.apply$mcV$sp(Function0.scala:40)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.collection.immutable.List.foreach(List.scala:383)

at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)

at scala.App$class.main(App.scala:76)

at net.hearthstats.core.CardData$.main(CardData.scala:28)

at net.hearthstats.core.CardData.main(CardData.scala)

[Michel Daviot]

I forgot to mention, I'm using version 0.10.0 for scala 2.11

<dependency>

++ <groupId>com.propensive</groupId>

++ <artifactId>rapture-io_2.11</artifactId>

++ <version>0.10.0</version>

++ </dependency>

++ <dependency>

++ <groupId>com.propensive</groupId>

++ <artifactId>rapture-net_2.11</artifactId>

++ <version>0.10.0</version>

++ </dependency>

++ <dependency>

++ <groupId>com.propensive</groupId>

++ <artifactId>rapture-fs_2.11</artifactId>

++ <version>0.10.0</version>

  </dependency>

--
You received this message because you are subscribed to a topic in the Google Groups "Rapture users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rapture-users/rEpuH5rnWWc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rapture-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jon Pretty]

Hi Michel!

Sorry for taking so long to reply. That's actually a bug which is fixed in version 1.1.0, and that'll give you a more detailed explanation of what the error is. I checked it, and some of the keys you're trying to extract via the case class don't exist in one of the JSON objects it's trying to extract from.

Unfortunately, that introduces a dependency issue if you're using fs and net as well, as rapture-json-1.1.0 depends on a version of rapture-core which I haven't made corresponding rapture-net, rapture-io and rapture-fs releases for.

For now, the best I can suggest is building those projects from source. Unfortunately, you'll need to fetch each of the following projects from github/propensive/rapture-<project>, switch to the scala-2.11 branch, and then `sbt publishLocal` on each one. Here's the list:

 - mime

 - codec

 - uri

 - crypto

 - io

 - fs

 - net

Off the top of my head, this might work:

for p in mime codec uri crypto io fs net; do

  git clone g...@github.com:propensive/rapture-$p.git

  cd rapture-$p

  git checkout scala-2.11

  sbt publishLocal

  cd ..

done

Good luck!

Cheers,
Jon

--
You received this message because you are subscribed to the Google Groups "Rapture users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rapture-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jon Pretty | @propensive

[Michel Daviot]

Ouch, seems a bit complicated ... since I have no urgent need of this fix, I will wait for the official release of 1.1.0.

Actually I'm trying to  replace old json libraries in my project, so I was evaluating rapture-json in this context. Since it is public and built on travis, I cannot rely on manually built dependencies.

Meanwhile, I could try to use only rapture-json 1.1.0 since I don't really need the other depencies in the real project. Is there an equivalent of @JsonIgnoreProperties(ignoreUnknown = true)

for rapture ? So I can ignore properties which are not present in some of the nodes.

Thanks and regards

Michel

[Jon Pretty]

Possibly. What should the missing keys instantiate to in the case class? If you make those fields `Option`s, then you'll get `None` when they're absent, but of course, that means you'll have to handle the missing values later on. Alternatively, you can make them `Try`s, and you'll see the exact error for each field which doesn't successfully extract.

There's an open issue to take advantage of default initialization values in case classes if they're defined, e.g.

  case class Foo(x: Int = 7)
  json"{}".as[Foo] // should give Foo(7)

but it's not implemented yet. It shouldn't be too difficult, so this will probably get rolled into version 1.1.1.

Cheers,
Jon

[Michel Daviot]

I guess I mixed things a bit ... I'd like to ignore values in the json which are not mapped to my case class (I don't need these values in my code)

[Jon Pretty]

Hi Michel,

So, would you like to disregard the entire case class in these cases? I think the best solution would be to extract as a `List[Option[CardData]]` instead of `List[CardData]`, and then flatten or collect it.

So, the important line in your code would look like this:

  val jsonMap = Json.parse(jsonStr).as[Map[String, List[Option[CardData]]]]

  val json = jsonMap.mapValues(_.flatten)

The general idea is that any type `T` you can extract, you can extract as an `Option[T]`, and anything that would be a failure will be caught and replaced by `None`. So in your example, you could extract any of these ways:

 - json.as[Option[Map[String, List[CardData]]]]

 - json.as[Map[String, Option[List[CardData]]]]

 - json.as[Map[String, List[Option[CardData]]]]

 - move `Option`s onto the parameters of `CardData`

And you get subtly different results each time. For examlpe, in the first example, with your test data, you would just get `None` because a single failure somewhere will cause the whole thing to fail to extract.

Does that help?

Cheers,
Jon

[Michel Daviot]

Thanks a lot for your support Jon, I made it work as you suggested by putting the correct fields as Option. What I finally understood :

- a field defined in the case class MUST be in the json (or an Option in the case class)

- a field in the json MAY be in the case class (will be ignored when it's not in the class)


Btw, the error messages in 1.1.0 are still the same :

java.util.NoSuchElementException: None.get

at scala.None$.get(Option.scala:322)

at scala.None$.get(Option.scala:320)

at rapture.json.jsonBackends.jackson.internal.JacksonAst$.dereferenceObject(ast.scala:79)

In case anyone is interested, this is what I ended with :

<dependency>

<groupId>com.propensive</groupId>

<artifactId>rapture-json-jackson_2.11</artifactId>

<version>1.1.0</version>

</dependency>

-----

package net.hearthstats.core

import java.util.NoSuchElementException

import rapture.json.jsonBackends.jackson._

import rapture.json._

import rapture._

import java.io.InputStreamReader

case class CardDataTest(

  id: String,

  name: String,

  `type`: String,

  rarity: Option[String],

  cost: Option[Int],

  attack: Option[Int],

  health: Option[Int],

  text: Option[String],

  collectible: Option[Boolean],

  playerClass: Option[String],

  mechanics: Option[List[String]])

object CardDataMain extends App {

  val is = getClass.getResourceAsStream("AllSets.json")

  val json = Json.parse(io.Source.fromInputStream(is).getLines.mkString("\n")).as[Map[String, List[CardDataTest]]]

  json("Basic").filter(_.rarity == None).foreach(println)

}