extractOpt weird behavior or bug?

Gilberto Garcia Jr

unread,

Jul 20, 2017, 1:36:22 PM7/20/17

to lif...@googlegroups.com

The following returns a None, as expected.

val json = JsonParser.parse("{}")
(json \ "foo").extractOpt[String]

However, I was perplexed to find out that (json \ "foo").extractOpt[List[String]] returns Some(List()).
I was expecting that no matter what I'm trying to extract from the json, if the key is missing, a None would return.

Looking at the code I've found that when dealing with collections, the extractor treats JNothing values as an empty JArray.

This raised some questions:
1) Is this the intended behavior or a bug?
2) If it is indeed the intended behavior, why extracting a missing key from a json should result on an empty collection instead of None?

Best,

--
Gilberto Garcia Jr || ggarcia.me || ggarciajr (github)

Matt Farmer

unread,

Jul 20, 2017, 1:55:50 PM7/20/17

to lif...@googlegroups.com

This is intended behavior.

This happens for the same reason that extractOpt[Option[String]] would return a None - collections are effectively treated as optional components that can default to an empty value across the board by lift-json. The reasoning from my point of view is that the semantic difference between "there is not a key" and "the list is empty" doesn't seem that wide. I can't speak for the reasoning when the behavior was originally implemented.

At a technical level, the AST to Object mapping engine just doesn't distinguish between these types and making it do so would be quite difficult at this point.

You can guard against this case in particular by tacking .filter(_.nonEmpty) onto the end.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matt Farmer

unread,

Jul 20, 2017, 1:56:57 PM7/20/17

to lif...@googlegroups.com

Oh before I forget, I would be interested if folks think this shouldn't be the behavior. Though we wouldn't make such a change to the existing mapping engine / extraction engine, it is a change we could produce in the new engine I've started writing at https://github.com/liftmodules/json-extractor-ng

Torsten Uhlmann

unread,

Jul 20, 2017, 2:27:10 PM7/20/17

to lif...@googlegroups.com

I ran into this problem together with Gilberto and he was kind enough to post it here.

For me it was rather unexpected. As I understand it, the reason for the Option is to indicate something is there or not.

I would expect extractOpt'ing a String that's not there to result in a None, I would expect the same for extracting any type, List[String] in this case.

But as you said the workaround is quiet easy so it's not really a big thing. But I would guess for many that behavior comes unexpected. And for me a year from now it might again come unexpected :)

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

AGYNAMIX(R). Passionate Software.

Inh. Torsten Uhlmann | Buchenweg 5 | 09380 Thalheim
Phone:    +49 3721 273445
Fax:    +49 3721 273446
Mobile:    +49 151 12412427
Web:   http://www.agynamix.de

Author of "Lift Web Applications How-To"

Antonio Salazar Cardozo

unread,

Jul 20, 2017, 3:09:53 PM7/20/17

to Lift

For me there's a slightly different explanation that sort of pipes into Matt's for why

this happens. This is a behavior that is strictly relevant to `extractOpt`, rather than

generically to optionals. For example:

@ case class Mackarel(datList: Option[List[String]])

defined class Mackarel

@ val json = JsonParser.parse("""{}""")

json: JValue = JObject(List())

@ json.extract[Mackarel]

res6: Mackarel = Mackarel(None)

@ json.extractOpt[Mackarel]

res7: Option[Mackarel] = Some(Mackarel(None))

When a part of a case class is defined as optional, we go through the regular optional

extraction handling and extract an absent field as a `None`. The reason `extractOpt`

doesn't do this is because `extractOpt` doesn't mean “extract this type inside an Option`,

but rather “try to extract this type and return it, or return a None if we cannot extract

the type”. That is, `extractOpt` is designed not as sugar for extracting an optional, but

as sugar for not blowing up if a given object is an invalid thing to extract. This is why

my personal preference is to wrap a regular `extract` in `tryo` than to use `extractOpt`,

since `extractOpt` won't tell you why it failed to extract.

Anyway, this sends us down the path of "what counts as a valid List”. As Matt mentioned,

when something doesn't exist, lift-json will plug in an empty list. In terms of `extract`, this

means:

@ JNothing.extract[List[String]]

res18: List[String] = List()

@ JNothing.extractOpt[List[String]]

res19: Option[List[String]] = Some(List())

@ JNothing.extract[Option[List[String]]]

res20: Option[List[String]] = None

Hope that helps add some extra color! (Also, `json \ "<nonexistent field>"` gives JNothing.)

Thanks,

Antonio

Diego Medina

unread,

Jul 20, 2017, 4:27:58 PM7/20/17

to Lift

I also work with Gilberto and Torsten, and I understand the explanation about why it is this way, I personally also see extractOpt as "don't blow up on my client's face", but if I don't send a value, I would not expect an Empty List, I would expect None. (Or blowing up if I call extract)

Thanks

Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

Matt Farmer

unread,

Jul 20, 2017, 4:44:39 PM7/20/17

to Lift

Good to know. What would be the sensible behavior while extracting a case class? If an empty list is declared and they key isn't present we currently drop in an empty list. Should we start blowing up when that occurs?

Antonio

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
AGYNAMIX(R). Passionate Software.
Inh. Torsten Uhlmann | Buchenweg 5 | 09380 Thalheim
Phone:    +49 3721 273445
Fax:    +49 3721 273446
Mobile:    +49 151 12412427
Web:   http://www.agynamix.de
Author of "Lift Web Applications How-To"

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--

--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

Diego Medina

unread,

Jul 20, 2017, 4:51:39 PM7/20/17

to Lift

I think so, if I wanted to make a key optional, I would make it Option[List[String]]

Antonio

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
AGYNAMIX(R). Passionate Software.
Inh. Torsten Uhlmann | Buchenweg 5 | 09380 Thalheim
Phone:    +49 3721 273445
Fax:    +49 3721 273446
Mobile:    +49 151 12412427
Web:   http://www.agynamix.de
Author of "Lift Web Applications How-To"

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Matt Farmer

unread,

Jul 21, 2017, 9:30:06 AM7/21/17

to Lift

I've filed a new issue on the new extractor: https://github.com/liftmodules/json-extractor-ng/issues/1

Please feel free to leave additional thoughts here. I'm still not sure where I land on this issue.

Antonio

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
AGYNAMIX(R). Passionate Software.
Inh. Torsten Uhlmann | Buchenweg 5 | 09380 Thalheim
Phone:    +49 3721 273445
Fax:    +49 3721 273446
Mobile:    +49 151 12412427
Web:   http://www.agynamix.de
Author of "Lift Web Applications How-To"

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

Antonio Salazar Cardozo

unread,

Jul 21, 2017, 5:09:18 PM7/21/17

to Lift

Part of what I was getting at is that an Option[List], during extraction, behaves as

expected---i.e., extracting nothing to an Option[List] gives you a None. The only

thing that's isn't necessarily what you'd expect is that extracting to a List when

you have nothing for it gives you an empty list. To me this feels like behavior that

isn't immediately expected, but is more likely to be helpful than harmful.

On the flip side I'm not super-fanatical about it, so if we want to look at changing

that behavior in an experimental extractor like Matt's, potentially for inclusion into

a hypothetical Lift 4 release, I'm not super-opposed.

Thanks,

Antonio

Antonio

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
AGYNAMIX(R). Passionate Software.
Inh. Torsten Uhlmann | Buchenweg 5 | 09380 Thalheim
Phone:    +49 3721 273445
Fax:    +49 3721 273446
Mobile:    +49 151 12412427
Web:   http://www.agynamix.de
Author of "Lift Web Applications How-To"

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Diego Medina
Go Consultant
di...@fmpwizard.com
https://blog.fmpwizard.com/

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+unsubscribe@googlegroups.com.

Reply all

Reply to author

Forward