PROPOSAL: Nil-safe Access.at/1

[Greg Vaughn]

I propose that the function returned from Access.at/1 special case nil such that the overall Kernel.get_in/2 call returns nil instead of raising an error.

Rationale:
I originally blamed this on Kernel.get_in/2 and I'd like to thank Eric Meadows-Jönsson for explaining the underlying reason to me on Slack.

I like to think of Kernel.get_in/2 as a nil-safe way of plucking values out of nested data structures, but I learned today that is only partially correct. The nil-safety comes from the underlying Access.get/2 calls. The docs for get_in includes:

In case any of the entries in the middle returns nil, nil will be returned as per the Access module:
iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
iex> get_in(users, ["unknown", :age])
nil

and I expected use of Access.at/1 in my keys to act similarly, but it doesn't. For example:

iex(185)> %{"items" => ["desired_value"]} |> get_in(["items", Access.at(0)])
"desired_value"
iex(186)> %{"items" => nil} |> get_in(["items", Access.at(0)])
** (RuntimeError) Access.at/1 expected a list, got: nil
(elixir) lib/access.ex:663: Access.at/4

I propose that the function returned from Access.at/1 special case nil such that the overall get_in/2 call returns nil instead of raising an error. I have not dug into the source yet but I'm happy to work up a PR if there is interest in this change.

-Greg Vaughn

[José Valim]

The proposal is reasonable however it would introduce an inconsistency since the other selectors in Access, such as Access.key, are also not nil safe. So whatever solution we choose needs to be consistent.

One possible suggestion is to introduce a "Access.maybe" that composes but composition would have to be back to front:

%{"items" => nil} |> get_in(["items", Access.at(0) |> Access.maybe])

Another idea is to introduce maybe_at, maybe_key, maybe_key! and so on. But I am not sure if this is desirable. Thoughts?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/6B6AB775-F3D5-40E5-BFBD-9852FBCBD1D0%40gmail.com.

[Manfred Bergmann]

I think that optionals, like maybe should have a very good reason.

Usually it’s either there or not. There is no maybe.
So I would rather return nil, which represents that a value is not there.

Something like:
Access.at(0) |> Access.maybe
looks quite awkward.


Just my two cents,
Manfred

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KZPZ5mpP6SSzhmq3jpuZBYA1irpmOa19UNH2fS_3QKQA%40mail.gmail.com.

[Greg Vaughn]

Thanks, José. I agree with the need to be consistent. I will look at the bigger picture, though, like Manfred I find the addition of "maybe" to be awkward, so my preference is to have the existing recommended functions in the Access module intended for use with get_in to be consistently nil safe. I'm open to more ideas, too.

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KZPZ5mpP6SSzhmq3jpuZBYA1irpmOa19UNH2fS_3QKQA%40mail.gmail.com.

[Tor Bjornrud]

I wouldn't mind having opts for something like this.  Avoids creating a slew of Access functions that then become difficult to sift through.

%{"items" => nil} |> get_in(["items", Access.at(0, nilsafe: true)

On Tuesday, January 28, 2020 at 8:10:04 PM UTC-6, Greg Vaughn wrote:

Thanks, José. I agree with the need to be consistent. I will look at the bigger picture, though, like Manfred I find the addition of "maybe" to be awkward, so my preference is to have the existing recommended functions in the Access module intended for use with get_in to be consistently nil safe. I'm open to more ideas, too.

-Greg Vaughn

> On Jan 28, 2020, at 12:45 PM, José Valim <jose...@dashbit.co> wrote:
>
> The proposal is reasonable however it would introduce an inconsistency since the other selectors in Access, such as Access.key, are also not nil safe. So whatever solution we choose needs to be consistent.
>
> One possible suggestion is to introduce a "Access.maybe" that composes but composition would have to be back to front:
>
> %{"items" => nil} |> get_in(["items", Access.at(0) |> Access.maybe])
>
> Another idea is to introduce maybe_at, maybe_key, maybe_key! and so on. But I am not sure if this is desirable. Thoughts?
>
> On Tue, Jan 28, 2020 at 7:33 PM Greg Vaughn <gva...@gmail.com> wrote:
> I propose that the function returned from Access.at/1 special case nil such that the overall Kernel.get_in/2 call returns nil instead of raising an error.
>
> Rationale:
> I originally blamed this on Kernel.get_in/2 and I'd like to thank Eric Meadows-Jönsson for explaining the underlying reason to me on Slack.
>
> I like to think of Kernel.get_in/2 as a nil-safe way of plucking values out of nested data structures, but I learned today that is only partially correct. The nil-safety comes from the underlying Access.get/2 calls. The docs for get_in includes:
>
>  In case any of the entries in the middle returns nil, nil will be returned as per the Access module:
>     iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
>     iex> get_in(users, ["unknown", :age])
>     nil
>
> and I expected use of Access.at/1 in my keys to act similarly, but it doesn't. For example:
>
> iex(185)> %{"items" => ["desired_value"]} |> get_in(["items", Access.at(0)])
> "desired_value"
> iex(186)> %{"items" => nil} |> get_in(["items", Access.at(0)])
> ** (RuntimeError) Access.at/1 expected a list, got: nil
>     (elixir) lib/access.ex:663: Access.at/4
>
> I propose that the function returned from Access.at/1 special case nil such that the overall get_in/2 call returns nil instead of raising an error. I have not dug into the source yet but I'm happy to work up a PR if there is interest in this change.
>
> -Greg Vaughn
>
> --
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/6B6AB775-F3D5-40E5-BFBD-9852FBCBD1D0%40gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

[Allen Madsen]

I'm in favor of them being nilsafe by default.

Allen Madsen
http://www.allenmadsen.com

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1ae0b9d3-9471-4750-8734-281033e9a1dc%40googlegroups.com.

[Greg Vaughn]

I just wanted to follow up and summarize here. I submitted a PR https://github.com/elixir-lang/elixir/pull/9773 with some more discussion, but the core point there was that we needed more discussion on the core list before a PR and it was closed. Nil-safety by default is undesirable in more Access functions than Access.get.

I'm exploring this on my own in my own codebase as I rework all the get_in calls I assumed were nil safe despite using Access.at. I am quite against a solution that is more verbose to gain mil safety as I use this at the edges of my system in an anti-corruption-layer. I'd rather see this implemented once, well, in the standard library than expect thousands of projects to do it themselves or bring in a 3rd party solutions to achieve it.

Feel free to discuss some more.

-Greg

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Cu%2BGBO1RWsdAjAHoaukV3w4QJPPdqqNU_miQ_%3Dv5%3DdDeQ%40mail.gmail.com.

[José Valim]

Hi Greg, I have been thinking more about this too, and I think there are some neat ways we can make this more accessible:

We could introduce Access.nillable (please suggest a better name) that you would use like this:

    get_in(root, Access.nillable([:foo, :bar, Access.at(0)]))

Basically, it traverses the path and sets all functions in the path to something that handles nil. In your apps, you can quickly encapsulate it like this:

    nillable_get_in(root, [:foo, :bar, Access.at(0)])

It is concise, backwards compatible, and clear in intent.

Thoughts?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/22988265-AB94-4666-894B-9ECF7B87905D%40gmail.com.

[Greg Vaughn]

Hi José, I have considered that as one possibility too. From Eric's comments I don't belive the Access module is the correct place to implement it though. Personally, I don't care what the module is called because I am prepared to create my own module to handle this.

What I find curious is that once we implement Kernel.nillable_get_in, why would anyone choose to use Kernel.get_in instead? Who wants to opt into nil-exceptions (Tony Hoare's billion dollar mistake)? We have other means, such as the dot syntax (my_struct.atom_key.sub_key) when we have confidence those keys exist and we want to "let it fail" if they don't. Kernel.get_in should be used only when we have tentative expectations of a data structure.

I use this when dealing with untrusted 3rd party json data on the edge of my system. Some fields that typically should be a list might be missing or present but nil until some state transition and they transition to an explicit field with a list value that Access.at can traverse.

I accept the argument that only something new like nillable_get_in is backwards compatible, but I wonder how valuable the backwards compatible get_in implementation is. There is support in this thread for mil-safety (or nil-ignorance as once presented) by default. I'm left wondering how many production systems rely upon nil exceptions during Kernel.get_in calls for proper behavior that would fail if we made it nil-safe by default.

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2B5ovo9YdQHQO2m6i%3DL_SxPKRN4O4fZejH%3DXMXfJWwWkQ%40mail.gmail.com.

[Greg Vaughn]

One more point. Even if my proposal is not accepted, these docs for Kernel.get_in really need to change

In case any of the entries in the middle returns nil, nil will be returned as
per the Access module:

iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
iex> get_in(users, ["unknown", :age])
nil

The Access module guarantees no nil-safety. It's an "accident" that Access.get does.

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2B5ovo9YdQHQO2m6i%3DL_SxPKRN4O4fZejH%3DXMXfJWwWkQ%40mail.gmail.com.

[José Valim]

> What I find curious is that once we implement Kernel.nillable_get_in, why would anyone choose to use Kernel.get_in instead?

When I don't expect anything to be nil, I want it to fail as soon as possible, instead of having nil further creeping into the system. Personally, most of the times I used get_in and friends, I am working with structured data (the opposite of your use case). If any nil shows up, it should be an error.

And changing get_in may not break code, expectations I had when I wrote the code would certainly be broken. And I would personally be unhappy if we simply changed get_in without introducing an option to write assertive code. Writing assertive code is an important of Elixir. It is why we have map.foo in addition to map[:foo]. So I think it is best to remove changing get_in from the discussion altogether, I don't see it happening.

We can continue discussing alternatives though.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4CE0D4F8-A341-4832-AC94-BDBC0D7E0911%40gmail.com.

[Greg Vaughn]

I recognize more use cases than mine. Given we will not change Kernel.get_in, I have ideas for other, less "nillable" names, such as "get_path" or "path_in" to make mil-safety less of an exceptional situation. Path expressions, as originally used in object oriented databases, typically did not raise exceptions when some data did not match expectations. This specific naming discussion can be deferred though.

I am in agreement on writing assertive code. That is the very reason I want something in the standard library that is a nil-safe navigation through untrusted input. I don't want to write an `if` or `with` dealing with each list key that might be nil, when I don't have to do it for maps. It it is the very reason I view the dot syntax as very confident keys exist vs. a get_in call which uses Access to determine existence of keys/lists.

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K-yXsZ2mxJ3sg6knRwLAFmMUky6c0G50gaBVnDpb18fA%40mail.gmail.com.

[José Valim]

For now, I don't think we should add a new function to Kernel. So we should find something that makes a path nillable for definition in Access, and then you can define get_path in your app if that's what you prefer.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/E9C59B12-1663-45E9-A8C3-163F75895D52%40gmail.com.

[Greg Vaughn]

I'm uncertain how best to proceed at this point. I have some code that is a proof of concept of what I believe is a compromise. I don't wish to open another PR prematurely and have it closed, so I'll try one more round of discussion here.

I have added 2 new Access functions: path/1 and path!/1. The first one is nil safe with a wrapper as José mentioned earlier. The second one offers consistent assertiveness, not the hybrid behavior of get_in today. Here's the doctests I started with for further discussion.

for path/1
iex> get_in(%{}, Access.path([:a, :b]))
nil

iex> get_in(%{}, Access.path([:a, Access.at(0)]))
nil

iex> get_in(%{a: nil}, Access.path([:a, Access.at(0)]))
nil

iex> get_in(%{a: []}, Access.path([:a, Access.at(0)]))
nil

for path!/1
iex> get_in(%{}, Access.path!([:a, :b]))
** (KeyError) key :a not found in: %{}

iex> get_in(%{}, Access.path!([:a, Access.at(0)]))
** (KeyError) key :a not found in: %{}

iex> get_in(%{a: nil}, Access.path!([:a, Access.at(0)]))
** (ArgumentError) Access.path!/1 encountered nil

This one is unimplemented yet, but it should raise for consistency:

iex> get_in(%{a: []}, Access.path!([:a, Access.at(0)]))
** (ArgumentError) [] has no element at index 0

For comparison, this is how get_in behaves with these cases today, highlighting the inconsistency that is the core of what bugs me. Half the cases return nil and half raise.
iex> get_in(%{}, [:a, :b])
nil

iex> get_in(%{}, [:a, Access.at(0)])

** (RuntimeError) Access.at/1 expected a list, got: nil

iex> get_in(%{a: nil}, [:a, Access.at(0)])

** (RuntimeError) Access.at/1 expected a list, got: nil

iex> get_in(%{a: []}, [:a, Access.at(0)])
nil

If this is a welcome direction, I'll be happy to submit my PR and work through details of optimization, cleanup, exception wording, etc.

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KwMg%3DHDM-B4B2Qh1irafG8_Y8tW%2BLYRTKGDoy%3DHOtkNA%40mail.gmail.com.

[José Valim]

To be honest, I am not sure either. I am not sold on Access.path/1 as a name. Using "path" makes it sound like it is a general abstraction but it isn't. Passing an Access.path to put_in or update_in won't make them suddenly accept nils because in there it is a more complex problem (you need to replace nils by actual semantic values).

So something like "nillable" or "unless_nil" is actually closer in intent IMO but I also agree those are not good names.

Also, I should have asked this sooner, but can't the complex path that you are writing be easily expressed with pattern matching?

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/D19053D7-E80B-4844-856E-2B63E6C71AE5%40gmail.com.

[Greg Vaughn]

On Feb 8, 2020, at 6:34 PM, José Valim <jose....@dashbit.co> wrote:
>
> Also, I should have asked this sooner, but can't the complex path that you are writing be easily expressed with pattern matching?

Can your use cases use pattern matching too?

Since you asked, my primary use of Kernel.get_in is when I have untrusted json at the edge of my system. I have multiple sources that have to be mapped to a common internal struct/schema. The first step is to look for the equivalent of all the keys we care about and create a map with known key names. Then we go through an Ecto changeset for validation and further processing.

I look for, let's guess, about 15 fields from each of these json payloads. I'd have to pattern match 15 times with an if statement, because if I have 14 real values but the path through the json of one of them is not present, I still want to go through the Ecto validation logic because that one key that is missing might not be critical to our business logic. Since that logic is in the next innermost layer, I don't wish to code it into this outer layer that just tries to pull what it can out of the untrusted json.

I am open to naming concerns. I do rather like the #{name} vs. #{name}! approach to highlight the inconsistency in the existing get_in behavior. I think we could call it `path_expression` which is a term used in object oriented databases and in other languages, though it seems long. You said to take modifying `get_in` off the table from consideration, but I think it leads naturally to `get_in` vs. `get_in!`.

I can accept if the decision of the core team is that my use case is an outlier and I should write my own module to handle this. I'd still hate leaving the inconsistent behavior of get_in in the standard library, but I'll adapt and move on.

-Greg Vaughn

[José Valim]

We are open to a mechanism that makes this possible but I am afraid we haven't found one yet. path_expression doesn't help, because again, it makes you think it is a general mechanism but it applies only to get_in and not the other functions. All other functions do not work with nil - so even in terms of inconsistency I am more inclined to think get_in should always fail on nil and not the opposite.

I am sorry but I cannot provide further guidance on this because I am myself not sure what the solution is. If others have suggestions, we will be glad to hear them.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/B090C168-E959-484B-ADB1-A81AB177D732%40gmail.com.

[Wiebe-Marten Wijnja]

The underlying problem to me seems that in this example `nil` is used both as a default to be returned when nothing is found and as an actual value in one of the data structures.

I do not think that finding a way to treating them as 'the same' is a good solution to the problem, because:

- It is difficult to do this in a backwards-compatible way.
- As José highlighted, there is a difference between `get_in` and the other Access-based `*_in`-calls.
- It becomes more difficult to reason about the code because the difference between the two approaches is very subtle.

Instead, I think that rather than treating this symptom (frustration at seemingly inconsistent behavior),
we should tackle the underlying cause (the behavior being consistent but confusing):

If we'd have an alternative to `get_in` that does not rely on `nil` being used as default 'nothing found' value, then the difference between the examples becomes immediately apparent.

We e.g. could introduce something named e.g. `fetch_in` that makes a clear distinction between values that are not in the nested collection vs values (like 'nil') that are there:

```

iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}

iex> fetch_in(users, ["unknown", :age])
:error

iex> %{"items" => ["desired_value"]} |> fetch_in(["items", Access.at(0)])
{:ok, "desired_value"}

# This is the important difference
# It is clear here that `nil` is there rather than the default being returned.
iex> %{"items" => nil} |> fetch_in(["items"])
{:ok, nil}

# Therefore, it now makes sense to the programmer
# that an error is raised here
iex> %{"items" => nil} |> fetch_in(["items", Access.at(0)])

** (RuntimeError) Access.at/1 expected a list, got: nil
    (elixir) lib/access.ex:663: Access.at/4

```

`fetch_in` is the most descriptive name that I thought of just now because of its return type being similar to `Access.fetch` (just like the other `*_in` functions).
However, since `Access.fetch` depends on struct-modules overloading the behaviour, we might want to search for yet another name, because it seems much more useful if we could use it for any structs.

Another problem with introducing `fetch_in` is that it adds a new function to Kernel.

Another approach (which would also tackle the aforementioned issue of wanting to use the new function on structs that do not overload `Access.fetch`) would be to introduce an option as third parameter (or keyword parameter?)
for `get_in` that would switch to explicit, ok/error-tuple-based, retrieval.

As for actually solving Greg's practical problem at hand: If you want to treat explicit `nil`'s the same way as 'the key does not exist',
then what about removing any fields that point to a `nil` value before performing your '`get_in` and friends'-based validation?

~Marten/Qqwy

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JH%3DjjhdWOkj8xErUvOPqwn4FYUvCXJcaRe4og4pUQwRA%40mail.gmail.com.

[Wiebe-Marten Wijnja]

As a quick addendum: Scratch the "because it seems much more useful if we could use it for any structs."-part of my previous message.
I mistakenly remembered that `get_in` allows you to fetch keys from structs that do not themselves follow the `Access` behaviour, which is nonsense :-).

~Marten/Qqwy

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/44a7d143-21d0-6450-3e30-d6001a6867c8%40resilia.nl.

[José Valim]

I am not sure it should be a property of the parent function. We *could* have both operations of traversal mixed:

    get_in(data, [:key, Access.at!(0), :other_key, Access.at(0), :final_key])

This would provide granular control over which paths are strict, which ones are not, which is more flexible than saying it is all strict or all lax.

The problem is that we already shipped "Access.at(0)" but it behaves like "Access.at!(0)" in the approach propose above. So we need to find out a different naming schema or a composable API, which is the Access.unless_nil, I originally proposed. The idea of having "Access.nillable([path])" was just a shortcut for making it easier on the eyes, in the same way :key today is equivalent to something like "Access.key_or_nil(:key)".

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/c0b39030-ac84-846f-d3c0-9f9a85401fd2%40resilia.nl.

[Greg Vaughn]

Replies inline

> On Feb 9, 2020, at 2:01 AM, José Valim <jose....@dashbit.co> wrote:
>
> so even in terms of inconsistency I am more inclined to think get_in should always fail on nil and not the opposite.

I agree that is an improvement over the current implementation. But that means the shortcut of supplying a key name should be wrapped with Access.fetch!/2, not Access.get/2 (which is a breaking change). And it leaves one corner case with Access.at/2 that I'm not fond of -- it cannot distinguish between `[]` and `[nil]`. If you ask for any element out of a zero length list, the assertive thing to do is raise.

> On Feb 9, 2020, at 8:30 AM, José Valim <jose....@dashbit.co> wrote:
>
> I am not sure it should be a property of the parent function. We *could* have both operations of traversal mixed:
>
> get_in(data, [:key, Access.at!(0), :other_key, Access.at(0), :final_key])
>
> This would provide granular control over which paths are strict, which ones are not, which is more flexible than saying it is all strict or all lax.

Yes. I have been thinking of something like `Access.nth/1` and `Access.nth!/1` as a "replacement" for `at` with new nil-handling behavior. This would allow for those mixed cases. However it makes fully strict not very natural. It pushes understanding of the subtleties to every developer.

Example fully strict:

get_in(data, [Access.key!(:key), Access.nth!(0), Access.key!(:other_key), Access.nth!(0), Access.key!(:final_key]))

Example fully lax:

get_in(data, [:key, Access.nth(0), :other_key, Access.nth(0), :final_key])


-Greg Vaughn

[Greg Vaughn]

Replies inline

> On Feb 9, 2020, at 6:24 AM, Wiebe-Marten Wijnja <w...@resilia.nl> wrote:
>
> The underlying problem to me seems that in this example `nil` is used both as a default to be returned when nothing is found and as an actual value in one of the data structures.

I don't think it's the leaf node that we're focused on though. The issue is the intermediate steps on the path taken through the data structure. If at any particular step you cannot make the next step, is that an exception or nil? I'd be happy if we're consistent with either approach, but the problem is that the current implementation raises if the "next step" is a list it can't find, but returns nil if it's a map it can't find.

> Instead, I think that rather than treating this symptom (frustration at seemingly inconsistent behavior),
> we should tackle the underlying cause (the behavior being consistent but confusing):

If you're saying the existing behavior is consistent, then that is the core of what I'm disagreeing with.

>
> We e.g. could introduce something named e.g. `fetch_in` that makes a clear distinction between values that are not in the nested collection vs values (like 'nil') that are there:
>
> ```
>
> iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}
> iex> fetch_in(users, ["unknown", :age])
> :error
>
> iex> %{"items" => ["desired_value"]} |> fetch_in(["items", Access.at(0)])
> {:ok, "desired_value"}
>
> # This is the important difference
> # It is clear here that `nil` is there rather than the default being returned.
> iex> %{"items" => nil} |> fetch_in(["items"])
> {:ok, nil}
>
> # Therefore, it now makes sense to the programmer
> # that an error is raised here
> iex> %{"items" => nil} |> fetch_in(["items", Access.at(0)])
> ** (RuntimeError) Access.at/1 expected a list, got: nil
> (elixir) lib/access.ex:663: Access.at/4
>
> ```

Your fetch_in approach still treats list traversals as a special case. Why does that last example not return :error as your first example does? In both cases it could not follow the path.

> As for actually solving Greg's practical problem at hand: If you want to treat explicit `nil`'s the same way as 'the key does not exist',
> then what about removing any fields that point to a `nil` value before performing your '`get_in` and friends'-based validation?

I know how I can solve it in my own codebase, but this suggestion won't solve it. I'm afraid I haven't explained the situation well enough. If get_in cannot navigate the path to the value specified, I don't want it to raise. I'd prefer it return nil, but :error would also be acceptable (or an optional `default` parameter). In my situation the inability to navigate the path should be the same as navigating the path successfully and finding an explicit `nil` at the end.


-Greg Vaughn

[Dmitry Belyaev]

On 9 February 2020 12:18:05 pm AEDT, Greg Vaughn <gva...@gmail.com> wrote:
>...

>The first step is to look for the equivalent of all the keys we care about and create a map with known key names. Then we go through an Ecto changeset for validation and further processing.

I think the problem is quite common and in my code every data type has a 'from_json' function which takes the data for the embedded struct and calls that struct 'from_json' and so on.

With Ecto it is solved by just having a 'changeset' function in the struct module which uses Ecto.Changeset.cast and Ecto.Changeset.cast_embed which in turn loads the embedded struct.

>I look for, let's guess, about 15 fields from each of these json payloads. I'd have to pattern match 15 times with an if statement,

So do you have 15 different paths now to handle different payloads? Will it be significantly better with nillable get_in?

All my code where I initially thought to use get_in/update_in, in the end I switched to direct struct pattern-matching/construction (for compile-time fields checks) and manual extraction of data using Map.get/3 (for more control).

I don't say that my way is any more convenient or better than any other. I'm just trying to suggest to look at the problem from scratch and try to apply some other solution -- maybe it will be reasonable without waiting for the standard library to be extended.

>
>-Greg Vaughn

--
Kind regards,
Dmitry Belyaev

[José Valim]

Resurrecting this old-thread. Greg, what if we had a Access.if_nil([]) that would put an empty list if there is a nil value? We could also have Access.on_nil(:error), which effectively halts.

A bit verbose but it will give you control exactly what to do and when, without coupling to the current functions.

[Greg Vaughn]

I have not thought about this in a while. I don't immediately see those new functions as something I'd use. They push extra burden on the client programmer to know all the corner cases and protect from them. I always envisioned `get_in` as something like searching an html document via a css selector. It never raises, even if the document is empty. It either finds matching nodes or it does not.

I learned from earlier in this thread that is not the mental model the core team has for `get_in`. I ended up creating a small library to meet my needs (https://hex.pm/packages/path_express).

-Greg

> On Jul 6, 2020, at 5:54 AM, José Valim <jose....@gmail.com> wrote:
>
> Resurrecting this old-thread. Greg, what if we had a Access.if_nil([]) that would put an empty list if there is a nil value? We could also have Access.on_nil(:error), which effectively halts.
>
> A bit verbose but it will give you control exactly what to do and when, without coupling to the current functions.
>

> --
> You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/36580c49-f276-4512-a2aa-da7ecb7daf60o%40googlegroups.com.

[José Valim]

After coming to this issue a couple times, I believe we should accept this proposal.

My arguments were:

1. This proposal won't work for update_in/pop_in. My counter-argument, as of now, is that this is already true given that get_in(nil[:foo][:bar]) returns nil but update_in(nil[:foo][:bar], & &1) raises.

2. This proposal would remove an API that raises on nil. My counter argument, as of now, is that if you don't want to raise on nils, you likely won't use get_in anyway. Instead you will do map["foo"]["bar"] |> Enum.map.

I plan to submit this to master soon, to be included on v1.13.

[José Valim]

Update: we also have precedents on this behaviour: pop_in also aborts as soon as it sees a nil.

José Valim

https://dashbit.co/

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4389ddde-4aff-4002-b5a6-bd519d693065n%40googlegroups.com.

[Greg Vaughn]

I am so impressed with how long the background processes in your brain live, José :-)

You don't need my review, but the commit looks :thumbsup: to me.

-Greg Vaughn

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4389ddde-4aff-4002-b5a6-bd519d693065n%40googlegroups.com.