Proposal to add Registry function to get all keys

[johanna.a...@gmail.com]

I propose the addition of a function for getting all the keys (and values associated with them) out of a registry. Currently the only way of doing that is knowing all the pids or keys, and looping over them to request the rest of the information from the registry, which is awkward and under-performant.

I came across the need for functionality like this when I was setting up an architecture where an outside source, a list of keys, should be used to determine what processes to run. For each key in that list a process should be started. Over time, the list will change, and the processes running should be adjusted to represent this. Assuming there's a DynamicSupervisor owning the workers and a coordinating process starting and stopping them, I need a list of the current running processes and their keys to compare with the latest version of the external state.

A simple solution to the problem would be to rely on the changes between the latest version of the list and the last version, and trust the supervisor to restart dying processes. This, however, means the workers can't be `:temporary`, because we would risk reality getting out of sync of the list of workers on the coordinator. But even if I go for `:transient` workers, I still risk getting out of sync if the DynamicSupervisor crashes. Even if I keep that process clean to avoid crashes, some worker crashing repeatedly can still bring it down, and all the workers with it. At that point, the only way to synchronize back to the correct state would be restarting the server or implement some complex code where the coordinator attempts to verify the status of each key by looking it up in the registry, but then we're back to inefficiently getting all keys.

I propose a function that looks and works something like this (naming and ordering of return values is obviously up for discussion)

```
Registry.all(MyRegistry)
=> [{key, pid, value}, {key, pid, value}, ...]
```

Just like `Registry.count/1` has a corresponding `Registry.count_match/{3,4}`, it could be designed to take a match specification.

[José Valim]

Hi Johanna, thank you for the proposal.

The proposal sounds really good my only question is, instead of returning a huge list of results, could this be implemented like Registry.dispatch in that we call a callback for you? My only concern is that if we have a lot of data and multiple partitions, building said lists can be expensive. But if you are using them for side-effects, then the dispatch approach can be a good fit.

--

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[johanna.a...@gmail.com]

I understand the value of implementing it the way `Registry.dispatch` works, but it makes using it a bit awkward. In the example use case I gave earlier, it would look something like this in the coordinating process

```
handle_info(:interval, state) do
new_keys = get_keys_from_external()
Registry.all(registry, fn entries ->
current_keys = Enum.map(entries, &(elem(&1, 0))
synchronize(current_keys, new_keys)
end)
end
```

Which is okay, I guess, but the data is stuck in the callback. Also, if we copy the way that dispatch handles partitions the code would only work if there aren't multiple partitions. Not sure how I'd go about collecting the entries from multiple partitions, maybe in an ETS table? Sending to myself and storing in state? Either way, it's not very ergonomic. Maybe I'm missing an obvious solution.

Would it make sense to implement it as a Stream? It should provide some of the same properties as the dispatch solution. We could iterate over the tables and handle partitions seamlessly.

[José Valim]

I agree. Let's go ahead with what you have originally proposed. In case of multiple partitions, we will simply concatenate the results. We will add a note that it is potentially expensive on duplicate registries with multiple partitions. We can add a stream_all/1 in the future if necessary.

My only last question is about the function name. Registry.all/1 makes sense but if we want to add a version that also supports match specs, then we would need to add Registry.all_match/3, if we are to keep the current convention. But I think "all_match/3" is a bit awkward? Any further thoughts on the name?

Note Registry.match/4 already exists and it performs a match under a given key.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/503ef13f-9e18-4e64-8e1c-84ca677ab504%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrea Leopardi]

I think even something similar like "all_with_match/3" or "all_by_match/3" makes this less awkward than "all_match/3".

> We can add a stream_all/1 in the future if necessary.

On this note, wouldn't it make more sense to start with stream implementation directly? It seems to be a lower abstraction than the concatenated list and the list is just an "Enum.to_list/1" call away for users.

Andrea Leopardi

an.le...@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JO%3DjveXLf5RHyGqsnNVnjpLW1%2BsDV%2BHpRv8LD9ZwWkkw%40mail.gmail.com.

[Derek Kraan]

Currently we have keys/2 and match/4.

We could also make a keys/1 that returns all keys, and a match/3 that omits the `key` argument (trickier because there is already a /3 variant with the 4th optional argument).

What do you think?

(I have also noticed from making Horde that a lot of people want this functionality, which I had added, but later deprecated since I was aiming for API parity with Elixir's Registry. It would be nice to have this functionality added back again so that there is one way to get this done.)

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JO%3DjveXLf5RHyGqsnNVnjpLW1%2BsDV%2BHpRv8LD9ZwWkkw%40mail.gmail.com.

[José Valim]

Hi Derek, thanks for jumping in.

I thought about keys/1 but they would have different return types so I am not quite sure if we should mix them.

One option to add for the discussion is to_list/1 and to_list_match/3. 

--

[johanna.a...@gmail.com]

I like to_list, it fits nicely in with the other functions. If we want a streamed version it could be to_stream then.

As it seems there's at least consensus on adding this feature (if not exactly on how it should work) would it be reasonable for me to attempt a PR for this? I could start out by implementing to_list at least, it should be straightforward.

I'm unsure of the practicalities, should I add an issue to the Github repo first?

[José Valim]

Sounds great, let’s start with to_list and we can discuss the name in detail later. Seeing the implementation may help figure the name out.

Please do attempt a PR. It is not necessary to open up an issue before. If you have any questions, feel free to reach out. You can also send the PR as a draft if you want early feedback.

Looking forward to it!

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/57494517-aaa9-4dd6-97d5-bf5fa0089922%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Derek Kraan]

You're right, I hadn't thought of that.

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Lt989nD909VBXrVpCdO3qdRVyafSLbzvYhc6bUwsGEvA%40mail.gmail.com.