Adding `Collectable.empty/1`

[Benjamin Milde]

I've had quite often needs for doing some computation (mapping) on ecto result, but having a flexible function similar to `Repo.preload` currently needs a whole bunch of boilerplate in terms of differenciating collections from single items and especially returning the same type afterwards (e.g. keep it a collection or single item). I know that Enumerable and Collectable were split consciously, but it would be great to have something like `Collectable.empty/1`, so one could do some work using `Enum` functions and in the end do: `Enum.into(changed, Collectable.empty(initial))` and it would empty the collectable and fill it up again using the changed data. This way the `Repo.preload` could e.g. additionally support custom enumerable and collectable collections as first argument like e.g. `%Scrivener.Page{}`.

[José Valim]

My concern is that Collectable.empty cannot be implemented by all structs. For example, what does it mean to call empty() on a IO.stream? Perhaps it would make sense as a separate protocol?

Also, can you please expand on how Repo.preload could use Scrivener.Page in detail? Thanks!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Tue, May 21, 2019 at 5:46 PM Benjamin Milde <lostko...@gmail.com> wrote:

I've had quite often needs for doing some computation (mapping) on ecto result, but having a flexible function similar to `Repo.preload` currently needs a whole bunch of boilerplate in terms of differenciating collections from single items and especially returning the same type afterwards (e.g. keep it a collection or single item). I know that Enumerable and Collectable were split consciously, but it would be great to have something like `Collectable.empty/1`, so one could do some work using `Enum` functions and in the end do: `Enum.into(changed, Collectable.empty(initial))` and it would empty the collectable and fill it up again using the changed data. This way the `Repo.preload` could e.g. additionally support custom enumerable and collectable collections as first argument like e.g. `%Scrivener.Page{}`.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/19c24b83-fc2a-4af3-950e-eba7f767db14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Milde]

All the pagination libs I know for ecto return structs with additional metadata besides the actual results of the db query. Currently one needs to build around those structs specifically to be able to do preloads or other mapping operations or one would loose the metadata. Having a `SomeProtocol.empty` protocol we could still use everything provided by Enum (even with it returning a list), but have a way to replace items in the container with the new modified ones without manually pattern matching between single results, lists of results or structs returned because one happens to be using pagination. Monadic collection types basically don't do it much differently. Extract the subject out of the container, do computation and replace the old value when done. I'd imagine `Enumerable` and the potential `SomeProtocol` would solve a lot of usecases, where people usually ask for Enum to retain the outer collection type.

Am Dienstag, 21. Mai 2019 18:14:49 UTC+2 schrieb José Valim:

My concern is that Collectable.empty cannot be implemented by all structs. For example, what does it mean to call empty() on a IO.stream? Perhaps it would make sense as a separate protocol?

Also, can you please expand on how Repo.preload could use Scrivener.Page in detail? Thanks!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Tue, May 21, 2019 at 5:46 PM Benjamin Milde <lostko...@gmail.com> wrote:

I've had quite often needs for doing some computation (mapping) on ecto result, but having a flexible function similar to `Repo.preload` currently needs a whole bunch of boilerplate in terms of differenciating collections from single items and especially returning the same type afterwards (e.g. keep it a collection or single item). I know that Enumerable and Collectable were split consciously, but it would be great to have something like `Collectable.empty/1`, so one could do some work using `Enum` functions and in the end do: `Enum.into(changed, Collectable.empty(initial))` and it would empty the collectable and fill it up again using the changed data. This way the `Repo.preload` could e.g. additionally support custom enumerable and collectable collections as first argument like e.g. `%Scrivener.Page{}`.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

[José Valim]

> All the pagination libs I know for ecto return structs with additional metadata besides the actual results of the db query. Currently one needs to build around those structs specifically to be able to do preloads or other mapping operations or one would loose the metadata. 

Can you please provide an example? I am just trying to full understand the problem. :)

> Having a `SomeProtocol.empty` protocol we could still use everything provided by Enum (even with it returning a list), but have a way to replace items in the container with the new modified ones

Wouldn't a functor be better suited then?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/1498525e-30fd-4cde-92a0-4b2e01cdbdf3%40googlegroups.com.

[Wiebe-Marten Wijnja]

Interesting question!

While there are many structures for which an 'empty' might make sense, there are some (probably more) for which it does not. To be more formal about this: `Collectable` is essentially an implementation of what is known as a `SemiGroup` in Category Theory terms (pedantically speaking, it implements the concatenation function ('Collectable.into') directly rather than having a binary combining operator that the concatenation function is built on top of).

If you have a SemiGroup that happens to also have a well-defined empty/default value, we end up with what is known as a `Monoid`.

While Elixir does not have a protocol or behaviour in place to work with things that are Monoids directly, there are still two ways you can make it work:

1. By allowing the user to provide the desired 'default' data structure directly (rather than the module it is defined in). This is what we already do when writing `into: %{}` for instance. The nice thing about this approach is that it allows the user to specify any data structure, not only empty ones. (There are many data structures for which there is more than one reasonable default, and there are also cases in which the user might want to collect multiple enumerables in the same collection. Both of these use cases are covered by this).

2. If you happen to write something where it is difficult for the user to provide a datastructure and they only give you a module atom, you can still look if the module has a `new/0` method: While not standardized, this is a convention which a very large amount of libraries as well as Elixir's standard library seem to follow.

~Qqwy/Marten

[José Valim]

Great points.

> pedantically speaking, it implements the concatenation function ('Collectable.into') directly rather than having a binary combining operator that the concatenation function is built on top of

And to further clarify, the reason why we do this is so it is simpler to make it work with resources. :)

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e98aeb01-b04b-4f0f-acec-bbf24fb95b88%40googlegroups.com.

[Benjamin Milde]

Take for example https://github.com/drewolson/scrivener_ecto

With a current version of ecto you can happily do this: 

```
Repo.all(query) |> Repo.preload(:user)
Repo.one(query) |> Repo.preload(:user)
```

but not this:

```

Repo.paginate(query)  |> Repo.preload(:user)
```

Given that doing some work on results of db calls is quite common I tend to end up with lot's of functions pattern matching if results are `%Scrivener.Page{}`, a list or an element. Say you also use https://github.com/duffelhq/paginator for infinite scrolling pages you might end up with yet another "collection" type struct, which implements Enumerable so displaying items in views is super easy, but with no simple solution on how to handle preloads or mappings.

I've also tried using `Map.update!(elements, :key_of_items, function)` to normalize stuff down to handling lists for those places using pagination, but now I'm wrapping a lot of computation in those updates.

It's also not just mapping, but I might also e.g. want to intersperse items with separators or zip some additional data into the resultset and still not lose the pagination information attached to those entries.

My naive approach was that I can already use Collectable to fill items back into lists as well as those pagination structs, but while for list and maps it's easy to use `Enum.into([])` or `Enum.into(%{})` it's not as easy to get a hold of an empty version of one of those pagination structs.

I'm not super into all the theoretic types in functional programming. I think I understand what a functor is, but I'm also not super certain I understand what you're pointing at. 

[José Valim]

Thanks! And if we were to add a Collectable.empty, what would be its implementation for Scrivener.Page?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/5dff9895-1921-4794-89b9-d4ff45578880%40googlegroups.com.

[Benjamin Milde]

It should basically „reset“ the part of the struct, which is used by the Enumerable protocol implementation.

def empty(page), do: %{page | elements: []}

So that after an Enum.into(changes, SomeProtocol.empty(initial_page)) we end up with exactly the same metadata, but different elements.

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/3iBUmpKIYr8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Kueg013YTHmg7jS5o94Qb0YbEjOWRNbKuENKgPiMJK0g%40mail.gmail.com.

[José Valim]

I see, thanks.

It seems we have a couple options:

1. Add empty/0 to Collectable, but I have reservations since it is not a valid operation for all collectables

2. Another option is to introduce a :reset operation into the existing Collectable.into/2, but all in all it has the same issues as 1)

3. Introduce a new Empty protocol (potentially as a lib)

4. Ecto could introduce its own Ecto.Preloadable protocol. This would be beneficial if we can see other functionality being added around this. Are there other issues that returning Scrivener.Page introduces?

5. Perhaps there is a relationship between empty / traversal and the current Access module that could be leveraged too, but i can't see it right now. Perhaps Wiebe-Marten Wijnja can?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/DEDD7A1D-7CBD-4EC9-82B0-2C597DC8CC85%40kobrakai.de.

[Benjamin Milde]

I’m actually with you that a different protocol is probably better.

This is basically what I added to my codebase over the duration of the conversion here.

Ecto could then decide to add support for such collections whenever it wants.

defprotocol Emptyable do
  def empty(type)
end

defimpl Emptyable, for: List do
  def empty(_), do: []
end

defimpl Emptyable, for: Map do
  def empty(_), do: %{}
end

defimpl Emptyable, for: Scrivener.Page do
  def empty(page), do: %{page | elements: []}
end

defmodule MyApp.Collection do
  def with_items(collection, function) do
    collection
    |> Enum.to_list()
    |> function.()
    |> Enum.into(Emptyable.empty(collection))
  end

  def map(collection, mapper) do
    with_items(collection, &Enum.map(&1, mapper))
  end

  def intersperse(collection, element) do
    with_items(collection, &Enum.intersperse(&1, element))
  end

  def zip(collection, enumerable) do
    with_items(collection, &Enum.zip(&1, enumerable))
  end

  def at(collection, index) do
    with_items(collection, &Enum.at(&1, index))
  end

  def preload(collection, preloads, opts \\ []) do
    with_items(collection, &MyApp.Repo.preload(&1, preloads, opts))
  end
end

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4Jc4E-1K7mXR1JCf_Z6_0rUo7UY7vQbPtB9Fs6bO-fnAA%40mail.gmail.com.

[Michał Muskała]

One thing to consider here is that ecto allows you to attach the preload to the query itself. So instead of doing:

Repo.paginate(query) |> Repo.preload(:user)

You can do:

query |> Ecto.Query.preload(:user) |> Repo.paginate()

and everything will work correctly. Yes, this does not solve all the cases, but should cover the vast majority.

Michał.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/5dff9895-1921-4794-89b9-d4ff45578880%40googlegroups.com.