many_to_many, checkboxes and cast_assoc [StackOverflow repost]

[Cory Gwin]

I am curious what is the preferred way to get the result of a form with checkboxes representing a many_to_many relationship to save the relationship in a changeset?

Here is an example:

schema:

      schema "users" do
        field :name, :string
        field :email, :string
        field :active, :boolean, default: true
        has_many :products, TestExAdmin.Product
        has_many :noids, TestExAdmin.Noid
        many_to_many :roles, TestExAdmin.Role, join_through: TestExAdmin.UserRole
      end

checkboxes:

        <div class="form-group">
           <label class="col-sm-2 control-label" for="user_roles">Roles</label>
           <div class="col-sm-10">
              <input name="user[roles][]" type="hidden" value="">
              <div class="checkbox"><label><input type="checkbox" name="user[roles][1]">role1</label></div>
           </div>
        </div>

changeset:

      def changeset(model, params \\ %{}) do
        model
        |> cast(params, @required_fields, @optional_fields)
        |> cast_assoc(:noids, required: false)
        |> cast_assoc(:products, required: false)
        |> cast_assoc(:roles, required: false)
      end

params received:

    %{email: "te...@example.com", name: "Cory",
      products: %{"1481120458618": %{_destroy: "0", price: "13.00",
          title: "A product title"}}, roles: %{"1": "on"}}

I currently get an error:

    errors: [roles: {"is invalid", [type: {:array, :map}]

I am wondering if there is an idiomatic solution. I am also curious about removing association of this sort in an edit/update scenario.

Thanks,

Cory

[José Valim]

When you can't send the parameters that cast_assoc expect, the best is to use put_assoc. so I would process the parameters manually and then call put_assoc with the roles loaded from the database.

I just wrote extensively about this in the Ecto 2 book which had the last beta out today. It is free and it covers a slightly more complex example than the one you posted: http://pages.plataformatec.com.br/ebook-whats-new-in-ecto-2-0

On Fri, Dec 9, 2016 at 00:20 Cory Gwin <gwin...@gmail.com> wrote:

Here is the SO question: http://stackoverflow.com/posts/41019905/edit

--
You received this message because you are subscribed to the Google Groups "elixir-ecto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/35335922-f158-4792-b4e9-d954204b9d49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cory Gwin]

Thanks Jose, Steve was also saying maybe use put_assoc, I just wanted to make sure there was not a default way to do this. I will definitely check out the book, thank you so much for your reply!

[Steve Pallen]

José,

I was playing around with this yesterday and could not get it working. I was following the todo_list example in your new book. I was wondering if could answer a couple questions, based on the schema/changesets below. Note that I have tried this on several occasions without any success. I would prefer to be able to use cast_assoc vs put_assoc.

1. Is it possible to add/update/remove existing roles with cast_assoc on an create/update User?
2. If so, what would the params look like to add/remove roles in create and update?
3. Is it possible to add a new role to the database on a create/update User?
4. If so, what would the params look like?

defmodule UcxRemote.User do
  use UcxRemote.Web, :model

  schema "users" do
    has_many :user_roles, UcxRemote.UserRole, on_delete: :delete_all, on_replace: :delete
    has_many :roles, through: [:user_roles, :role], on_delete: :delete_all, on_replace: :delete
    field :first_name, :string
    field :last_name, :string
    field :email, :string
    timestamps
  end

  def changeset(model, params \\ %{}) do

    model
    |> cast(params, [:first_name, :last_name, :email])
    |> validate_required([:first_name, :last_name, :email])
    |> cast_assoc(:user_roles, required: false)
    |> unique_constraint(:email)
  end
end

defmodule UcxRemote.UserRole do
  use UcxRemote.Web, :model

  schema "roles_users" do
    belongs_to :role, UcxRemote.Role
    belongs_to :user, UcxRemote.User
    timestamps()
  end

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:role_id, :user_id])
    |> cast_assoc(:role, required: true)
  end
end

defmodule UcxRemote.Role do
  use UcxRemote.Web, :model

  schema "roles" do
    has_many :user_roles, UcxRemote.UserRole, on_delete: :delete_all
    has_many :users, through: [:user_roles, :user], on_delete: :delete_all
    field :name, :string
    timestamps()
  end

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:name])
    |> validate_required([:name])
  end
end

Steve

[José Valim]

1, 2. Yes, the format needs to be:

%{"user" => %{

  "user_roles" => [

    %{"role_id" => 1}, # Associating the user to an existing role

    %{"role_id" => 2, "id" => 3} # Updating the user_role with id of 3

  ]

}

If you want to delete, there are a couple options: one option is to simply not send the map for the entry you want to have removed and then set "on_replace: :delete" on that association. Alternatively, you can use the "action: :delete" trick explained in this blog post: blog.plataformatec.com.br/2015/08/working-with-ecto-associations-and-embeds/

3, 4. Yes, the format needs to be:

%{"user" => %{

  "user_roles" => [

    %{"role" => %{"name" => "admin"}}

  ]

}

Overall you have:

%{"user" => %{

  "user_roles" => [

    # Associating the user to an existing role

    %{"role_id" => 1},

    # Updating the user_role with id of 3

    %{"role_id" => 2, "id" => 3} 

    # Add a new role and associate it to user

    %{"role" => %{"name" => "admin"}}

    # Update the user role with id of 4 by updating its inner role name (with id of 5) to "new admin"

    %{"id" => 4, "role" => %{"name" => "new admin", "id" => 5}}

  ]

}

The rules are always the same: you need an ID for updates, otherwise it is inserted. Omissions are treated as replacements and will call the :on_replace option. I believe they are explained in the chapter 8 or 9 of the book (the ones about many to many).

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--

You received this message because you are subscribed to the Google Groups "elixir-ecto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/361ed2ba-271b-4e40-97d2-c57592b0a8a9%40googlegroups.com.

[Steve Pallen]

José,

That worked great. Thanks a bunch. One follow question? Is the same possible with many_to_many?

with |> cast_assoc(:roles) and params

%{"user" => %{"roles" => [%{"id" => 1}]}} I get "name can't be blank" and when I give a name, it inserts a Role.

Steve

BTW, the example in the book assumes that the new records are inserted.

[Cory Gwin]

Just a couple thoughts. We are scrubbing params that are coming into the controller to feed them into changesets in a way that works with cast_assoc. I don't feel like this sort of task really belongs in ExAdmin. If we support other Relational Mappers we will need to scrub in different ways intelligently.

Should the work we are doing to get standard html form params ready for Ecto actually live in Ecto or EctoPhoenix?

I am also concerned that we are going to end up making developers have multiple changesets just because we are manipulating things in certain was to get the associations to work, or that they in another part of the app may face the same association issues and implement them differently.

I also think it would be nice if both the many_to_many and the has many through api's operated in the same way.

Thanks again,
Cory 

[José Valim]

BTW, the example in the book assumes that the new records are inserted.

Yes, because that's how all associations in Ecto behave.

 

%{"user" => %{"roles" => [%{"id" => 1}]}} I get "name can't be blank" and when I give a name, it inserts a Role. 

I believe though you can make it work by doing the same "action: :delete" trick in the blog post I deleted in the previous post. You could check if there is an ID and, if so, you could mark its action as updated. Be careful though about what will happen if someone passes an ID that does not exist in the database or is invalid. Do you raise? Do you create it ignoring the value? If it is for ExAdmin though, you are probably fine, since it is internal and unlikely to be a vector of attack. 

[Steve Pallen]

José, Is there a good reason for the inconsistency between has_many :through and many_to_many and why I can't find it documented? The docs for has_many :through indicate if an ID is present, the join table will be updated with the existing association and if its not present, a new association will be created and the join table updated. However, I could not find anything in the many_to_many docs indicating this behaviour is different.

Cory, I think its time for ex_admin to get out of the Ecto intelligence business and leave this to the programmer. You're right, they are likely going to need to solve these issues in other parts of their project. 

Steve

[José Valim]

The associations should behave exactly the same when it comes to cast_assoc. many_to_many does maintain the join table but the goal is to do it transparently.

--

You received this message because you are subscribed to the Google Groups "elixir-ecto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/1a35f115-3405-465d-842e-46a3b236e1a5%40googlegroups.com.

[Cory Gwin]

Thanks for the help so far José

I cannot seem to get this to work when using many_to_many

Here are the models:

defmodule ContactDemo.User do
  use ContactDemo.Web, :model
  use Coherence.Schema


  alias ContactDemo.AppConstants


  schema "users" do
    field :name, :string, null: false
    field :username, :string, null: false
    field :email, :string, null: false
    field :active, :boolean, null: false, default: true
    field :expire_on, Timex.Ecto.Date

    #has_many :users_roles, ContactDemo.UserRole
    #has_many :roles, through: [:users_roles, :role]
    #
    many_to_many :roles, ContactDemo.Role, join_through: ContactDemo.UserRole


    coherence_schema


    timestamps
  end


  @doc """
  Creates a changeset based on the `model` and `params`.


  If no params are provided, an invalid changeset is returned
  with no validation performed.
  """

  def changeset(model, params \\ %{}) do

    model
    |> cast(params, ~w(name email username active expire_on) ++ coherence_fields)
    |> cast_assoc(:roles)
    |> validate_required([:name, :email, :username, :active]) # TODO: Add 'expire_on'
    |> validate_format(:name, AppConstants.name_format)
    |> validate_length(:name, min: 1, max: 255)
    |> validate_format(:username, AppConstants.username_format)
    |> validate_length(:username, min: 1, max: 255)
    |> validate_length(:email, min: 1, max: 255)
    |> unique_constraint(:username, name: :users_username_index)
    |> validate_format(:email, AppConstants.email_format)
    |> unique_constraint(:email, name: :users_email_index)
    |> validate_coherence(params)
  end
end

defmodule ContactDemo.Role do
  use ContactDemo.Web, :model


  alias ContactDemo.AppConstants


  schema "roles" do
    field :name, :string, null: false
    many_to_many :users, ContactDemo.User, join_through: ContactDemo.UserRole
    timestamps
  end


  @required_fields ~w(name)
  @optional_fields ~w()


  @doc """
  Creates a changeset based on the `model` and `params`.


  If no params are provided, an invalid changeset is returned
  with no validation performed.
  """

  def changeset(model, params \\ %{}) do
    model
    |> cast(params, @required_fields, @optional_fields)

    |> validate_required(:name)
    |> validate_format(:name, AppConstants.name_format)
    |> validate_length(:name, min: 1, max: 255)
    |> unique_constraint(:name, name: :roles_name_index)
  end
end

User.changeset(%User{}, %{"roles" => [%{"id" => "2"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})                  
#Ecto.Changeset<action: nil,
 changes: %{email: "te...@test.com", name: "c", password: "password",
   roles: [#Ecto.Changeset<action: :insert, changes: %{},
     errors: [name: {"can't be blank", []}], data: #ContactDemo.Role<>,
     valid?: false>], username: "test"}, errors: [], data: #ContactDemo.User<>,
 valid?: false>

If I change it to a has many through it works fine. I also tried it with user_roles, if the many to many code worked the same way as the has many through I figured maybe this was the way to use it, it fails to work though:

User.changeset(%User{}, %{"user_roles" => [%{"role_id" => "2"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})
#Ecto.Changeset<action: nil,
 changes: %{email: "te...@test.com",
   encrypted_password: "$2b$12$AH67FYsDHT2HG/mv1kwVc.giS7w.bNE3wKmCKpK5WX79GpXveuJ/W",
   name: "c", password: "password", username: "test"}, errors: [],
 data: #ContactDemo.User<>, valid?: true>

I also tried this by changing the cast assoc method to:

    model
    |> cast(params, ~w(name email username active expire_on) ++ coherence_fields)
    |> cast_assoc(:user_roles)

Which gives this error:

User.changeset(%User{}, %{"user_roles" => [%{"role_id" => "2"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})
** (ArgumentError) cannot cast assoc `user_roles`, assoc `user_roles` not found. Make sure it is spelled correctly and properly pluralized (or singularized)
            (ecto) lib/ecto/changeset.ex:606: Ecto.Changeset.relation!/4
            (ecto) lib/ecto/changeset.ex:554: Ecto.Changeset.cast_relation/4
    (contact_demo) web/models/user.ex:38: ContactDemo.User.changeset/2

So I am not sure that many_to_many works in the same was as has_many_through. If it does then I am clearly doing something wrong. 

If I set the name with the first changeset things work in a sense, however it is creating a new role and user role instead of a new user role entry associated with the role and new user.

User.changeset(%User{}, %{"roles" => [%{"id" => "2", "name" => "HI"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})
#Ecto.Changeset<action: nil,
 changes: %{email: "te...@test.com",
   encrypted_password: "$2b$12$YAWASYtXWiLVs7viaUAKVe4YYX9cxnC9PMGSKsgzSSNYflFW34Inm",
   name: "c", password: "password",
   roles: [#Ecto.Changeset<action: :insert, changes: %{name: "HI"}, errors: [],
     data: #ContactDemo.Role<>, valid?: true>], username: "test"}, errors: [],
 data: #ContactDemo.User<>, valid?: true>

If I get the role and cast it into the user schema, it marks it as updated if I change a schema attirbute on the embedded item:

User.changeset(%User{roles: [role]}, %{"roles" => [%{"id" => "2", "name" => "HI"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})
#Ecto.Changeset<action: nil,
 changes: %{email: "te...@test.com",
   encrypted_password: "$2b$12$Ak3S6zvmXLpHlqRspro5tO1fbz6Y6ehnUnCN5TBegdSS5QrCGDPGO",
   name: "c", password: "password",
   roles: [#Ecto.Changeset<action: :update, changes: %{name: "HI"}, errors: [],
     data: #ContactDemo.Role<>, valid?: true>], username: "test"}, errors: [],
 data: #ContactDemo.User<>, valid?: true>

But not if I just pass the id:

User.changeset(%User{roles: [role]}, %{"roles" => [%{"id" => "2"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})    
#Ecto.Changeset<action: nil,
 changes: %{email: "te...@test.com",
   encrypted_password: "$2b$12$Z32J7S5Uk47uXAf0PahA7e8n0CTjRD2tXWNL0ob4Qiz8h.iNrjN12",
   name: "c", password: "password", username: "test"}, errors: [],
 data: #ContactDemo.User<>, valid?: true>

But what I really want is just to pass in the ID's to associate with the new entry:

User.changeset(%User{}, %{"roles" => [%{"id" => "2"}], "name" => "c", "password" => "password", "email" => "te...@test.com", "username" => "test"})

[José Valim]

Cory, the behaviour you describe is exactly how Ecto behaves. Some clarifications which may have been the source of confusion:

1. many_to_many works the same as has_many when it comes as long as you changing the end of the association. Because many_to_many hides away the join association, if you need to change or set it, you won't be able to.

2. If you pass an ID and that ID is associated to the struct, it will update it, otherwise it will attempt to insert it. Once you pass %{"role" => %{"id" => 1, "name" => "admin"}}, if there is no role with ID 1 associated to the struct, it will try to insert it by calling the changeset function in role with the parameters.

Point 2 is very important in terms of security. Imagine roles were not per application but scoped to the current user or current company. If %{"role" => %{"id" => 1}} allowed you to associate to any role with "id" => 1, then someone somewhere would maybe be able to something like %{"company" => %{"id" => 2}} and maybe try to associate them to another company.

So the operation of "associate to an existing ID", which is exactly what you are trying to do, is not supported by has_many, many_to_many, etc. You can only do it with has_many :through because the "associate to an existing ID" then effectively becomes an insert operation but you need to be wary of the caveat above. Or, alternatively, use put_assoc.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/179a5f94-47ba-46b8-8873-f4427fea6bd8%40googlegroups.com.

[Steve Pallen]

My take away from this discussion is that I may have had an incorrect assumption about the new many_to_many api.

I assumed that many_to_many was a replacement for has_many :through, and that the latter was kept for backwards compatibility. However, I see now that there cases when one my want to continue to use has_many through. One such case providing the ability to update the join table as most of this discussion has been centred around. 

I guess this makes sense since the has_many :through API was not depreciated in favour of many_to_many :)

I'm not sure I've seen this explicitly documented anywhere, presuming I now understand the design and behaviour. If so, perhaps a brief note should be added to the docs??

I must say, that this is the one area of Ecto that I have really struggled with.

Steve

[Cory Gwin]

Yes, I have also found it odd that the effect of each schema type is the same in some ways but different in others. It seems weird to me that in one scenario the reasoning for not allowing cast_asoc to work is that it is a security issue, but in another scenario the security issue seems to be ignored. I would personally prefer a consistent api, that worked with the cast_assoc.

Would it be better to handle the cast_assoc security issue in another way? Perhaps an optional list of allowed id's for the association? This is essentially what is happening either way, unless I am missing something:

|> cast_assoc(:user_roles, required: false, allowed_join: [1, 2, 5], allow_creation: true)




This sort of api seems to better describe what is happening, otherwise it just feels to constricting. It could be an opt in or opt out.

Just an idea

- Cory

[José Valim]

> I guess this makes sense since the has_many :through API was not depreciated in favour of many_to_many :)

Yes, it was not deprecated.

As the book describes, many_to_many is semantically close to a has_many. Except one internally uses join tables for associating them and the other uses a single foreign key.

Hopefully those things will become clearer with the book and as more resources come out. For example, I believe this month is the first time we are having public discussions on the topic.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--

You received this message because you are subscribed to the Google Groups "elixir-ecto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/3e533e08-52ae-4cf2-b3af-1593a2b5ab27%40googlegroups.com.

[José Valim]

Yes, I have also found it odd that the effect of each schema type is the same in some ways but different in others. It seems weird to me that in one scenario the reasoning for not allowing cast_asoc to work is that it is a security issue, but in another scenario the security issue seems to be ignored. I would personally prefer a consistent api, that worked with the cast_assoc.

I am afraid there has been some confusion on what Ecto provides and what you should do to solve the problem in your particular domain. Let me try to be a bit clearer!

Imagine you have a todo list which is associated to todo items. *It doesn't matter the type of relationship (many_to_many, has_many, etc), they all work the same*:

  * if a todo item sent as parameter has an ID and it matches an existing associated todo item, we consider that todo item should be updated

  * if a todo item sent as parameter does not have an ID (nor a matching ID), we consider that todo item should be inserted

  * if a todo item is currently associated but its ID was not sent as parameter, we consider the todo item is being replaced and we act according to the `:on_replace` callback. By default `:on_replace` will raise so you choose a behaviour between replacing, deleting, ignoring or nilifying the association

What changes between associations is the work done behind the scenes to persist the relationship. has_many sets the ID in the child, belongs_to does it in the parent and many_to_many in a join table.

The reason why cast_assoc does not allow you to give an ID of a non-associated entry is because it would be a security issue in the huge majority of scenarios. I am talking from experience here. *Having a secure default is a good default*.

And the problem we are talking about here is a problem of authorization:

* can I create roles?

* to which roles am I allowed to associate to?

* can I update roles?

* can I replace roles? or can I delete them?

* etc

Authorization is a too complex topic to try to fit it under some options that would be given to cast_assoc. The defaults are restrictive in that you can create entries that will be associated to the parent resource and also update them. Even deleting is not allowed out of the box and you need to set the :on_replace option.

Your example, on the other hand, seems to have a fairly lax set of authorization rules. It seems anyone can create, update or associate to any existing role and that makes things more straight-forward. But between the most restrictive and the most lax, there is a whole range of combinations, including with regards to where the authorization rules are stored, which could also be in the database (for example, I can only see the roles that belong to a given group).

In other words, if your application requires a different set of rules, then it is up to you to make Ecto work the way you want to. Ecto provides two tools: cast_assoc is the rigid version that was designed for some restrictive scenarios and put_assoc is the generic one. The building block is there. Ecto won't be able to handle all use cases. The best we can do is to educate when cast_assoc works and when to fallback to "roll your own"™.

Now going back to the email reply:

> Yes, I have also found it odd that the effect of each schema type is the same in some ways but different in others. 

This is not specific to Ecto at all. It is about your domain and which rules you allow. Once you pick an association, they all handle parameters in the same way, and you need to see if it maps correctly to your data. It is not that we "ignored" the security issue: it is rather because your example does not care about that security limitation so you can expose different relationships, such as UserRole and Role. You can imagine an application that does not allow users to create roles on the fly would not invoke something such as cast_assoc(:role).

I do understand though that trying to make "cast_assoc" happy may be too complicated and that's exactly why we have put_assoc. With put_assoc, you are free to model the parameters sent by the form in any way you want, then you take care of associating and converting the data as you prefer.

[Cory Gwin]

Thank you for the detailed reply :) I think I understand why things are they way they are now.

Best,

Cory

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-ecto" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-ecto/s4heBrGN9J0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-ecto...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/CAGnRm4%2BzRm4RJ9cSych8FD2QRBpE%2BuiwPwA6HSOkAM4wMPoPLA%40mail.gmail.com.

[Cory Gwin]

Hi Jose,

I have ran into another use case question. In my scenario, I have a many to many relationship on an existing entry. I am updating the entry with the following actions:

1. Delete a required field on the entry, causing a validation error.
2. Remove an association
3. Update the entry.

In order to remove the association using cast_assoc, I need to pass in all the association on the entry except the one I am deleting, the changeset returned will reflect only this group of associations, however I also deleted a required field on the parent entry, so in theory I need to show an invalid form and should show the user all the actions they took including the deleted entry. In order to do this I have had to pull data from both the conn params and the changeset. Is there a better way to accomplish this?

Thanks,
Cory

To unsubscribe from this group and all its topics, send an email to elixir-ecto+unsubscribe@googlegroups.com.