I think such a change would lead to a MapSet that would no longer act like a set. Consider, for example:
set =
MapSet.new(fn string -> String.length(string) end)
|> MapSet.put("foo")
|> MapSet.put("bar")
With such a set, only one of MapSet.member?(set, "foo") or MapSet.member?(set, "bar") could return true (and it’s not clear to me which one would) but an invariant of sets is that set |> MapSet.put(val) |> MapSet.member?(val) should return true for any val.
Using a set that doesn’t act like a set would be confusing, so I think you’d be better off writing your own abstraction that provides the functionality you need on top of Map.