Under the hood, sets are hash tables (the elements of the sets are keys and the values are all 1). The order of keys of hash tables is deterministic, but doesn't necessarily agree with the order that we might expect. Integers serve as their own hash codes, but we mod them out by the length of the underlying array used by hash table.
For this particular example, we have a very small hash table, so we start with an array with the default length of 4. Since 1 and 2 are less than 4, they end up in expected positions in the array. But for 3 and 5, 3 ends up in the expected position and we mod out 5 by 4 and put it in the 1st position.
You can see this using the unexported "buckets" function:
i1 : debug Core
i2 : buckets set {1, 2}
o2 = {{}, {(1, 1)}, {(2, 1)}, {}}
o2 : List
i3 : buckets set {3, 5}
o3 = {{}, {(5, 1)}, {}, {(3, 1)}}
o3 : List