"sort" not working with row vectors?!

[Umberto]

Hi all

I find this puzzling:

julia> sort([11 3 4 2])
11 3 4 2

julia> sort([11, 3, 4, 2])
2
3
4
11

Same issue with "sortr!".

I am working with the latest Windows build.

Thanks for any comment.

[Viral Shah]

That is because these are really matrices, and the default is to sort along columns. In order to sort the rows, you need to give the optional dim argument:

julia> sort([11 3 4 2], 2)

1x4 Int64 Array:

 2  3  4  11

-viral

[Umberto]

Ok I understand the issue.

However I have to say that, wow, thank God I tried to check for the function behavior before actually writing it in my very large project. I would have take me days if not weeks to find where the mistake was...
Sorting a vector is such a basic, fundamental (and most of all, intuitive) operation that should work "out of the box" without too much worries. In this case it would have meant a disaster for me...

[Viral Shah]

This is the good old issue about "What is really a vector". Better not restart the discussion. :-)

-viral

> --
>
>
>

[Jeff Bezanson]

That is exactly the case with the matlab behavior, which causes an obscure bug the one time your data has one row. Maybe the dimension argument should be required for matrices.

--
 
 
 

[Tim Holy]

On Wednesday, August 29, 2012 02:57:08 AM Umberto wrote:
> Ok I understand the issue.
>
> However I have to say that, wow, thank God I tried to check for the
> function behavior before actually writing it in my very large project. I
> would have take me days if not weeks to find where the mistake was...
> Sorting a vector is such a basic, fundamental (and most of all, intuitive)
> operation that should work "out of the box" without too much worries. In
> this case it would have meant a disaster for me...

It does work out of the box. The alternative would be "if the number of rows
is 1, then sort along columns." In my view, that is much more of a recipe for
disaster. Suppose I've written a routine that requires me to sort along
columns. I pass it matrices, and every once in a while one of those is a
matrix with only one row. I still want to sort along columns, but now suddenly
it sorts along the row, which is a bug.

This is a much, much harder bug to find than the one you're describing, because
it might occur only 0.01% of the time. In contrast, the one you view as a bug
happens every time and I would think it should be very easy to find.

Unfortunately, this poor behavior is exactly how Matlab works, and perhaps by
now you won't be surprised to hear that it's a bug that I (as a recovering
Matlab user) have been bitten by several times. In Matlab I long ago learned
to never, ever use the "convenient" syntax sort(A), but instead to always
supply the dimension.

There are many examples where Julia behaves in a much more predictable way
than Matlab because it does have real one-dimensional objects. You just have
to get used to typing "," every place you used to type " " between elements of
a vector.

--Tim

[John Cowan]

On Wed, Aug 29, 2012 at 9:31 AM, Tim Holy <tim....@gmail.com> wrote:

> Unfortunately, this poor behavior is exactly how Matlab works, and perhaps by
> now you won't be surprised to hear that it's a bug that I (as a recovering
> Matlab user) have been bitten by several times. In Matlab I long ago learned
> to never, ever use the "convenient" syntax sort(A), but instead to always
> supply the dimension.

Perhaps, then, the dimension should be required in Julia when the
argument is a matrix.

--
GMail doesn't have rotating .sigs, but you can see mine at
http://www.ccil.org/~cowan/signatures

[Viral Shah]

On 29-Aug-2012, at 7:17 PM, John Cowan wrote:

> On Wed, Aug 29, 2012 at 9:31 AM, Tim Holy <tim....@gmail.com> wrote:
>
>> Unfortunately, this poor behavior is exactly how Matlab works, and perhaps by
>> now you won't be surprised to hear that it's a bug that I (as a recovering
>> Matlab user) have been bitten by several times. In Matlab I long ago learned
>> to never, ever use the "convenient" syntax sort(A), but instead to always
>> supply the dimension.
>
> Perhaps, then, the dimension should be required in Julia when the
> argument is a matrix.

Although it is reasonable, we should do this consistently for all functions that expect a dimension argument, and there are lots of such functions. It does make the code clear, if a bit verbose.

-viral

[Stefan Karpinski]

Yes. This is absolutely what we should do. 

[Harlan Harris]

In the interest of readability, could we allow something like one of these to work for matrices?

sort(mat, "row")
sort(mat, :row)
sort(mat, ROW)

The 1/2/x indexes are not easy to read quickly.

Harlan

--
 
 
 

[Stefan Karpinski]

Rather than that I'd prefer to define sortcols(a) = sort(a,1) and sortrows(a) = sort(a,2).

--
 
 
 

[Harlan Harris]

Sure, that works.

--
 
 
 

[Viral Shah]

Matlab has a sortrows, which does something different.

-viral

> --
>
>
>

[Harlan Harris]

Oh, yeah, it sorts a matrix by a given column (default first): http://www.mathworks.com/help/techdoc/ref/sortrows.html

That's presumably useful behavior, but the name isn't that clear. I'd vote for breaking Matlab compatibility here. What Matlab calls sortrows might be better something like "reorder"?

 -Harlan

--

[John Myles White]

+1 for all of sortrows, sortcols and reorder.

 -- John

--
 
 
 

[Stefan Karpinski]

Having a function with the same name as a Matlab function that does something different is a terrible idea. We should definitely not do that. I forgot about that function. It's quite useful: it lexicographically sorts the rows of a matrix. Reorder doesn't convey that to me at all.

--
 
 
 

[Dmitri Paduchikh]

How about sorteachrow and sorteachcol?

Stefan Karpinski:

--
With best regards
Dmitri Paduchikh

[Stefan Karpinski]

That's starting to be a bunch of typing compared to sort(a,1) and sort(a,2) :-\

--

[John Myles White]

But sort(a, 1) and sort(a, 2) are really opaque. R has this with apply() and I have to look up the meaning of those numbers every time because I'm never 100% sure I remembered their meaning correctly.

 -- John

 

--
 
 
 

[Harlan Harris]

Yeah. Thus my suggestion for enumerations or similar. The meaning of sort(a, ROWS) or sort(a, COLS) is very clear.

 -Harlan

--
 
 
 

[Stefan Karpinski]

I agree. Maybe rowsort and colsort?

--
 
 
 

[Stefan Karpinski]

I guess having const ROWS = 2 and const COLS = 1 would be ok. Seems like kind of a weird definition to have around but I guess it could be used in a lot of places for additional readability.

--
 
 
 

[Mike Nolta]

On Wed, Aug 29, 2012 at 1:37 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> I guess having const ROWS = 2 and const COLS = 1 would be ok. Seems like
> kind of a weird definition to have around but I guess it could be used in a
> lot of places for additional readability.
>

Yuck.

-Mike

> --
>
>
>

[Stefan Karpinski]

yeah, I kind of agree with that "yuck".

--

[John Myles White]

rowsort and colsort work for me. Adding new constants like that seems troubling to me. If we headed that way, I'd feel better about symbols than constants.

The only trouble is that rowsort and colsort break the English-like SVO defaults that occur in things like println. Seems worth it, though.

 -- John

--
 
 
 

[David Campbell]

On Wed, Aug 29, 2012 at 1:49 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> yeah, I kind of agree with that "yuck".
>
>
> On Wed, Aug 29, 2012 at 1:47 PM, Mike Nolta <mi...@nolta.net> wrote:
>>
>> On Wed, Aug 29, 2012 at 1:37 PM, Stefan Karpinski <ste...@karpinski.org>
>> wrote:
>> > I guess having const ROWS = 2 and const COLS = 1 would be ok. Seems like
>> > kind of a weird definition to have around but I guess it could be used
>> > in a
>> > lot of places for additional readability.
>> >
>>
>> Yuck.
>>
>> -Mike
>>

Giving some rational as to why defining these constants is good or bad
is a lot more useful than saying "yuck" and "I kind of agree". I think
we should also keep in mind that how this is handled really matters
for more than just sort since there are numerous other general
functions that already exist and which either have row/col versions
already or take a dimension argument (e.g. size, stride, vec (rowvec,
colvec), cat (vcat, hcat). An attempt to keep the API small and
uniform would make code more maintainable and require users to
memorize less. I see defining two constants as being better than
creating two additional functions for every function that takes a
dimension argument.


--
David Campbell

[Harlan Harris]

Also, if we anticipate that someone ports the matlab sortrows and sortcols functions, then we'll have both sortrows and rowsort, with very similar signatures but rather different behavior. That doesn't seem great.

Oh, and I agree with David.

--
 
 
 

[Stefan Karpinski]

On Wed, Aug 29, 2012 at 2:20 PM, David Campbell <dcamp...@gmail.com> wrote:

Giving some rational as to why defining these constants is good or bad is a lot more useful than saying "yuck" and "I kind of agree". I think we should also keep in mind that how this is handled really matters for more than just sort since there are numerous other general functions that already exist and which either have row/col versions already or take a dimension argument (e.g. size, stride, vec (rowvec, colvec), cat (vcat, hcat). An attempt to keep the API small and uniform would make code more maintainable and require users to memorize less. I see defining two constants as being better than creating two additional functions for every function that takes a dimension argument.

Sure, but sometimes you just have a gut reaction to these things and a lot of design is a matter of taste. So "yuck", while not exactly a coherent argument, is a pretty reasonable reaction as an expression of distaste. These constants do become part of an API too, and I just find writing sort(a,ROWS) kind of gross and it seems that Mike agrees. I'd honestly rather write sort(a,2), but then again I never really have too much difficulty remembering the 1 vs 2 thing.

Your mention of hcat and vcat does suggest an alternative naming: hsort and vsort. This naming convention could be generalized to other kinds of operations — e.g. hmap and vmap which could be specific versions of amap for horizontal and vertical dimensions. But as you point out, this starts to snowball if we apply it everywhere.

[Gabor]

Just my 2 cents:

sort(a,1) and sort(a,2) with a compulsory second argument are fine.

There are more important issues to be sorted out soon.

[Harlan Harris]

I disagree. I feel like having a nice, tight, well-thought-out set of core libraries, with sensical, consistent function names and readable arguments is absolutely critical. Both R and Matlab have issues with this, in my experience. I think it'd be a shame to just make Julia a faster Matlab, without addressing this issue. I've been very happy to see the core Julia devs favor consistency and interpretability over Matlab compatibility in the past, and I hope they do so for this issue too.

The problem with hsort() and vsort() is that sorting rows or columns doesn't feel like a horizontal or vertical operation to me in any reasonable way.

Ideally, for me, the operations should use keyword parameters to change what sort of sorting you're doing. Something like:

sort(matrix, each=rows) # each suggests a repeated operation
sort(matrix, by=column(1)) # like Matlab's sortrows()

To me, that's maximally readable. operation(on what, how). Obviously this isn't currently possible.

 -Harlan

--
 
 
 

[John Cowan]

On Wed, Aug 29, 2012 at 3:26 PM, Harlan Harris <har...@harris.name> wrote:

> sort(matrix, each=rows) # each suggests a repeated operation
> sort(matrix, by=column(1)) # like Matlab's sortrows()

I agree with the general spirit of this, but it's easily achieved by
choosing better function names, like sort_each_row (or sortEachRow, or
whatever, I take no position on that) and sort_by_column in this case,
plus of course sort_each_column and sort_by_row.

[Patrick O'Leary]

On Wednesday, August 29, 2012 1:20:04 PM UTC-5, David Campbell wrote:

Giving some rational as to why defining these constants is good or bad
is a lot more useful than saying "yuck" and "I kind of agree".

You won't see the names binding these constants when you break in the debugger, but instead the integers "1" and "2". It doesn't extend to higher-dimensional arrays. The aliases will be used inconsistently, particularly by MATLAB programmers who are perfectly comfortable with dimension numbers.

Patrick

[Stefan Karpinski]

What he said. 

[David Campbell]

You can already do this sort(>, [1,2,3]), so this paradigm could just
be extended.

On Wed, Aug 29, 2012 at 5:49 PM, Gabor <g...@szfki.hu> wrote:
> Without introducing new function names a compromise could be:
>
> sort(v) - sort a vector
>
> sort(a,dim) - sort an array along dim ( "sort_each" )
>
> sort(a,dim,i) - sort an array along dim, based on index i ( "sort_by" )
>
>
> Ascending/descending order is still a question.

> --
>
>
>



--
David Campbell

[David Campbell]

I see there is also sortr. Is this for optimization reasons?

--
David Campbell

[John Cowan]

On Wed, Aug 29, 2012 at 5:49 PM, Gabor <g...@szfki.hu> wrote:
> Without introducing new function names a compromise could be:
>
> sort(v) - sort a vector
>
> sort(a,dim) - sort an array along dim ( "sort_each" )
>
> sort(a,dim,i) - sort an array along dim, based on index i ( "sort_by" )

It's awfully easy to get that sort of thing wrong if you mix up the
order of arguments in the three-argument case.

[Viral Shah]

I prefer sort(mat, :row) as an alternative to sort(mat, 2). The second argument could be mandatory for Matrices and higher dimensional arrays.

I prefer not to prefix/suffix the word row or col in the function name, because we need a general way to target various functions that work along dimensions of arrays.

sortrows in matlab is a poor name in my opinion, and we could use a better name - but as already discussed, it should not have a different meaning altogether in julia.

-viral

[BobC]

Let's try a different perspective.  Rather than thinking of sort() as something that has to be told what to do with/to its first parameter, Let's think of it as a more generic function that is sorts only a list of simple items fed to it, as a single set.  Feed it a set of column vectors, and it sorts them as a set.  Feed it a single item (value, vector or matrix), it would do nothing.  If you want to sort the elements within a vector, split it into separate elements, sort them, then reassemble the pieces.  Same for the rows or columns of a matrix.

That's our "simple" sort, that knows how to sort only a single list of simple items.  Disassembly and reassembly are left as exercises for the caller.  But as a caller, I'd sure like to have a function that would do this for me!  Let's call one form of this function "split_matrix_by()".  While Julia supports matrix slicing and dicing with operators, that only gets you a single part of the matrix, not all the parts, the set of parts.  How could we best specify these operations as parameters to a function call?  How about a universal way to specify matrix dissection in a way that's amenable to being passed as extra parameters (a recipe) to sort()?

There should be no need to actually do the external disassembly/reassembly operations, but there should be an explicit and unambiguous way to tell sort() how to attack a complex data structure like a matrix.

I wish I had a Matrix-O-Matic to do the slicing and dicing for me, 'cause I'm not sure where to go from here: My brain cell is tired.

-BobC

[Harlan Harris]

Oh, this is the split-apply-combine strategy for working with data. See: http://www.jstatsoft.org/v40/i01

We're definitely supporting this for DataFrames and presumably DataMatrices in JuliaData. It's hard to imagine it ever being as efficient as:

for col in 1:size(mat,2)
  mat[:,col] = sort(mat[:,col])
end

To do this strategy requires a type that's a list of SubArrays, with methods that let you operate on the original array via each SubArray.

You might be able to get a syntax like the following to work, potentially:

split(mat, :col) | :sort

--
 
 
 

[Kevin Squire]

Actually, If it can be made efficient, it would offer a nice way to process items in parallel, e.g., 

@parallel for row in split_matrix_by(M, 2)   # or split_matrix_by(M, :row)

    sort!(row)

end

Scala uses this split-apply-combine approach for its entire parallel collections implementation:

   http://infoscience.epfl.ch/record/150220/files/pc.pdf

Kevin

[Cunningham, Robert]

Yes, there’s a functional way to do this.  But how about parametrically?  I’d prefer a single sort() call rather than one or more lines of code.

 

But isn’t sort() a special case?  Is it worthy of having special matrix dissection parameter syntax?  Should sort() be special?  Is the split-apply-combine (SAC) strategy actually best overall?  Should sort be made simple and dumb, with only data parameters, and no special parameters?  Not even for sort order?  (Which can easily be done as part of the reassembly procedure.)

 

I dunno.  Generic vs. convenient.  SAC is a knife that can cut anything, but it takes more code.  By comparison, sort() is short and sweet, but fixing it to be as clear and unambiguous as SAC seems quite difficult. 

 

It seems sort() will always have a limit on its wisdom: What if I want to sort matrix tiles by their determinant and reassemble the matrix accordingly?  So we’ll need a simple sort() anyway.

 

Why not make that the only sort()?

 

The only reason I can think of to have a “smart” sort() would be to support in-place sorting, to obtain either or both of speed and space benefits.  But if LLVM optimizes across function calls, that explicit optimization may be a red herring, and may not be worth the time needed to implement it.  Or the time needed to fix the parameters to sort().

 

-BobC

 

 

--
 
 
 

[Jeffrey Sarnoff]

the 3am test:

      If, at 3am, i write code intended to sort the rows, or the cols of a matrix, or the deeps of a 3trix
      and it works correctly, then the functions are properly named.


On Wednesday, August 29, 2012 5:34:56 AM UTC-4, Umberto wrote:

Hi all

I find this puzzling:

julia> sort([11 3 4 2])
11 3 4 2-

julia> sort([11, 3, 4, 2])
2
3
4
11

Same issue with "sortr!".

I am working with the latest Windows build.

Thanks for any comment.