indexing with non Integer Reals is deprecated

[Christoph Ortner]

I've started to work mostly with Julia 0.4 recently since has a lot of useful tools for me. I noticed the following error message:

"indexing with non Integer Reals is deprecated"

I guess this is due to Issue https://github.com/JuliaLang/julia/issues/10154. I may well be missing a crucial point here, but I think it is a huge mistake to take this step and it is taking Julia in the wrong direction. It feels to be something along the lines of "premature optimisation". For casual scripting, or just quick testing, teaching numerical computing to beginners, converting from Matlab, and probably for many other reasons, not having to worry about types is really important.

More generally, it is important to keep the option of very simple (Matlab-style) Julia. I wonder whether users could come up with other examples where "simplicity, convenience" are sacrificed for "performance, elegance of design, etc" ? 

Christoph

[Christoph Ortner]

Here is, right away, one other  example that has annoyed me for a long time: `meshgrid`. It may not be "Julian", but it is incredibly convenient for quick experiments. I know I can get it from examples, but I don't understand why it hurts having it in Base.

Christoph

[Stefan Karpinski]

I have to confess that I find it overly finicky to require indices to be of integer type. Reals that convert cleanly to Int generally seem reasonable to me.

[Jiahao Chen]

One edge case that arises is indexing with a float where epsilon > 1:

julia> (1:10^17)[1e16:(1e16+5)] #Not the same as indexing with (10^16:10^16+5)
5-element Array{Int64,1}:
10000000000000000
10000000000000000
10000000000000002
10000000000000004
10000000000000004

Thanks,

Jiahao Chen
Staff Research Scientist
MIT Computer Science and Artificial Intelligence Laboratory

[Ivar Nesje]

That's one place off at the end of a 10 petabyte array. We're really planning for the future. But as I think about it, we've grown 10 MB to 10 GB arrays in 20 years, so who knows what the future will bring.

Personally I think learning the difference between Float and Int is a pretty basic programming skill, that is better thought at entry level, than in a "post mortem" when something really bad happened. This distinction also makes programs easier to read, because it forces one style of programming.

[ele...@gmail.com]

On Monday, March 16, 2015 at 12:13:42 AM UTC+10, Christoph Ortner wrote:

I've started to work mostly with Julia 0.4 recently since has a lot of useful tools for me. I noticed the following error message:

"indexing with non Integer Reals is deprecated"

I guess this is due to Issue https://github.com/JuliaLang/julia/issues/10154. I may well be missing a crucial point here, but I think it is a huge mistake to take this step and it is taking Julia in the wrong direction. It feels to be something along the lines of "premature optimisation". For casual scripting, or just quick testing, teaching numerical computing to beginners, converting from Matlab, and probably for many other reasons, not having to worry about types is really important.

The problem is that its fairly easy to be casually wrong when using reals as indexes.  What does 10. index? (Hint: floating point numbers can't represent 10 exactly, its 9.999999...) 

Do we round? truncate? how far is the real allowed to be away from an integer to "convert cleanly"?

Its setting people up with more ways of getting errors, and it will be harder for inexperienced programmers to debug those problems.  

Cheers

Lex

[Stefan Karpinski]

On Sun, Mar 15, 2015 at 7:24 PM, <ele...@gmail.com> wrote:

The problem is that its fairly easy to be casually wrong when using reals as indexes.  What does 10. index? (Hint: floating point numbers can't represent 10 exactly, its 9.999999...)

This is simply untrue. All integers between ±2^53 can be represented exactly as Float64s. Beyond that, many integers can be represented exactly, but not all of them.

[Christian Peel]

> More generally, it is important to keep the option of
> very simple (Matlab-style) Julia. I wonder whether users
> could come up with other examples where "simplicity,
>  convenience" are sacrificed for "performance, elegance
> of design, etc" ?

I confess that I don't like that all four of the following display in major or slightly different ways in 0.4.  Yes, Float16 is only a storage class and so on for the other types, but it still looks awkward to me that they all display differently.

exp(Float16(1))
exp(Float32(1))
exp(Float64(1))
exp(BigFloat(1))

Also, I really want to type [1:2] rather than [1:2;] for a column vector.

Even though I occasionally have problems like those above, I generally find Julia simple to learn and that I can use simple (Matlab-style) code if I want to.

--

chris...@ieee.org

[ele...@gmail.com]

Stephan, do we know that a real is always float64?

Cheers

Lex

[Christoph Ortner]

Lex, Stefan: Would it not be possible to  throw an exception when the conversion is considered "unsafe"?

[Christoph Ortner]

On Sunday, 15 March 2015 17:21:18 UTC, Ivar Nesje wrote:

That's one place off at the end of a 10 petabyte array. We're really planning for the future. But as I think about it, we've grown 10 MB to 10 GB arrays in 20 years, so who knows what the future will bring.

Personally I think learning the difference between Float and Int is a pretty basic programming skill, that is better thought at entry level, than in a "post mortem" when something really bad happened. This distinction also makes programs easier to read, because it forces one style of programming.

I agree it is easy to learn, but numerical analysts are *not* computer scientists, nor should they be expected to be. There is a huge community of numerical analysts who want to absolutely minimise coding effort; for 70% of my own projects this is also true. For those, Julia would be a natural replacement for Matlab/Octave, because it gives them the tool to then go beyond their dirty first implementation. And the exception handling facilities in Julia make it so much easier to catch bugs related to this issue than would be possible in Matlab.

[ele...@gmail.com]

Stefan,

To expand now I have a non-phone keyboard ;-S

First of all you are right about 10. my bad.

My question was "does Julia always use float64 even if its a 32 bit build?" so that a real can be assumed to at least be that size.  For float32 the integers only go up to 16 million.

Of more concern is that a real used for indexing would be likely to be calculated, I would have thought type inference would give an integer for simple index situations.  So the concern is more about what happens when a calculated real is not quite exactly the integer?

Cheers

Lex

On Monday, March 16, 2015 at 9:34:57 AM UTC+10, Stefan Karpinski wrote:

[Mauro]

On Mon, 2015-03-16 at 08:40, Christoph Ortner <christop...@gmail.com> wrote:
> Lex, Stefan: Would it not be possible to throw an exception when the
> conversion is considered "unsafe"?

That's the current behavior:

julia> [1,2][1.0]
1

julia> [1,2][1.0001]
ERROR: InexactError()
in getindex at array.jl:246

julia> [1,2][1.+eps(1.)]
ERROR: InexactError()
in getindex at array.jl:246

[ele...@gmail.com]

Christoph,

As I said in another post, because Julia infers types, most of the time indexes will just automatically be integers.

It is only when the index is calculated from something, probably from some real value.  In that case it is important that the programmer actually understands what is going on and decides explicitly how to handle non-integer indexes, and I understand in uses like statistics you have to be careful not to bias results with how you round things.

Cheers

Lex

[Andreas Lobinger]

On Monday, March 16, 2015 at 8:46:36 AM UTC+1, Christoph Ortner wrote:

I agree it is easy to learn, but numerical analysts are *not* computer scientists, nor should they be expected to be.

From wikipedia i read: Numerical analysis is the study of algorithms that use numerical approximation (as opposed to general symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics)
So i somehow expect some understanding about discretization of numbers. I agree that NAs do not need to know the background how numbers are represented in the computer.

[Stefan Karpinski]

I don't really think this is about *understanding* the difference between floats and ints. You can be well aware of the difference and still find it annoying when you are forced to explicitly do the conversion when it's obvious that there is only one thing that could possibly work and that thing is safe and correct. That's how most of Julia works – if you assign an integer-valued float to an integer-typed field or array, it just does the conversion for you. I just don't really see a particularly good reason why indexing with integer-value floats shouldn't work the same way.

[David van Leeuwen]

On a somewhat related note: if you really require non-integer indices, you can always try NamedArrays:

using NamedArrays

m = NamedArray(rand(4), ([1//1, 1//2, 1//3, 1//4],), ("weird",))

m[1//1]

m[1//2] == m[2//4]

m[1//4] == m[4]

---david

[Eric Forgy]

Just a sanity check...

I've been studying Grid.jl and one of the neat tricks there is to inherit from AbstractArray, which allows us to write our own getindex. However, the indices are not integers. I like this trick enough I am considering making it a common structure when dealing with "arrays with additional info". At one point, I also got this deprecation error and it made me nervous.

Although non-integer indexing is deprecated, am I still allowed to do it if I build my own getindex? Or, put another way, will Grid.jl still work?

Thanks.

PS: I am finally starting to do some real modeling with Julia and I have to say that although it is sometimes tough for my aging grey matter, when I do get something to work the experience is intellectual satisfying. Kudos to the Julia development team. This is a programming language "done right".

[Steven G. Johnson]

Every extant floating-point type represents 10 exactly, so that was a particularly bad example — even Float16 represents integers between ±2048 exactly.

[Steven G. Johnson]

On Monday, March 16, 2015 at 3:48:47 AM UTC-4, ele...@gmail.com wrote:

My question was "does Julia always use float64 even if its a 32 bit build?" so that a real can be assumed to at least be that size. 

 

Yes.  Floating-point literals in Julia are always Float64.

(The width of floating-point types has nothing to do with 32- vs. 64-bit CPUs, which refers to the width of addresses.  Even purely 32-bit CPUs had 64-bit floating-point registers, and in fact often supported even wider types in hardware.)

[Steven G. Johnson]

On Saturday, November 14, 2015 at 7:39:28 PM UTC-5, Eric Forgy wrote:

Although non-integer indexing is deprecated, am I still allowed to do it if I build my own getindex? Or, put another way, will Grid.jl still work?

Yes.

[Steven G. Johnson]

See also

    https://github.com/JuliaLang/julia/issues/10154

where a lot of this was discussed.

Non-integer indices often indicate a mistake on the part of the programmer (e.g. using / rather than ÷, i.e. div), and it is more reliable to force the programmer to be explicit about what was intended (both to indicate whether they want round/floor/ceil, and to correct unintentional type instabilities that often accrue from accidental floating-point indices).   It is also a headache for people implementing AbstractArray subtypes, because it creates an expectation that they will implement non-obvious getindex methods for Real types, with calls to the to_index method as needed.

I feel like 99% of the problems that people have with deprecating floating-point indices arise because they used / rather than ÷ for an integer division.   I wonder if there is some way to give a helpful warning in such cases?

[Eric Forgy]

Thanks for confirming, Steven.

Btw, this is also a really interesting read: https://github.com/JuliaLang/julia/pull/10525

[Sheehan Olver]

I didn't realize ÷ was div!  I think people would be happier if they knew they could use A[n÷2] instead of A[div(n,2)].

[Peter Kovesi]

Can I add a plea for allowing indexing with integers that are represented by a floating point type

Following the principle of 'minimum surprise': If I have an integer just that happens to be represented via a floating point type I would still like to be able to use it as an integer.

After all we are happy with the idea of allowing mathematical operations between integer and float types via  automatic promotion of the integer.  That is, we are not required to cast the integer to a float in order to multiply the two.  Indexing with integer values that are represented by floats would seem an appropriate place for automatic demotion. Please do not make me write a[round(Int, 1.0)] !

As a long time Matlab user who is attempting to reform himself I confess to having some mixed feelings about Julia.  Much of the time I find Julia very elegant and I find coding flows nicely but there are also times when I find Julia to be excessively finicky and a generator of too much 'surprise'. Indeed, sometimes I feel as if I might as well be coding in C.  I am hoping much of this will go as I learn the language better.   Still, let me say Julia is an impressive language and I greatly appreciate the work of the developers.  I am looking forward to its future development.

Cheers

Peter

[Steven G. Johnson]

On Monday, November 16, 2015 at 11:54:39 AM UTC-5, Peter Kovesi wrote:

Following the principle of 'minimum surprise': If I have an integer just that happens to be represented via a floating point type I would still like to be able to use it as an integer.

Can you give an example of where you would like to use a floating-point index that does not involve doing integer division with / (where you should probably use ÷ instead)? 

[Eric Forgy]

Hi Peter,

I've learned that Julia is flexible enough to make it do just about whatever you want it to do. If you want Julia to handle non-integer indices, you can very easily create a type MatlabArray (or something) and have its indices be non-integer and the matrix performance would be on par in terms of performance as native arrays.

The fact that base Julia doesn't allow non-integer indices does not really limit you or anyone in any way because you can simply create your own type that does what you want it to do.

[Eric Forgy]

Here is a quick example I threw together (with some clues from Tim here):

https://github.com/EricForgy/RoundingIndexArrays.jl

Indexing only works for 1-d at the moment, but hopefully it helps get the point.

[Eric Forgy]

Oops: Link to hint from Tim is here: https://groups.google.com/d/msg/julia-users/p_7-Odiei1M/uJYr3bXWBgAJ

[Peter Kovesi]

Thanks Eric.  Yes I appreciate that the language is highly flexible and one can do lots of things.  I don't want to get hung up on using indexing with integer valued floats in particular, my concern is more philosophical

For much of what I do I am wanting to solve some technical problem within an environment that involves a minimal overhead in solving the coding problem.  If, as I am developing my solution, I end up writing some code that happens to use indexing with integer valued floats, or would find a meshgid() function handy, then I do not want the language to 'get in my way'.    (Hands up everyone who has their own implementation of meshgrid() !)  Yes I know these things are not required and are not efficient but I may find them handy and I don't want to be distracted from solving my technical problem while I attend to any finicky language issues.     I want to develop my initial solution with the minimal amount of code and the minimal number of special data types.  Later, when I have things solved, I can return to the code and re-engineer it for efficiency 

It is extremely attractive that you can engineer your Julia code to be  highly efficient but I am hoping the language can develop in a way that does not compromise simplicity and convenience.

Cheers

Peter

[Christoph Ortner]

Hi Peter,

I've had multiple discussions along similar lines over the past year, since I started coding in Julia. I think I lost every single argument. Personally, I think you are right and I think this is an important issue. However, Julia is open source, so the decisions are made by those who develop the language. They are naturally strong computer scientists and have different priorities from us.

I'd love to contribute to developing Julia, but I simply don't have the time for it. It is just a question of priority: (1) I care enough to make that time - at considerable expense to my research and family life; (2) I just go with the flow and adapt. (3) I sigh wistfully and go back to Matlab which it seems might start catching up again on performance. 

For now, I've gone with (2), but a lot will depend on how the language continues to develop. I agree with your sentiment that 

>  Indeed, sometimes I feel as if I might as well be coding in C.

Only *sometimes*, however, these sometimes can get very frustrating.

All the best,

   Christoph

[Mauro]

Julia tries, and I think succeeds, in solving the two language problem.
The two language problem being that one uses one language for most
things but drops down to a fast language for bottlenecks and/or one
language to prototype one for production.

This means that Julia has to cater to a wide range of programming style
and tastes: ranging from a throw-away script to big application code.
To pick up the not-integer-indexing example: this is probably in the
"throw-away-script" category. It seems it has been deemed too much so,
thus support was removed. In my big-application-code, I'd certainly
want not-integer-indexing to error.

Note that this goes the other way too: people hoping for a more strict
approach, presumably to make big-application-code more maintainable (see
e.g. this thread discussing private-functions of modules:
https://groups.google.com/d/msg/julia-users/by3y9JEMra8/9xqvoy_xd6YJ)

I think, the devs have found good middle ground in their design of Julia
so far. As Julia is evolving, discussions on these topics are important
(although not at infinitum...).

Mauro

[ele...@gmail.com]

On Tuesday, November 17, 2015 at 10:34:03 PM UTC+10, Mauro wrote:

Julia tries, and I think succeeds, in solving the two language problem.
The two language problem being that one uses one language for most
things but drops down to a fast language for bottlenecks and/or one
language to prototype one for production.

This means that Julia has to cater to a wide range of programming style
and tastes: ranging from a throw-away script to big application code.
To pick up the not-integer-indexing example: this is probably in the
"throw-away-script" category.  It seems it has been deemed too much so,
thus support was removed.  In my big-application-code, I'd certainly
want not-integer-indexing to error.

Note that this goes the other way too: people hoping for a more strict
approach, presumably to make big-application-code more maintainable (see
e.g. this thread discussing private-functions of modules:
https://groups.google.com/d/msg/julia-users/by3y9JEMra8/9xqvoy_xd6YJ)

I think, the devs have found good middle ground in their design of Julia
so far.  As Julia is evolving, discussions on these topics are important
(although not at infinitum...).

Mauro,

Nicely put, and note that Julia allows for this wide range of features by allowing features that are defined in packages to have the same "first class" capabilities and performance as built-in features.  

Those who have a need for real indexing are able to define a package providing the feature without any penalty just because it isn't "built-in", and in this case there is even an existing template for the package in the deprecated code.

Cheers

Lex

[Milan Bouchet-Valat]

Le mardi 17 novembre 2015 à 02:46 -0800, Peter Kovesi a écrit :
> Thanks Eric. Yes I appreciate that the language is highly flexible
> and one can do lots of things. I don't want to get hung up on using
> indexing with integer valued floats in particular, my concern is more
> philosophical

I'm sympathetic to the idea of allowing indexing with reals, but so far
I haven't found very compelling examples to support my position. Could
you provide concrete use cases where this makes life easier, at least
for newcomers? Julia's development is often driven by actual
experiences of what is really useful and what isn't.


Regards

[Sheehan Olver]

For both non-integer indexing and linrange, it seems that complaints about "hard for newcomers" really mean "hard for newcomers from Matlab".  Maybe there should be a MatlabCompat.jl package that restores Matlab behaviour.  For some things, this can be done without overloading base:

julia> module MatlabCompat

            linspace(a,b,c)=collect(Base.linspace(a,b,c))

       end

julia> using MatlabCompat

julia> linspace=MatlabCompat.linspace

julia> linspace(1,2,3)

3-element Array{Float64,1}:

 1.0

 1.5

 2.0

julia> Base.linspace(1,2,3)

linspace(1.0,2.0,3)

As long as no package depends on MatlabCompat, then the issue of non-standard code will be localized.

[Jon Norberg]

On a very practical note:

I need to to do inline calculations of indexes such as

D[i+1-d/2:j-3+q*5]

where i,d,j and q are Int64 and d is a multiple of 2 i.e. d/2 should always be integer

What is the most efficient way to use an expression as above but to avoid the Warning: indexing with non Integer....... ?

Thanks, Jon

[Eric Forgy]

Hi Jon,

Have a look at this comment above: https://groups.google.com/d/msg/julia-users/sEUvnjvtryI/g_vbGQiDBgAJ

You can try:

D[i+1-div(d,2):j-3+q*5]

or the unicode equivalent:

D[i+1-d÷2:j-3+q*5]

[Jon Norberg]

Thanks Eric!

[DNF]

You can also do

D[i+1-d>>1:j-3+q*5]

which is actually slightly faster. It looks a bit obscure at first, but now I think of it as the 'halve stuff n times' operator, which is often the correct idiom.

On Wednesday, November 18, 2015 at 10:08:08 AM UTC+1, Eric Forgy wrote:

[Steven G. Johnson]

On Saturday, November 14, 2015 at 8:48:54 PM UTC-5, Steven G. Johnson wrote:

I feel like 99% of the problems that people have with deprecating floating-point indices arise because they used / rather than ÷ for an integer division.   I wonder if there is some way to give a helpful warning in such cases?

I submitted a PR to give a more helpful deprecation warning that suggests using ÷ instead of /:

https://github.com/JuliaLang/julia/pull/14049 

[Tony Kelman]

MatlabCompat.jl already exists: https://github.com/MatlabCompat/MatlabCompat.jl