why Passing list-likes to .loc or [] with any missing label will raise Key Error in the future, you can use .reindex() as an alternative.

10,027 views
Skip to first unread message

Neal Becker

unread,
Nov 15, 2017, 7:43:59 AM11/15/17
to pydata
I am getting this warning from this code:

    interesting = ['channel', 'lambda', 'G_info', 'pwr', 'rate', 'arrival', 'threshold', 'ber', 'per', 'errs', 'packet_errs', 'miss/burst', 'fake/good',  'mean_dem_iter', 'random_freq', 'recheck_corr']
    cols = interesting + [c for c in df.columns if c not in interesting]
    return df.loc[:,cols]

It would seem with this construction it's impossible that any label could be missing, so I don't understand why this warning.

Joris Van den Bossche

unread,
Nov 15, 2017, 10:11:11 AM11/15/17
to PyData
There is some explanation about this in the release notes of 0.21.0: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#indexing-with-a-list-with-missing-labels-is-deprecated (see also the linked github issue). The main reason for this warning (and future key error) is that we wanted to make .loc more strict to let it only select existing labels, and no longer introduce new columns or rows with NaNs.

Does that clarify things? Otherwise feel free to ask further!

Joris

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Neal Becker

unread,
Nov 16, 2017, 7:29:00 AM11/16/17
to pyd...@googlegroups.com
But there aren't any missing columns!
cols = interesting + [c for c in df.columns if c not in interesting]
return df.loc[:,cols]
the set 'cols' must include all df.columns, by construction

To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

Tom Augspurger

unread,
Nov 16, 2017, 7:32:02 AM11/16/17
to pyd...@googlegroups.com
In `cols = interesting + [c for c in df.columns if c not in interesting]`, any cols in interesting that aren't in df.columns will be included.
Are you sure you want the `interesting +` at the front?

You probably just want cols = df.columns & interesting.

- Tom

On Thu, Nov 16, 2017 at 6:28 AM, Neal Becker <ndbe...@gmail.com> wrote:
But there aren't any missing columns!
cols = interesting + [c for c in df.columns if c not in interesting]
return df.loc[:,cols]
the set 'cols' must include all df.columns, by construction
On Wed, Nov 15, 2017 at 10:11 AM Joris Van den Bossche <jorisvand...@gmail.com> wrote:
There is some explanation about this in the release notes of 0.21.0: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#indexing-with-a-list-with-missing-labels-is-deprecated (see also the linked github issue). The main reason for this warning (and future key error) is that we wanted to make .loc more strict to let it only select existing labels, and no longer introduce new columns or rows with NaNs.

Does that clarify things? Otherwise feel free to ask further!

Joris

2017-11-15 13:43 GMT+01:00 Neal Becker <ndbe...@gmail.com>:
I am getting this warning from this code:

    interesting = ['channel', 'lambda', 'G_info', 'pwr', 'rate', 'arrival', 'threshold', 'ber', 'per', 'errs', 'packet_errs', 'miss/burst', 'fake/good',  'mean_dem_iter', 'random_freq', 'recheck_corr']
    cols = interesting + [c for c in df.columns if c not in interesting]
    return df.loc[:,cols]

It would seem with this construction it's impossible that any label could be missing, so I don't understand why this warning.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

Neal Becker

unread,
Nov 16, 2017, 7:45:19 AM11/16/17
to pyd...@googlegroups.com
This may not be the simplest way, my intention was to reorder the columns to put the "interesting" ones first

On Thu, Nov 16, 2017 at 7:31 AM Tom Augspurger <tom.augs...@gmail.com> wrote:
In `cols = interesting + [c for c in df.columns if c not in interesting]`, any cols in interesting that aren't in df.columns will be included.
Are you sure you want the `interesting +` at the front?

You probably just want cols = df.columns & interesting.

- Tom
On Thu, Nov 16, 2017 at 6:28 AM, Neal Becker <ndbe...@gmail.com> wrote:
But there aren't any missing columns!
cols = interesting + [c for c in df.columns if c not in interesting]
return df.loc[:,cols]
the set 'cols' must include all df.columns, by construction
On Wed, Nov 15, 2017 at 10:11 AM Joris Van den Bossche <jorisvand...@gmail.com> wrote:
There is some explanation about this in the release notes of 0.21.0: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#indexing-with-a-list-with-missing-labels-is-deprecated (see also the linked github issue). The main reason for this warning (and future key error) is that we wanted to make .loc more strict to let it only select existing labels, and no longer introduce new columns or rows with NaNs.

Does that clarify things? Otherwise feel free to ask further!

Joris

2017-11-15 13:43 GMT+01:00 Neal Becker <ndbe...@gmail.com>:
I am getting this warning from this code:

    interesting = ['channel', 'lambda', 'G_info', 'pwr', 'rate', 'arrival', 'threshold', 'ber', 'per', 'errs', 'packet_errs', 'miss/burst', 'fake/good',  'mean_dem_iter', 'random_freq', 'recheck_corr']
    cols = interesting + [c for c in df.columns if c not in interesting]
    return df.loc[:,cols]

It would seem with this construction it's impossible that any label could be missing, so I don't understand why this warning.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

Joris Van den Bossche

unread,
Nov 16, 2017, 7:49:42 AM11/16/17
to PyData
But still, are you sure everything in 'interesting' is actually included as a column in the dataframe? If you get that warning, it would seem not.

2017-11-16 13:45 GMT+01:00 Neal Becker <ndbe...@gmail.com>:
This may not be the simplest way, my intention was to reorder the columns to put the "interesting" ones first
On Thu, Nov 16, 2017 at 7:31 AM Tom Augspurger <tom.augs...@gmail.com> wrote:
In `cols = interesting + [c for c in df.columns if c not in interesting]`, any cols in interesting that aren't in df.columns will be included.
Are you sure you want the `interesting +` at the front?

You probably just want cols = df.columns & interesting.

- Tom
On Thu, Nov 16, 2017 at 6:28 AM, Neal Becker <ndbe...@gmail.com> wrote:
But there aren't any missing columns!
cols = interesting + [c for c in df.columns if c not in interesting]
return df.loc[:,cols]
the set 'cols' must include all df.columns, by construction
On Wed, Nov 15, 2017 at 10:11 AM Joris Van den Bossche <jorisvand...@gmail.com> wrote:
There is some explanation about this in the release notes of 0.21.0: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#indexing-with-a-list-with-missing-labels-is-deprecated (see also the linked github issue). The main reason for this warning (and future key error) is that we wanted to make .loc more strict to let it only select existing labels, and no longer introduce new columns or rows with NaNs.

Does that clarify things? Otherwise feel free to ask further!

Joris

2017-11-15 13:43 GMT+01:00 Neal Becker <ndbe...@gmail.com>:
I am getting this warning from this code:

    interesting = ['channel', 'lambda', 'G_info', 'pwr', 'rate', 'arrival', 'threshold', 'ber', 'per', 'errs', 'packet_errs', 'miss/burst', 'fake/good',  'mean_dem_iter', 'random_freq', 'recheck_corr']
    cols = interesting + [c for c in df.columns if c not in interesting]
    return df.loc[:,cols]

It would seem with this construction it's impossible that any label could be missing, so I don't understand why this warning.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

Neal Becker

unread,
Nov 16, 2017, 7:55:59 AM11/16/17
to pyd...@googlegroups.com
It says 'all interesting cols' + 'all !interesting cols', so that should be all cols

To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

Joris Van den Bossche

unread,
Nov 16, 2017, 8:06:41 AM11/16/17
to PyData
Then please provide a small reproducible example. If all labels in 'interesting' are actual existing column names, then this would be a bug in the deprecation warning. But I suspect there are actual missing columns.

Joris

To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe@googlegroups.com.

Brian Tsui

unread,
Aug 15, 2018, 1:09:16 PM8/15/18
to PyData
On a side note, I think it would be great if it can just raise a warning instead of throwing errors. Whenever there is a major change every time it means more versioning control for pandas package for code reproducibility. 
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.

Pietro Battiston

unread,
Aug 15, 2018, 2:08:24 PM8/15/18
to pyd...@googlegroups.com
Il giorno mer, 15/08/2018 alle 09.34 -0700, Brian Tsui ha scritto:
> On a side note, I think it would be great if it can just raise a
> warning instead of throwing errors.

We typically do not raise warnings just for the sake of having the user
feel guilty ;-)
Concerning the (future) change of behavior, see
https://github.com/pandas-dev/pandas/issues/15747 for the reasoning
behind.

> Whenever there is a major change every time it means more versioning
> control for pandas package for code reproducibility. 

We do not take backward incompatible changes lightly, but I guess it
does mean something that pandas version numbers are still 0.*.

Cheers,

Pietro

saif almulazem

unread,
Jan 6, 2019, 1:11:25 PM1/6/19
to PyData
Hi,
I have the same issue, and no labels missing, and the results as I need exactly, but the warning still appears, any final answer please?

Thanks,  
Reply all
Reply to author
Forward
0 new messages