read_csv( skiprows ) note working for bad rows.

[Vincent Davis]

I was trying to use skiprows to skip rows that are bad, but it does not work. Am I doing something wrong or is this a bug?

First this works. You can get the data here is you would like to try it yourself.

denverChar = pd.read_csv('real_property_residential_characteristics.csv', quotechar='"', warn_bad_lines=True, error_bad_lines=False, na_values="", low_memory=False,)

I get this

b'Skipping line 62070: expected 46 fields, saw 47\nSkipping line 62073: expected 46 fields, saw 47\nSkipping line 62076: expected 46 fields, saw 47\nSkipping line 66662: expected 46 fields, saw 48\n'

But if I try to use skiprows, skipping the rows listed above.

denverChar = pd.read_csv('real_property_residential_characteristics.csv', quotechar='"', skiprows = [62070, 62073, 62076, 66662], na_values="", low_memory=False,)

CParserError                              Traceback (most recent call last)
<ipython-input-26-ca2e2986e191> in <module>()
      1 import pandas as pd
----> 2 denverChar = pd.read_csv('real_property_residential_characteristics.csv', quotechar='"', skiprows = [62070, 62073, 62076, 66662], na_values="", low_memory=False,)

/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
    472                     skip_blank_lines=skip_blank_lines)
    473 
--> 474         return _read(filepath_or_buffer, kwds)
    475 
    476     parser_f.__name__ = name

/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    258         return parser
    259 
--> 260     return parser.read()
    261 
    262 _parser_defaults = {

/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py in read(self, nrows)
    719                 raise ValueError('skip_footer not supported for iteration')
    720 
--> 721         ret = self._engine.read(nrows)
    722 
    723         if self.options.get('as_recarray'):

/Users/vincentdavis/anaconda/envs/py35/lib/python3.5/site-packages/pandas/io/parsers.py in read(self, nrows)
   1168 
   1169         try:
-> 1170             data = self._reader.read(nrows)
   1171         except StopIteration:
   1172             if nrows is None:

pandas/parser.pyx in pandas.parser.TextReader.read (pandas/parser.c:8094)()

pandas/parser.pyx in pandas.parser.TextReader._read_rows (pandas/parser.c:9157)()

pandas/parser.pyx in pandas.parser.raise_parser_error (pandas/parser.c:22231)()

CParserError: Error tokenizing data. C error: Expected 46 fields in line 62070, saw 47

[Goyo]

El jueves, 1 de octubre de 2015, 6:23:56 (UTC+2), Vincent Davis escribió:

I was trying to use skiprows to skip rows that are bad, but it does not work. Am I doing something wrong or is this a bug?

Bad line warnings count lines from 1 but skiprows counts from 0 so you have to substract 1 to the line numbers. This works for me (I don't know why my bad lines are different than yours but anyway):

>>> import pandas as pd
>>> fname = 'real_property_residential_characteristics.csv'
>>> df = pd.read_csv(fname, quotechar='"', warn_bad_lines=True, error_bad_lines=False)
Skipping line 62706: expected 46 fields, saw 47
Skipping line 62709: expected 46 fields, saw 47
Skipping line 62712: expected 46 fields, saw 47

Skipping line 67502: expected 46 fields, saw 48

>>> skiprows = [62705, 62708, 62711, 67501]
>>> df = pd.read_csv(fname, quotechar='"', skiprows=skiprows)
>>>

Regards

Goyo

[Vincent Davis]

On Friday, October 2, 2015 at 2:12:11 AM UTC-6, Goyo wrote:

Bad line warnings count lines from 1 but skiprows counts from 0 so you have to substract 1 to the line numbers. This works for me (I don't know why my bad lines are different than yours but anyway):

Ok, That works. Maybe your line numbers are different because they updated the file.