beangulp loader is confused by csv module name

[kuba....@gmail.com]

Hi,

I'm having some problems using the debugger in the beangulp project.

I'm using PyCharm 2021.1.

Python 3.9.2 (default, Feb 20 2021, 00:00:00) 

[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux

When I run pytest normally everything runs fine, but if I try and use the debugger, the application bombs out with 

AttributeError: partially initialized module 'csv' has no attribute 'Dialect' (most likely due to a circular import)

My guess is that since the csv_importer was renamed to csv, the loader is getting confused with the importer module name and the csv python module. Given this only happens with the debugger, there must be some race condition. 

When I have renamed the csv importer module name to something else, my problem is resolved.

Any thoughts?

[Daniele Nicolodi]

Can you reproduce the problem with anything else than PyCharm?

Cheers,
Dan

[Daniele Nicolodi]

Also, please note that the csv importer is in the
beancount.ingest.importers.csv module in beancount v2, thus the rename
is just bringing back the old module name and not something recent.

Cheers,
Dan

[kuba....@gmail.com]

Hi Dan,

Yes, I am able to reproduce the same problem using the command line if I invoke the csv_test.py script.

(venv) bash-5.0$ git status

On branch master

Your branch is up to date with 'upstream/master'.

nothing to commit, working tree clean

(venv) bash-5.0$ python beangulp/importers/csv_test.py 

Traceback (most recent call last):

  File "/home/picasso/src/beangulp/beangulp/importers/csv_test.py", line 9, in <module>

    from beancount.parser import cmptest

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/beancount/parser/cmptest.py", line 10, in <module>

    import pytest

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/pytest/__init__.py", line 3, in <module>

    from . import collect

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/pytest/collect.py", line 8, in <module>

    from _pytest.deprecated import PYTEST_COLLECT_MODULE

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/_pytest/deprecated.py", line 13, in <module>

    from _pytest.warning_types import PytestDeprecationWarning

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/_pytest/warning_types.py", line 8, in <module>

    from _pytest.compat import final

  File "/home/picasso/src/beangulp/venv/lib64/python3.9/site-packages/_pytest/compat.py", line 43, in <module>

    from importlib import metadata as importlib_metadata

  File "/usr/lib64/python3.9/importlib/metadata.py", line 5, in <module>

    import csv

  File "/home/picasso/src/beangulp/beangulp/importers/csv.py", line 110, in <module>

    class Importer(identifier.IdentifyMixin, filing.FilingMixin):

  File "/home/picasso/src/beangulp/beangulp/importers/csv.py", line 121, in Importer

    csv_dialect: Union[str, csv.Dialect] = 'excel',

AttributeError: partially initialized module 'csv' has no attribute 'Dialect' (most likely due to a circular import)

(venv) bash-5.0$ pip freeze

astroid==2.5.2

attrs==20.3.0

beancount==3.0.0.dev0

beangulp==0.1.dev0

beautifulsoup4==4.9.3

cachetools==4.2.1

certifi==2020.12.5

chardet==4.0.0

click==7.1.2

google-api-core==1.26.3

google-api-python-client==2.1.0

google-auth==1.28.0

google-auth-httplib2==0.1.0

googleapis-common-protos==1.53.0

httplib2==0.19.1

idna==2.10

iniconfig==1.1.1

isort==5.8.0

lazy-object-proxy==1.6.0

lxml==4.6.3

mccabe==0.6.1

packaging==20.9

pluggy==0.13.1

ply==3.11

protobuf==3.15.7

py==1.10.0

pyasn1==0.4.8

pyasn1-modules==0.2.8

pylint==2.7.4

pyparsing==2.4.7

pytest==6.2.3

python-dateutil==2.8.1

python-magic==0.4.22

pytz==2021.1

requests==2.25.1

rsa==4.7.2

six==1.15.0

soupsieve==2.2.1

toml==0.10.2

uritemplate==3.0.1

urllib3==1.26.4

wrapt==1.12.1

Kind regards,

Jakub.

[Daniele Nicolodi]

On 11/04/2021 23:55, kuba....@gmail.com wrote:
> Hi Dan,
>
> Yes, I am able to reproduce the same problem using the command line if I
> invoke the csv_test.py script.

I see it too. I'm in it. In the meanwhile:

bin/python -m pytest beangulp/importers/csv_test.py

works (here at least).

Cheers,
Dan

[Daniele Nicolodi]

After thinking about it for minute: I am afraid there is no solution.

The issue is that when you invoke the tests like this:

python beangulp/importers/csv_test.py

the directory where the scripts that is execute by the interpreter is
located gets added as first entry of sys.path, thus, when the test code
does:

from beangulp.importers import csv

Python goes to interpret benagulp/importers/csv.py

which does

import csv

which, because of the entry in sys.path, resolves again to
beangulp/importers/csv.py instead than to the standard library module.

The change of sys.path is one of the main reasons why I don't make the
tests modules I write work as stand alone scripts. Sometimes it gets in
the way, like in this occasion, and sometimes it hides issues. Both
pytest and unittest allow to easily load tests from specific modules,
and I find this solution much cleaner.

I think I'll be actually remove the

if __name__ == '__main__':
unittest.main()

from this and other test modules.

Cheers,
Dan

[Martin Blais]

We *could* just rename csv.py to csvimp.py

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/435ee204-1946-3e17-1394-388ee0329431%40grinta.net.

[kuba jamro]

Thanks for the quick support on this. 

I've tried removing the __name__ == '__main__' and while that doesn't fix the problem it does at least show that the script should not be run in isolation.

In the meantime I have found that if I debug the full set of tests then I am able to debug the tests that I am interested in. While this works it does have the following disadvantages; 1) It takes more time before you are actually debugging the code you are interested in and 2) if you are debugging a piece of code deep inside the project the breakpoints could be hit from other tests and not just the ones of interest and 3) this works by chance and given a change of execution, the failure could be exposed again.

1) is not really a problem as the tests all complete very quickly however 2) is a little annoying. It can be avoided by setting up breakpoints in the tests of interest and using them as preconditions for the actual breakpoints in the core of the project but this is fiddly to set up.

In any case, I've found a way to work around this so I don't need anything to be fixed or changed right now. 

You received this message because you are subscribed to a topic in the Google Groups "Beancount" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beancount/BVI5zn4gqbA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhNbM_YbfhpCVNqrY5duv4%2BVaRCddGPhaR0aCbkO8vFxXA%40mail.gmail.com.

[Daniele Nicolodi]

On 12/04/2021 13:53, kuba jamro wrote:
> Thanks for the quick support on this. 
>
> I've tried removing the __name__ == '__main__' and while that doesn't
> fix the problem it does at least show that the script should not be run
> in isolation.
>
> In the meantime I have found that if I debug the full set of tests then
> I am able to debug the tests that I am interested in. While this works
> it does have the following disadvantages; 1) It takes more time before
> you are actually debugging the code you are interested in and 2) if you
> are debugging a piece of code deep inside the project the breakpoints
> could be hit from other tests and not just the ones of interest and 3)
> this works by chance and given a change of execution, the failure could
> be exposed again.
>
> 1) is not really a problem as the tests all complete very quickly
> however 2) is a little annoying. It can be avoided by setting up
> breakpoints in the tests of interest and using them as preconditions for
> the actual breakpoints in the core of the project but this is fiddly to
> set up.
>
> In any case, I've found a way to work around this so I don't need
> anything to be fixed or changed right now. 

As I reported before, both pytest and unittest allow to select which
tests to run from the whole set. With pytest:

python -m pytest beangulp/importers/csv_tests.py

runs only the tests defined in csv_tests.py

Cheer,
Dan