questions about adding a test to test both C-core and Python bindings changes

[cau...@gmail.com]

Hi,

I'm working with Justin Burke on adding support for server SSL certificate reloading [1].

The implementation (C-core and Python bindings) is going through review. In the mean time, I'd like to add an "end-to-end" test that tests Python bindings and C-core by doing:

1. creates one server and two clients, that will talk some basic RPC, e.g., "hello world" or "ping"

2. clientA establishes a long-lived channel before the server switches certificates; the channel should work (i.e., successfully performs RPCs) before and after the server's switch

3. clientB establishes 3 short-lived channels: one before the server's switch, which succeeds; one after that, that fails certificate verification; and then one that succeeds once clientB itself switches over to new CA to verify the server's new certificate.

I have such a "test" [2], but it's not hooked up with gRPC's (automated) test suites.

So I'm looking for guidance on how best to proceed (Justin is new to that part of the code and is not familiar), e.g.,

- should I add to somewhere under `src/python/grpcio_tests/tests`?

- is `tools/run_tests/run_tests.py -l python [--compiler ...]` the right way to run the tests?

I'm looking around to try to find out on my own, but any help appreciated!

Thanks!

[1] https://github.com/grpc/grpc/issues/10528

[2] https://github.com/cauthu/grpc/tree/master/test/core/security/server_credentials_reloading

[cau...@gmail.com]

Ok so I find that I can run

`tools/run_tests/run_tests.py -l python --compiler python2.7 -r <my test name>`

I've added my test to `src/python/grpcio_tests/tests/interop` and updated the `tests.json` accordingly.

Next question is, is there a flag to make `run_tests.py` not do `build_python.sh`? because the build step seems to recompile things everything, and that takes a while. I've just hacked the scripts to skip the build step after I build the first time, since I'm not modifying the library code, and just working on my test.

Thanks!

[Nathaniel Manista]

On Thu, Sep 21, 2017 at 9:32 AM, <cau...@gmail.com> wrote:

Ok so I find that I can run

`tools/run_tests/run_tests.py -l python --compiler python2.7 -r <my test name>`

I've added my test to `src/python/grpcio_tests/tests/interop` and updated the `tests.json` accordingly.

Next question is, is there a flag to make `run_tests.py` not do `build_python.sh`? because the build step seems to recompile things everything, and that takes a while. I've just hacked the scripts to skip the build step after I build the first time, since I'm not modifying the library code, and just working on my test.

Do you have ccache installed? Until we do away with build_python.sh entirely, using ccache in our experience eliminates the having-to-wait-ten-minutes-for-build_python.sh step in most circumstances.

Thank you for running the tests!

-Nathaniel

[Giang Nguyen]

Ah, I don't/didn't have ccache. Good to know. Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/CGMnKNv199A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAEOYnARj7%3DYT489RKthTBtUKGd0kNJpA8%2BqSwAF--zMzkY%3DPgw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Giang Nguyen]

Ah I'm running into weird error: [Errno 2] No such file or directory: 'test_lite' with all the tests:

2017-09-21 12:10:35,113 START: /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/build_python.sh

Unexpected error getting flaky tests: <type 'exceptions.ImportError'>

2017-09-21 12:10:35,116 WAITING: 0 queued, 1 jobs running, 0 complete, 0 failed

2017-09-21 12:11:04,842 PASSED: /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/build_python.sh [time=29.7sec; retries=0:0]

2017-09-21 12:11:04,847 detected port server running version 20

2017-09-21 12:11:04,877 my port server is version 20

2017-09-21 12:11:04,878 START: py27.test.unit._metadata_code_details_test.MetadataCodeDetailsTest

2017-09-21 12:11:04,881 START: py27.test.unit._channel_ready_future_test.ChannelReadyFutureTest

2017-09-21 12:11:04,884 START: py27.test.unit._cython._cancel_many_calls_test.CancelManyCallsTest

2017-09-21 12:11:04,888 START: py27.test.unit._cython.cygrpc_test.InsecureServerInsecureClient

2017-09-21 12:11:04,892 WAITING: 58 queued, 4 jobs running, 0 complete, 0 failed

2017-09-21 12:11:06,956 ++ dirname /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/run_python.sh

+ cd /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/../../..

++ realpath py27/bin/python

+ PYTHON=/home/cauthu/upstream-grpc/py27/bin/python2.7

++ pwd

+ ROOT=/home/cauthu/upstream-grpc

+ /home/cauthu/upstream-grpc/py27/bin/python2.7 /home/cauthu/upstream-grpc/src/python/grpcio_tests/setup.py test_lite

running test_lite

I0921 12:11:05.660718415    6566 ev_epoll1_linux.c:93]       grpc epoll fd: 9

D0921 12:11:05.660756260    6566 ev_posix.c:107]             Using polling engine: epoll1

D0921 12:11:05.660795832    6566 dns_resolver.c:301]         Using native dns resolver

error: [Errno 2] No such file or directory: 'test_lite'

2017-09-21 12:11:06,957 FAILED: py27.test.unit._cython._cancel_many_calls_test.CancelManyCallsTest [ret=1, pid=6558]

2017-09-21 12:11:06,957 START: py27.test.unit._credentials_test.CredentialsTest

2017-09-21 12:11:06,963 WAITING: ETA 127.1 sec; 57 queued, 4 jobs running, 1 complete, 1 failed

2017-09-21 12:11:06,969 ++ dirname /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/run_python.sh

+ cd /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/../../..

++ realpath py27/bin/python

+ PYTHON=/home/cauthu/upstream-grpc/py27/bin/python2.7

++ pwd

+ ROOT=/home/cauthu/upstream-grpc

+ /home/cauthu/upstream-grpc/py27/bin/python2.7 /home/cauthu/upstream-grpc/src/python/grpcio_tests/setup.py test_lite

running test_lite

I0921 12:11:05.662446075    6569 ev_epoll1_linux.c:93]       grpc epoll fd: 9

D0921 12:11:05.662486586    6569 ev_posix.c:107]             Using polling engine: epoll1

D0921 12:11:05.662526117    6569 dns_resolver.c:301]         Using native dns resolver

error: [Errno 2] No such file or directory: 'test_lite'

In the end all test fail.

Trying to looking into `class TestLite` in `commands.py`.

Other tips? Thanks!

[Giang Nguyen]

Huh,

Looks like the `loader.loadTestsFromNames(['tests'])` line is where things go wrong. I guess the `test_lite` is somehow passed to my test scripts, which do not expect it.

    mod = imp.load_module(fullname, self.file, self.filename, self.etc)

  File "/home/cauthu/upstream-grpc/src/python/grpcio_tests/tests/interop/server_ssl_credentials_reload_test/greeter_client.py", line 25, in <module>

    server_ca_pem = open(sys.argv[1]).read()

IOError: [Errno 2] No such file or directory: 'test_lite'

I must be hooking into the test suites incorrectly. Looking...

[Giang Nguyen]

OK, so I temporarily fix my script to prevent the error, but now I have a more mysterious issue: looks like `loadTestsFromNames()` doesn't return? I've modified `commands.py` like so:

        import tests

        loader = tests.Loader()

        with open('/tmp/bar.txt','a') as fp:

            fp.write('loader is {}\n'.format(loader))

        try:

            zzz = loader.loadTestsFromNames(['tests'])

            with open('/tmp/bar.txt','a') as fp:

                fp.write('DONE loading zzz= {}!\n'.format(zzz))

        except Exception as exc:

            with open('/tmp/bar.txt','a') as fp:

                fp.write(str(exc))

        runner = tests.Runner()

And here's attempt to run:

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ cat /tmp/bar.txt 

cat: /tmp/bar.txt: No such file or directory

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ 

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ ./run_tests.py -l python --compiler python2.7 -r SanityTest

Unexpected error getting flaky tests: <type 'exceptions.ImportError'>

PASSED: /home/cauthu/upstream-grpc/tools/run_tests/helper_scripts/build_python.sh [time=1.8sec; retries=0:0]2017-09-21 13:36:47,390 detected port server running version 20

2017-09-21 13:36:47,418 my port server is version 20

SUCCESS: All tests passed

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ 

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ cat /tmp/bar.txt 

loader is <tests._loader.Loader object at 0x7f98821aca10>

cauthu @ laptop:~/upstream-grpc/tools/run_tests$ 

So it claims that all tests passed, there doesn't appear to be any exception with `loadTestsFromNames`, and yet `DONE loading!` is not appended to `/tmp/bar.txt`?

I'm super confused!? Any help appreciated. Thanks!

[Giang Nguyen]

OK, further issues caused by my new directory of test scripts; moved it out of the tree and the `SanityTest` is being run and working as expected.

[Nathaniel Manista]

On Thu, Sep 21, 2017 at 1:54 PM, Giang Nguyen <cau...@gmail.com> wrote:

OK, further issues caused by my new directory of test scripts; moved it out of the tree and the `SanityTest` is being run and working as expected.

I think I may have lost track of what you're trying to do, but if you're adding new tests, why not add them in the existing test tree and add them to the grpcio_tests/tests/tests.json list-of-tests that _sanity_test.SanityTest reads? It would be unexpected that you would have to make changes to the loader and runner modules.

-Nathaniel

[Giang Nguyen]

Yea it was my incorrect use/hooking into the test suites.

I have scripts that use sys.argv while not being run... And that is causing "loadTestsByNames()" to fail because my scripts are using sys.argv while being imported.

And that is causing everything to fail. I have moved my use of sys.argv and things are much better now!

Thanks!

(sent from phone)