replacing Python-based out-of-process HTTP testserver with in-process C++ based net/server/http_server

[Paweł Hajdan, Jr.]

With Python-based test server flakily failing to start (https://code.google.com/p/chromium/issues/detail?id=96594) in a hard to debug way, I wonder if it would be a good idea to replace it with net/server/http_server, which it looks like we could start in-process, and have much more fine-grained info about its state.

Now a possible concern is that our test tool would be also using some code under test. While this is not a concern for code under say content or chrome, it may be one for net.

What do you think? This is one of the biggest flake generators in our testing infrastructure, and I'd like to clean it up.

Paweł

[Ryan Sleevi]

I have serious concerns about this proposal.

The use of the out-of-test server for SSL/TLS testing has been vital
for interoperability testing. There is no way we can reasonably induce
the many broken or odd SSL/TLS behaviours, and testing against the
same library is a recipe for failure. Given that we also maintain two
different libraries (OpenSSL for Android, NSS for other platforms -
and in the past, we had more), it's also unreasonable to think we'd be
intentionally inducing broken behaviours into these libraries for
testing purposes.

Additionally, testing on mobile devices relies on the fact that our
of-of-process design makes it rather easy to swap in localhost with a
remote address (eg: available via ADB), without requiring running
local native code.

I'm extremely sympathetic to flaky tests, but I think the proposal to
scrap it all carries with it significantly more cost and risk than
attempting to debug the test server, at least at present.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>
>
>

[Scott Hess]

I think it would be interesting to have a more captive test server, though.  For instance, you could envision a separate process spawned from the same executable, with standard IPC communication to let the parent process manipulate the server side.  It could devolve to the worst of mock-based unit tests, I suppose.

-scott

[Jonathan Dixon]

As a data point, many java-based Android tests already have a java based local test server and it's proven a reliable approach w.r.t stability.

Server: https://src.chromium.org/viewvc/chrome/trunk/src/net/test/android/javatests/src/org/chromium/net/test/util/TestWebServer.java?revision=188447&view=markup

Example usage: https://code.google.com/p/chromium/codesearch#chromium/src/android_webview/javatests/src/org/chromium/android_webview/test/GetTitleTest.java&q=loadFromUrlAndGetTitle&sq=package:chromium&type=cs&l=63

On mobile, a lot of the benefit comes from avoiding the need to either run python test script on the target device, or forward TCP ports back to a host machine where the test script is running and synchronize the startup & tear-down across machines.

For interop & perf testing, host-based test server is pretty unavoidable... for this a mix of lighttpd and testserver.py are also used.

[Paweł Hajdan, Jr.]

On Tue, Mar 19, 2013 at 12:37 PM, Ryan Sleevi <rsl...@chromium.org> wrote:

I have serious concerns about this proposal.

The use of the out-of-test server for SSL/TLS testing has been vital
for interoperability testing. There is no way we can reasonably induce
the many broken or odd SSL/TLS behaviours, and testing against the
same library is a recipe for failure. Given that we also maintain two
different libraries (OpenSSL for Android, NSS for other platforms -
and in the past, we had more), it's also unreasonable to think we'd be
intentionally inducing broken behaviours into these libraries for
testing purposes.

I was expecting a concern like that.

Alright, now what if we used mongoose instead? https://code.google.com/p/mongoose/

It's already in src/third_party, used for ChromeDriver I think, and we could also possibly use it for tests.

What do you think about that?

Paweł 

[Ryan Sleevi]

mongoose does not address the SSL/TLS testing concerns.

I'm all for reducing our dependency on the test server - there are a
lot of ways we can already simulate resource loads in-proc. There are
a number of tests in the network that, for example, fully simulate a
connected socket without actually requiring any socket activity,
allowing us to test the higher layers without any dependencies on
actually making a connection.

But I don't see us getting away from needing to have an out of process
test server. So now we're talking about 2 (or N) ways to do testing.
And then we're further adding the complexity of no longer being able
to write tests in Python (like the XMPP testing for Sync or policies)
by switching to something native - where we'd need to rewrite all of
that infrastructure as well, if I understand your proposal.

[Tomasz Mikolajewski]

Some time ago we've written an in-process http server which is reliable and tested (src/chrome/browser/google_apis/test_server/) and runs on the IO thread. Since then we removed support for Drive from the Python server and stopped using it. The new server doesn't need to be run in a browser test, so it is much faster. It is also designed to be used for testing, using our callback architecture.

Tomasz.

[Paweł Hajdan, Jr.]

Oh, cool!

Any concerns about using that one for everything except net_unittests? Most of the problems with Python test server appear outside of net_unittests anyway, so this would solve a majority of the problems here.

[Tomasz Mikolajewski]

I think we can use it. It is used to test Drive integration, for the new Streams api, as well as Contacts. I think that this test server is pretty good because it is super fast, simple, reliable and *tested*. 

[Paweł Hajdan, Jr.]

I'd like to move it to src/content/test then (from src/chrome/browser/google_apis/test_server).

Any concerns before I do that?

Paweł

[John Abd-El-Malek]

On Fri, Apr 5, 2013 at 10:00 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

I'd like to move it to src/content/test then (from src/chrome/browser/google_apis/test_server).

I assume you mean content/public/test since it's likely that it'll have to be configured by tests in chrome?

Is content the best place for this? why not src/net? 

[Paweł Hajdan, Jr.]

On Fri, Apr 5, 2013 at 11:06 AM, John Abd-El-Malek <j...@chromium.org> wrote:

I assume you mean content/public/test since it's likely that it'll have to be configured by tests in chrome?

Right. Thank you for making this more precise.

 

Is content the best place for this? why not src/net? 

I think content is right.

I ruled out src/net, because it will continue to use a fully independently implemented server to test the network stack (otherwise bugs in the network stack could "cancel themselves out" when the test server would be made of the same bits). In other words, testing code should not be heavily based on the code under test, and that would be the case if the server lived in src/net.

For content and above this is not a concern - we just need to serve some pages locally.

Paweł 

[John Abd-El-Malek]

On Fri, Apr 5, 2013 at 2:40 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

On Fri, Apr 5, 2013 at 11:06 AM, John Abd-El-Malek <j...@chromium.org> wrote:

I assume you mean content/public/test since it's likely that it'll have to be configured by tests in chrome?

Right. Thank you for making this more precise.

 

Is content the best place for this? why not src/net? 

I think content is right.

I ruled out src/net, because it will continue to use a fully independently implemented server to test the network stack (otherwise bugs in the network stack could "cancel themselves out" when the test server would be made of the same bits). In other words, testing code should not be heavily based on the code under test, and that would be the case if the server lived in src/net.

These seems like reasons not to run net's own tests using that server. That doesn't mean that a C++ http server doesn't belong there.

[Paweł Hajdan, Jr.]

The problem with src/net is naming. We have Python test server and this C++ test server. Former is out-of-process and latter is in-process. The Python one also has a special case for Android (LocalTestServer vs. RemoteTestServer) which exists because apparently one can't launch new processes on the phone itself, so the test server is running on another machine. This is not a concern for an in-process test server.

During code review it has been brought up that it's better to distinguish between the servers based on in-process or out-of-process mode, rather than Python vs C++.

Now this creates a rather verbose naming scheme: net/test/test_server files would move to net/test/test_server/out_of_process directory, making room for new net/test/test_server/in_process directory. net::TestServer would move to net::OutOfProcessTestServer, making room for net::InProcessTestServer.

I'm not really a fan of the above naming scheme. For me net::TestServer (in net/test, as is now) vs. content::TestServer (in content/test) would be cleaner.

I know there are advantages of having that code in net, but all my attempts to fit that lead to the ugliness.

Ideas are welcome.

Paweł

[Ryan Hamilton]

I continue to believe that moving this server into net/ is the right idea.  Finding the best name is always a bit tricky, but having a slightly longer class name seems highly preferable to having this code live in content/  (where it would be inaccessible to code in net/ that might wish to use it).

Cheers,

Ryan

[John Abd-El-Malek]

What about using one interface for in-process and out-of-process, and just having the factory method pick the implementation based on that? Then there would be one (public) name.

[Ryan Sleevi]

For in-proc servers, the caller is going to have to implement all of
the testing code within their unit test - that is, there is no generic
factory function to call, because there's no "generic" test server
here that contains all of the logic for unit tests.

For out of proc servers (local or remote), all such logic resides
within the TestServer's python code.

[Paweł Hajdan, Jr.]

Alright, the names du jour are: EmbeddedTestServer for the C++ in-process one and SpawnedTestServer for the Python out-of-process one.

Corresponding directory structure would be net/test/{embedded,spawned}_test_server .

What do you think?

Paweł

[Ryan Hamilton]

SGTM