Proxy server has a maximum number of proxy connections it can create
554 views
Skip to first unread message
Douglas Shuga
Oct 14, 2015, 4:36:26 PM10/14/15
to BrowserMob Proxy
Proxy server has a maximum number of proxy connections it can create
BrowserMob Proxy seems to fail after 180 connections are made:
[INFO 2015-10-14T15:04:15,272 org.eclipse.jetty.util.log] (main) Started SelectChann...@0.0.0.0:8080
[INFO 2015-10-14T15:04:15,882 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Starting proxy at address: 0.0.0.0/0.0.0.0:8081
[INFO 2015-10-14T15:04:15,897 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:15,944 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy started at address: /0:0:0:0:0:0:0:0:8081
[INFO 2015-10-14T15:04:16,010 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8082
[INFO 2015-10-14T15:04:16,011 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,012 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy started at address: /0:0:0:0:0:0:0:0:8082
[INFO 2015-10-14T15:04:16,019 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Starting proxy at address: 0.0.0.0/0.0.0.0:8083
[INFO 2015-10-14T15:04:16,020 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,020 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Proxy started at address: /0:0:0:0:0:0:0:0:8083
[INFO 2015-10-14T15:04:16,027 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Starting proxy at address: 0.0.0.0/0.0.0.0:8084
[INFO 2015-10-14T15:04:16,028 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,028 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy started at address: /0:0:0:0:0:0:0:0:8084
[INFO 2015-10-14T15:04:16,035 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8085
[INFO 2015-10-14T15:04:16,035 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy listening with TCP transport
[...]
[INFO 2015-10-14T15:21:24,625 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Starting proxy at address: 0.0.0.0/0.0.0.0:8260
[INFO 2015-10-14T15:21:24,626 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1468) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1468) Proxy started at address: /0:0:0:0:0:0:0:0:8257
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Proxy started at address: /0:0:0:0:0:0:0:0:8258
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8259
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-923) Starting proxy at address: 0.0.0.0/0.0.0.0:8261
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-14) Starting proxy at address: 0.0.0.0/0.0.0.0:8262
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Starting proxy at address: 0.0.0.0/0.0.0.0:8263
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Proxy started at address: /0:0:0:0:0:0:0:0:8259
[INFO 2015-10-14T15:21:24,633 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-16) Starting proxy at address: 0.0.0.0/0.0.0.0:8264
[INFO 2015-10-14T15:21:24,634 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8265
[INFO 2015-10-14T15:21:24,634 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-862) Starting proxy at address: 0.0.0.0/0.0.0.0:8266
[WARN 2015-10-14T15:21:24,634 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-862) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,634 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1166) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,636 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-982) Starting proxy at address: 0.0.0.0/0.0.0.0:8267
[INFO 2015-10-14T15:21:24,641 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Proxy started at address: /0:0:0:0:0:0:0:0:8260
[WARN 2015-10-14T15:21:24,641 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1472) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,642 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8248
[INFO 2015-10-14T15:21:24,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1008) Starting proxy at address: 0.0.0.0/0.0.0.0:8269
[INFO 2015-10-14T15:21:24,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1480) Starting proxy at address: 0.0.0.0/0.0.0.0:8272
[WARN 2015-10-14T15:21:24,644 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1480) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,644 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-915) Proxy started at address: /0:0:0:0:0:0:0:0:8250
[INFO 2015-10-14T15:21:24,655 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8273
[INFO 2015-10-14T15:21:24,662 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Starting proxy at address: 0.0.0.0/0.0.0.0:8275
[INFO 2015-10-14T15:21:24,662 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1481) Starting proxy at address: 0.0.0.0/0.0.0.0:8274
[INFO 2015-10-14T15:21:24,674 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1469) Starting proxy at address: 0.0.0.0/0.0.0.0:8277
[INFO 2015-10-14T15:21:24,674 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-2274) Starting proxy at address: 0.0.0.0/0.0.0.0:8276
[WARN 2015-10-14T15:21:24,675 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-2274) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,675 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-863) Starting proxy at address: 0.0.0.0/0.0.0.0:8278
[WARN 2015-10-14T15:21:24,676 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-863) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,676 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-3125) Starting proxy at address: 0.0.0.0/0.0.0.0:8268
[INFO 2015-10-14T15:21:24,677 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1470) Starting proxy at address: 0.0.0.0/0.0.0.0:8270
[INFO 2015-10-14T15:21:24,677 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1303) Starting proxy at address: 0.0.0.0/0.0.0.0:8271
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-861) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1481) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1470) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1303) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-3125) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,839 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1473) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-16) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-923) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-982) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,851 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1344) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-821) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-2273) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-14) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1008) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,853 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-19) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,867 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1166) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,880 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1475) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,888 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-17) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,888 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1469) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,898 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-924) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:37,775 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Shutting down proxy gracefully
[INFO 2015-10-14T15:21:37,775 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Closing all channels...
[INFO 2015-10-14T15:21:37,781 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Shutting down event loops
[INFO 2015-10-14T15:21:39,996 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Done shutting down proxy
[WARN 2015-10-14T15:21:51,629 org.littleshoot.proxy.impl.ClientToProxyConnection] (LittleProxy-ClientToProxyWorker-7) (NEGOTIATING_CONNECT) [id: 0xc2c0217c, /127.0.0.1:50861 => /127.0.0.1:8121]: Caught an exception on ClientToProxyConnection java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_75]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_75]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_75]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [browsermob-dist-2.1.0-beta-2.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
[WARN 2015-10-14T15:21:51,632 org.littleshoot.proxy.impl.ClientToProxyConnection] (LittleProxy-ClientToProxyWorker-1) (NEGOTIATING_CONNECT) [id: 0x975f018c, /127.0.0.1:50864 => /127.0.0.1:8121]: Caught an exception on ClientToProxyConnection java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_75]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_75]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_75]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [browsermob-dist-2.1.0-beta-2.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
(etc.)
:8261 seems to be the first to fail.
This causes an exception to be thrown in my Python script:
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/browsermobproxy/server.py", line 37, in create_proxy
client = Client(self.url[7:], params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/browsermobproxy/client.py", line 30, in __init__
jcontent = json.loads(resp.content.decode('utf-8'))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
BrowserMob Proxy seems to fail after 180 connections are made:
[INFO 2015-10-14T15:04:15,272 org.eclipse.jetty.util.log] (main) Started SelectChann...@0.0.0.0:8080
[INFO 2015-10-14T15:04:15,882 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Starting proxy at address: 0.0.0.0/0.0.0.0:8081
[INFO 2015-10-14T15:04:15,897 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:15,944 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy started at address: /0:0:0:0:0:0:0:0:8081
[INFO 2015-10-14T15:04:16,010 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8082
[INFO 2015-10-14T15:04:16,011 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,012 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy started at address: /0:0:0:0:0:0:0:0:8082
[INFO 2015-10-14T15:04:16,019 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Starting proxy at address: 0.0.0.0/0.0.0.0:8083
[INFO 2015-10-14T15:04:16,020 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,020 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-19) Proxy started at address: /0:0:0:0:0:0:0:0:8083
[INFO 2015-10-14T15:04:16,027 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Starting proxy at address: 0.0.0.0/0.0.0.0:8084
[INFO 2015-10-14T15:04:16,028 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy listening with TCP transport
[INFO 2015-10-14T15:04:16,028 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-15) Proxy started at address: /0:0:0:0:0:0:0:0:8084
[INFO 2015-10-14T15:04:16,035 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8085
[INFO 2015-10-14T15:04:16,035 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1288515448-17) Proxy listening with TCP transport
[...]
[INFO 2015-10-14T15:21:24,625 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Starting proxy at address: 0.0.0.0/0.0.0.0:8260
[INFO 2015-10-14T15:21:24,626 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1468) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1468) Proxy started at address: /0:0:0:0:0:0:0:0:8257
[INFO 2015-10-14T15:21:24,627 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Proxy started at address: /0:0:0:0:0:0:0:0:8258
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8259
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-923) Starting proxy at address: 0.0.0.0/0.0.0.0:8261
[INFO 2015-10-14T15:21:24,628 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-14) Starting proxy at address: 0.0.0.0/0.0.0.0:8262
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Proxy listening with TCP transport
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1473) Starting proxy at address: 0.0.0.0/0.0.0.0:8263
[INFO 2015-10-14T15:21:24,630 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Proxy started at address: /0:0:0:0:0:0:0:0:8259
[INFO 2015-10-14T15:21:24,633 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-16) Starting proxy at address: 0.0.0.0/0.0.0.0:8264
[INFO 2015-10-14T15:21:24,634 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8265
[INFO 2015-10-14T15:21:24,634 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-862) Starting proxy at address: 0.0.0.0/0.0.0.0:8266
[WARN 2015-10-14T15:21:24,634 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-862) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,634 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1166) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,636 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-982) Starting proxy at address: 0.0.0.0/0.0.0.0:8267
[INFO 2015-10-14T15:21:24,641 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1347) Proxy started at address: /0:0:0:0:0:0:0:0:8260
[WARN 2015-10-14T15:21:24,641 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1472) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,642 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-17) Starting proxy at address: 0.0.0.0/0.0.0.0:8248
[INFO 2015-10-14T15:21:24,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1008) Starting proxy at address: 0.0.0.0/0.0.0.0:8269
[INFO 2015-10-14T15:21:24,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1480) Starting proxy at address: 0.0.0.0/0.0.0.0:8272
[WARN 2015-10-14T15:21:24,644 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1480) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,644 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-915) Proxy started at address: /0:0:0:0:0:0:0:0:8250
[INFO 2015-10-14T15:21:24,655 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1166) Starting proxy at address: 0.0.0.0/0.0.0.0:8273
[INFO 2015-10-14T15:21:24,662 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Starting proxy at address: 0.0.0.0/0.0.0.0:8275
[INFO 2015-10-14T15:21:24,662 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1481) Starting proxy at address: 0.0.0.0/0.0.0.0:8274
[INFO 2015-10-14T15:21:24,674 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1469) Starting proxy at address: 0.0.0.0/0.0.0.0:8277
[INFO 2015-10-14T15:21:24,674 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-2274) Starting proxy at address: 0.0.0.0/0.0.0.0:8276
[WARN 2015-10-14T15:21:24,675 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-2274) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,675 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-863) Starting proxy at address: 0.0.0.0/0.0.0.0:8278
[WARN 2015-10-14T15:21:24,676 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-863) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:24,676 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-3125) Starting proxy at address: 0.0.0.0/0.0.0.0:8268
[INFO 2015-10-14T15:21:24,677 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1470) Starting proxy at address: 0.0.0.0/0.0.0.0:8270
[INFO 2015-10-14T15:21:24,677 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-1303) Starting proxy at address: 0.0.0.0/0.0.0.0:8271
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-861) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1481) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1470) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1303) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:24,677 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-3125) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,839 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1473) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-16) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-923) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,847 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-982) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,851 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1344) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-821) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-2273) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-14) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,852 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1008) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,853 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-19) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,867 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1166) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,880 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1475) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,888 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-17) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,888 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-1469) Attempted to stop proxy server, but proxy was never successfully started.
[WARN 2015-10-14T15:21:26,898 net.lightbody.bmp.BrowserMobProxyServer] (qtp1752741221-924) Attempted to stop proxy server, but proxy was never successfully started.
[INFO 2015-10-14T15:21:37,775 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Shutting down proxy gracefully
[INFO 2015-10-14T15:21:37,775 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Closing all channels...
[INFO 2015-10-14T15:21:37,781 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Shutting down event loops
[INFO 2015-10-14T15:21:39,996 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp1752741221-861) Done shutting down proxy
[WARN 2015-10-14T15:21:51,629 org.littleshoot.proxy.impl.ClientToProxyConnection] (LittleProxy-ClientToProxyWorker-7) (NEGOTIATING_CONNECT) [id: 0xc2c0217c, /127.0.0.1:50861 => /127.0.0.1:8121]: Caught an exception on ClientToProxyConnection java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_75]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_75]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_75]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [browsermob-dist-2.1.0-beta-2.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
[WARN 2015-10-14T15:21:51,632 org.littleshoot.proxy.impl.ClientToProxyConnection] (LittleProxy-ClientToProxyWorker-1) (NEGOTIATING_CONNECT) [id: 0x975f018c, /127.0.0.1:50864 => /127.0.0.1:8121]: Caught an exception on ClientToProxyConnection java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_75]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_75]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_75]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_75]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.setBytes(UnpooledUnsafeDirectByteBuf.java:447) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241) ~[browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [browsermob-dist-2.1.0-beta-2.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [browsermob-dist-2.1.0-beta-2.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
(etc.)
:8261 seems to be the first to fail.
This causes an exception to be thrown in my Python script:
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/browsermobproxy/server.py", line 37, in create_proxy
client = Client(self.url[7:], params)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/browsermobproxy/client.py", line 30, in __init__
jcontent = json.loads(resp.content.decode('utf-8'))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
je...@outlook.com
Dec 27, 2015, 10:09:23 PM12/27/15
to BrowserMob Proxy
It looks like you're starting a lot of new proxy servers, perhaps one for every request? A single proxy server can serve multiple requests. At the very least, shut down a proxy server instance after you're done using it. Running hundreds of instances simultaneously is going to eat up a lot of resources.
Douglas Shuga
Dec 30, 2015, 5:34:46 PM12/30/15
to BrowserMob Proxy
No, there's only one proxy server, but I'm trying to get it to server 100+ connections simultaneously. I'm trying to leverage browsermob proxy for parallelized selenium tests running locally to e.g. make assertions about response codes, redirect urls, timing, etc. I'm sure there's only one server running based on the processes osx shows me as I'm running this (only one instance of "java" up as it runs, which is terminated as the script ends).
je...@outlook.com
Dec 30, 2015, 5:52:11 PM12/30/15
to BrowserMob Proxy
Every time you POST to http://machine/proxy, or every time you do new BrowserMobProxyServer().start(), it creates a new proxy server instance. It's still running in the same JVM with the same process, so you won't see anything different in your process list (other than more memory consumed).
Generally, you only need one proxy server instance, even if you are serving multiple requests. If you do happen to want/need multiple instances (for example, if you're trying to isolate different users' sessions in the HAR), you should make sure to call stop() or abort() on old instances once you're done with them.
Hopefully that clarifies things!
-Jason
Generally, you only need one proxy server instance, even if you are serving multiple requests. If you do happen to want/need multiple instances (for example, if you're trying to isolate different users' sessions in the HAR), you should make sure to call stop() or abort() on old instances once you're done with them.
Hopefully that clarifies things!
-Jason
Douglas Shuga
Dec 30, 2015, 7:31:28 PM12/30/15
to BrowserMob Proxy
Ah, I see what you mean. Yeah, I'm starting multiple instances for the reason you anticipated: I need to isolate different tests' sessions. Here's how it plays out:
The main script creates an instance of browsermobproxy.Server, then calls Server.start(). It creates many subprocesses, which create their own instances of browsermobserver.Server, but only call Server.create_proxy() to create their own client instances, not start(). As the subprocesses terminate they call Client.close(), but do not call Server.stop() on their own instances of Server. When the main script finishes, it calls Server.stop().
If I try to have each subprocess call Server.stop() on their own instance of Server, I consistently see this exception raised:
My current implementation seems to work nicely (does what I want, cleans itself up), but it sounds like I can only reasonably expect it to sustain a certain amount of load.
The main script creates an instance of browsermobproxy.Server, then calls Server.start(). It creates many subprocesses, which create their own instances of browsermobserver.Server, but only call Server.create_proxy() to create their own client instances, not start(). As the subprocesses terminate they call Client.close(), but do not call Server.stop() on their own instances of Server. When the main script finishes, it calls Server.stop().
If I try to have each subprocess call Server.stop() on their own instance of Server, I consistently see this exception raised:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/Users/dshuga/indeed/qa/shield/common/test_suite_wrapper.py", line 633, in test_process
test_case_instance.teardown()
File "/Users/dshuga/indeed/qa/shield/common/test_case_skeletons.py", line 73, in teardown
self.webdriver.server.stop()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/browsermobproxy/server.py", line 110, in stop
if self.process.poll() is not None:
AttributeError: 'NoneType' object has no attribute 'poll'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dshuga/indeed/qa/shield/common/diagnostic_test_suite.py", line 69, in <module>
suite.run_test_cases(retry_failed_tests=False)
File "/Users/dshuga/indeed/qa/shield/common/test_suite_wrapper.py", line 254, in run_test_cases
run_tests()
File "/Users/dshuga/indeed/qa/shield/common/test_suite_wrapper.py", line 244, in run_tests
results_dict = dict(zip(self.unique_test_processes.keys(), results.get()))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
AttributeError: 'NoneType' object has no attribute 'poll'
My current implementation seems to work nicely (does what I want, cleans itself up), but it sounds like I can only reasonably expect it to sustain a certain amount of load.
je...@outlook.com
Jan 1, 2016, 9:08:34 PM1/1/16
to BrowserMob Proxy
Unfortunately the Python wrapper is beyond my level of comprehension, so I can't assist with that exception trace. The current REST API (which the Python wrapper uses) is pretty old and is scheduled for replacement. I think the best I can promise at this point is that the new REST API will provide a much better mechanism for stopping, starting, and expiring proxy instances, which should allow you to close unused proxies more easily.
In the meantime, are you able to achieve your goals by simply using a reduced load?
In the meantime, are you able to achieve your goals by simply using a reduced load?
Douglas Shuga
Jan 4, 2016, 11:37:06 PM1/4/16
to browserm...@googlegroups.com
In the meantime, are you able to achieve your goals by simply using a reduced load?
Yeah, it's not a crippling problem. Thanks. When the API is replaced, will the new one be backwards compatible? In other words, would upgrading break my current implementation?
On Fri, Jan 1, 2016 at 8:08 PM, <je...@outlook.com> wrote:
Unfortunately the Python wrapper is beyond my level of comprehension, so I can't assist with that exception trace. The current REST API (which the Python wrapper uses) is pretty old and is scheduled for replacement. I think the best I can promise at this point is that the new REST API will provide a much better mechanism for stopping, starting, and expiring proxy instances, which should allow you to close unused proxies more easily.
In the meantime, are you able to achieve your goals by simply using a reduced load?
--
---
You received this message because you are subscribed to a topic in the Google Groups "BrowserMob Proxy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/browsermob-proxy/kt57w6Vf8Z0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to browsermob-pro...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
je...@outlook.com
Jan 5, 2016, 2:43:07 AM1/5/16
to BrowserMob Proxy
The new REST API will not be compatible with the current REST API, but the Python wrapper will probably isolate you from many if not most of the changes. I'll keep the current REST API around for the entire 2.x line, so it's not going anywhere anytime soon.
Reply all
Reply to author
Forward
0 new messages