Request filter not triggered after calling Selenium WebDriver.navigate
117 views
Skip to first unread message
Alexander Johnson
May 5, 2015, 10:15:19 PM5/5/15
to littl...@googlegroups.com
I'm using LittleProxy to redirect browser traffic to my local machine when running tests with Selenium WebDriver. This enables me to seamlessly integrate pages which are running on developer boxes with the rest of the test environment for pre check-in testing.
I accomplish this in the requestPre method of an HttpFilters implementation. If the HttpObject type is an HttpRequest then I simply modify the URI. For instance, if a request was originally made for https://www.foo.com I change it to 127.0.0.1:8443 where 8443 is the SSL port on my local webserver.
This usually works, but I've found that there are some scenarios where the requestPre method is not called after navigating to a new page. This prevents me from redirecting the browser to a local machine and performing testing there. My local webserver is never hit, and the old version of the page in the testing environment still loads instead.
Is this a bug? Am I correct in assuming that WebDriver.navigate will always cause the browser to make an HTTP get call that should always trigger the requestPre method? What else might be happening here?
Alexander Johnson
May 5, 2015, 11:10:51 PM5/5/15
to littl...@googlegroups.com
I figured I should share some code too. Most of this isn't going to be relevant, but I want to be absolutely sure. Here's where I start the proxy:
private synchronized static HttpProxyServer generateProxyServer()
{
// Load port configuration for proxy or find a new one
int port = Config.getInteger("proxyPort");
if(port == 0) port = SystemUtils.findAvailablePort();
// Record current thread ID so filters can use it to log to the current test context
final long associatedThreadId = Thread.currentThread().getId();
// Create and start the proxy server object
HttpProxyServer proxyServer = null;
String proxyInfo = String.format("Starting proxy server on port [%s]... ", port);
try
{
HttpProxyServerBootstrap proxyServerBootstrap = DefaultHttpProxyServer.bootstrap();
proxyServerBootstrap = proxyServerBootstrap.withPort(port);
proxyServerBootstrap = proxyServerBootstrap.withFiltersSource(new HttpFiltersSourceAdapter()
{
public HttpFilters filterRequest(HttpRequest originalRequest, ChannelHandlerContext ctx)
{
return new ProxyFilters(associatedThreadId);
}
});
// Allow external connections to the proxy port if using an external browser via Selenium Grid
if(Config.getBoolean("useGrid"))
{
proxyServerBootstrap = proxyServerBootstrap.withAllowLocalOnly(false);
}
proxyServer = proxyServerBootstrap.start();
}
catch(Exception e)
{
CombinedLogger.error(proxyInfo + "FAILURE (HALTING)!", e);
}
CombinedLogger.info(proxyInfo + "success");
return proxyServer;
}
Here's my code for attaching the proxy to a WebDriver instance:
private static void configureGenericCapabilities(DesiredCapabilities capabilities)
{
if(Config.getBoolean("useProxy"))
{
// Start the proxy server
HttpProxyServer proxyServer = generateProxyServer();
Context.put("proxyServer", proxyServer);
// Get the proxy server address. Note: The NODE_NAME variable (used by Jenkins) overrides any value
// which was set for proxyServerAddress. This allows us to redirect browsers back to the Jenkins
// machines without knowing beforehand what the machine label is. If you still want to manually configure
// the proxy server address for Jenkins jobs you'll also need to set
// proxyServerAddressOverridesJenkinsNodeName to true.
String proxyServerAddress = Config.getString("proxyServerAddress");
if(!Config.getBoolean("proxyServerAddressOverridesJenkinsNodeName"))
{
String jenkinsNodeName = Config.getString("NODE_NAME");
if(jenkinsNodeName != null && !jenkinsNodeName.isEmpty())
{
String infoString = String.format("Jenkins NODE_NAME variable has been externally set to [%s]. " +
"Using this value to assign proxy server address to current " +
"node. Please set proxyServerAddressOverridesJenkinsNodeName " +
"to true if you still wish to manually define the proxy " +
"server address.", jenkinsNodeName);
CombinedLogger.info(infoString);
proxyServerAddress = jenkinsNodeName;
}
}
// Add proxy connection information to the WebDriver capabilities object
Proxy proxy = new Proxy();
proxy.setProxyType(Proxy.ProxyType.MANUAL);
proxy.setHttpProxy(String.format("%s:%s", proxyServerAddress, proxyServer.getListenAddress().getPort()));
proxy.setSslProxy(String.format("%s:%s", proxyServerAddress, proxyServer.getListenAddress().getPort()));
capabilities.setCapability(CapabilityType.PROXY, proxy);
}
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
capabilities.setCapability(CapabilityType.SUPPORTS_WEB_STORAGE, false);
}
And here's my code which handles the actual filtering:
public class ProxyFilters implements HttpFilters
{
private long associatedThreadId;
public ProxyFilters(long associatedThreadId)
{
this.associatedThreadId = associatedThreadId;
}
/**
* Performs request filtering for HTTP calls between the client and the proxy
* @param request: The request object being filtered
* @return null: Signals that we're not mocking any responses
*/
public HttpResponse requestPre(HttpObject request)
{
if(request == null)
{
String warningMessage = String.format("%s detected a null request!");
CombinedLogger.warn(warningMessage, associatedThreadId);
return null;
}
if(!(request instanceof HttpRequest))
{
String warningMessage = String.format("Unsupported HttpObject type detected! %s does not have the " +
"capability to process [%s] objects!",
this.getClass().getSimpleName(), request.getClass().getName());
if(Config.getBoolean("verboseProxy", associatedThreadId)) CombinedLogger.warn(warningMessage, associatedThreadId);
}
else
{
handleRequestURIRedirection((HttpRequest)request);
handleRequestHeaderRedirection((HttpRequest)request);
if(Config.getBoolean("reportProxyTransactions", associatedThreadId)) handleReporting(request);
}
return null;
}
/**
* Performs request filtering for HTTP calls between the proxy and the server
* @param request: The request object being filtered
* @return null: Signals that we're not mocking any responses
*/
public HttpResponse requestPost(HttpObject request)
{
// no-op since request was already handled in requestPre;
return null;
}
/**
* Performs response filtering for HTTP calls between the server and the proxy
* @param response: The request object being filtered
* @return response: The unmodified response object
*/
public HttpObject responsePre(HttpObject response)
{
if(response == null)
{
String warningMessage = String.format("%s detected a null response!");
CombinedLogger.warn(warningMessage, associatedThreadId);
return response;
}
if(Config.getBoolean("reportProxyTransactions", associatedThreadId)) handleReporting(response);
return response;
}
/**
* Performs response filtering for HTTP calls between the proxy and the client
* @param response: The request object being filtered
* @return response: The unmodified response object
*/
public HttpObject responsePost(HttpObject response)
{
// no-op since response was already handled in responsePre;
return response;
}
/**
* Performs request filtering by checking the current test context for configured redirection entries
* @param request: The request object to perform redirection on
*/
private void handleRequestURIRedirection(HttpRequest request)
{
String currentUri = request.getUri();
Map<String, String> redirections = (Map<String, String>)Context.get("redirections", associatedThreadId);
if(redirections == null) return;
for(Entry<String, String> redirection : redirections.entrySet())
{
// Get the redirection info
String redirectFrom = redirection.getKey();
String redirectTo = redirection.getValue();
// Perform redirection for URI
if(currentUri.contains(redirectFrom))
{
String newUri = currentUri.replace(redirectFrom, redirectTo);
// If the redirected URL already contains a port (IE: 127.0.0.1:8443) then we want to strip out any
// remaining port in the rest of the URI since it will cause a conflict (IE: 127.0.0.1:8443:80).
Pattern twoPortPattern = Pattern.compile("(:\\d+):\\d+");
Matcher twoPortMatcher = twoPortPattern.matcher(newUri);
while(twoPortMatcher.find())
{
newUri = newUri.replace(twoPortMatcher.group(), twoPortMatcher.group(1));
}
// Update the URI
CombinedLogger.info(String.format("Changing request URI from [%s] to [%s]", currentUri, newUri),
associatedThreadId);
request.setUri(newUri);
}
}
}
private void handleRequestHeaderRedirection(HttpRequest request)
{
Map<String, String> redirections = (Map<String, String>)Context.get("headerRedirections", associatedThreadId);
if(redirections == null) return;
for(Entry<String, String> redirection : redirections.entrySet())
{
// Get the redirection info
String redirectFrom = redirection.getKey();
String redirectTo = redirection.getValue();
HttpHeaders requestHeaders = request.headers();
Iterator<Map.Entry<String, String>> requestHeaderIterator = requestHeaders.iterator();
while(requestHeaderIterator.hasNext())
{
// Check every header
Map.Entry<String, String> requestHeader = requestHeaderIterator.next();
String requestHeaderKey = requestHeader.getKey();
String requestHeaderValue = requestHeader.getValue();
if(requestHeaderValue.contains(redirectFrom))
{
// Update any redirected URLs in the header
String newRequestHeaderValue = requestHeaderValue.replace(redirectFrom, redirectTo);
// If the redirected URL in the header already contains a port (IE: 127.0.0.1:8443) then we want to
// strip out any remaining ports from the URIs in the rest of the header since they will cause
// conflicts (IE: 127.0.0.1:8443:80).
Pattern twoPortPattern = Pattern.compile("(:\\d+):\\d+");
Matcher twoPortMatcher = twoPortPattern.matcher(newRequestHeaderValue);
while(twoPortMatcher.find())
{
newRequestHeaderValue = newRequestHeaderValue.replace(twoPortMatcher.group(),
twoPortMatcher.group(1));
}
// Replace the old header with the modified one
if(Config.getBoolean("verboseProxy", associatedThreadId))
{
CombinedLogger.info(String.format("Changing header [%s] from [%s] to [%s]", requestHeaderKey,
requestHeaderValue, newRequestHeaderValue),
associatedThreadId);
}
requestHeaders.remove(requestHeaderKey);
requestHeaders.add(requestHeaderKey, newRequestHeaderValue);
}
}
}
}
private void handleReporting(HttpObject httpObject)
{
String transactionTitle;
StringBuilder transactionDetails = new StringBuilder();
HttpHeaders headers;
// Record request details
if(httpObject instanceof HttpRequest)
{
transactionTitle = String.format("Request at %s", new Date().toString());
transactionDetails.append(String.format("URI = %s</br>", ((HttpRequest)httpObject).getUri()));
headers = ((HttpRequest)httpObject).headers();
}
// Record response details
else if(httpObject instanceof HttpResponse)
{
transactionTitle = String.format("Response at %s", new Date().toString());
transactionDetails.append(String.format("Status = %s</br>", ((HttpResponse)httpObject).getStatus()));
headers = ((HttpResponse)httpObject).headers();
}
// Can't record transaction details
else
{
String warningMessage = String.format("Unsupported HttpObject type detected! %s does not have the " +
"capability to record [%s] object transactions!",
this.getClass().getSimpleName(), httpObject.getClass().getName());
if(Config.getBoolean("verboseProxy", associatedThreadId)) CombinedLogger.warn(warningMessage, associatedThreadId);
return;
}
// Record header details
if(headers != null && !headers.isEmpty())
{
transactionDetails.append("Headers:</br><ul style=\"list-style-type:bullet\">");
Iterator<Map.Entry<String, String>> headerIterator = headers.iterator();
while(headerIterator.hasNext())
{
// Record every header
Map.Entry<String, String> requestHeader = headerIterator.next();
String requestHeaderKey = requestHeader.getKey();
String requestHeaderValue = requestHeader.getValue();
transactionDetails.append(String.format("<li>%s - %s/<li>", requestHeaderKey, requestHeaderValue));
}
}
// Save the transaction info to the test method report
Report testMethodReport = (Report)Context.get("testMethodReport", associatedThreadId);
if(testMethodReport != null) testMethodReport.setField(transactionTitle, transactionDetails.toString());
else
{
// Test method report is null so we'll record to the test set report instead.
Report testSetReport = (Report)Context.get("testSetReport", associatedThreadId);
testSetReport.setField(transactionTitle, transactionDetails.toString());
}
if(Config.getBoolean("verboseProxy", associatedThreadId)) CombinedLogger.info(String.format("Logged transaction: [%s: %s]",
transactionTitle,
transactionDetails.toString()),
associatedThreadId);
}
}
Alexander Johnson
May 5, 2015, 11:30:16 PM5/5/15
to littl...@googlegroups.com
A couple more things worth noting:
- This is not browser specific (happens in both Chrome and Firefox)
- If I navigate to another page first (IE: http://www.google.com) and then navigate back the request is intercepted as desired. However, this is not an ideal solution (having to navigate to Google before each page we actually want to visit will double the execution time).
On Tuesday, May 5, 2015 at 7:15:19 PM UTC-7, Alexander Johnson wrote:
Reply all
Reply to author
Forward
0 new messages