Authors: Emre Kıcıman and Benjamin Livshits
Date: SOSP 2007
Novel idea: Since changes can be made instantaneously to Web 2.0 applications, we can perform on-the-fly, per-user JavaScript rewriting. We can also take advantage of large userbases in order to distribute heavyweight instrumentation and crowdsource testing.
Main results: The authors built a prototype proxy that instruments web applications using on-the-fly JavaScript rewriting and a set of testing policies, some of which employ distributed instrumentation to gain particularly detailed insights into application performance.
Impact: The browser terrain varies greatly, and it is almost impossible to do thorough testing of a sophisticated web application before deployment. Therefore, gathering statistics from actual users may be one of the only ways to isolate and fix certain bugs.
Evidence: The authors instrument several applications with AjaxScope. They employ an adaptive policy to drill down into slow calls, a distributed policy in order to detect memory leaks (a costly operation), and an optimization policy in order to determine the impact of a potential application change. They find that while full-performance profiling instruments a median of 89 points per application, their drill-down profiler only instruments a median of 3 points. They were able to find several memory leaks using their distributed test, which may have been impossible to diagnose otherwise. Finally, their A/B test allows them to identify functions with a high probability of benefiting from cache optimization.
Prior/competitive work: In comparison to what I am familiar with, AjaxScope is unique.
Reproducibility: Apparently as of April 2009 one can download AjaxView [1], which is an implementation of the ideas of this paper. Consider my curiosity piqued.
Praise: AjaxScope seems to have a low barrier to entry, since it does not require extensive modification of existing applications.
Paper Title |
|
AjaxScope: A Platform for Remotely Monitoring |
the Client-Side Behavior of Web 2.0 Applications | |
Author(s) |
Emre Kıcıman and Benjamin Livshits |
Date | 2007 |
Novel Idea | Broaden the possibilities for testing and data mining of AJAX webapp use by introducing dynamic and distributed instrumentation for javascript. |
Main Result(s) |
They describe AjaxScope, a platform for deploying a range of such instrumentations. They divide these into categories, of which they study two in particular: performance monitoring. and runtime analysis and debugging. It functions by sitting in between the webapp servers and the client's browser, passing requests through unmodified and potentially rewriting responses in accordance with the current instrumentation policy. From this vantage point, it can instrument likely sources of error and introduce logging triggered by client errors. It can also react differently to the same input over time, allowing large applications to be thoroughly instrumented without causing noticeable slowdowns to any individual user. Since any successful web app comes with a free, now-highly-observable population of users, such instrumentation can simply be divided among them. |
Impact | A good number of the papers that cite this one seem to be specifically interested in using the techniques it presents to track down security vulnerabilities or enforce security protocols. The rest seem more in line with this paper's direct goals, benchmarking and tracing app usage. |
Evidence |
To test adaptive profiling, they compare a naive full instrumentation and an adaptive 'drill down' instrumentation (which automatically zeroes in on the bottlenecks in code by making successive refinements to the instrumentation served to each user) on a number of different applications. They found modest execution time improvements, and slightly better logging overhead improvements. However, neither effect was particularly pronounced. To test distributed profiling, they randomly applied a profiling mechanism to parts of the DOM for various users, aggregating the results to obtain a complete picture. Success here was more clear-cut, with equally good profiling results obtained with sometimes up to a second and a half less latency per user than with full instrumentation. |
Prior Work | They claim to be the first group to really look at instrumenting client-side code. |
Reproducibility | They don't really describe their code very much. |
Question | Why doesn't adaptive instrumentation provide more of a speed-up? The logging gains look kind of impressive, but they aren't really the main issue - companies tend to care a lot more about how long their site takes to load than wasting a few megabytes of logfile. It seems like such a clever idea, and as they illustrate with distributed instrumentation, the number of instrumented code points really does make a difference - so why doesn't it here? |
Criticism | The ideas presented in this paper seem very strong. From the references out there, it seems that they have gained traction, though I'm a little surprised there haven't been more (Google shows 43, though maybe that is plenty for only having been out 3 years?). The only weak point seems to be how little they talk about their actual implementation; however, this probably doesn't matter a lot, as the main purpose/impact of the paper is to get the ideas out there, not market a particular instantiation of them. |
Ideas for further work | There's lots more to do -- this paper only lays the groundwork. Starting to build applications that leverage the ideas it presents seems to be the next step.. I suppose there is also room for some more rigorous theoretical work into when/which of their ideas are really worth it. |