11/10 Reviews - X-Trace

[Rodrigo]

Please post your reviews here.

Since I am one of the authors, you may not feel comfortable writing
criticism. Criticism is good, I won't mind it. You can disguise it as
questions, or not write it if you don't want to. Either way, that
won't affect your grade, as long as the review makes sense.

Rodrigo

[Andrew Ferguson]

Paper Title
"X-Trace: A Pervasive Network Tracing Framework"

Authors
Rodrigo Fonseca, George Porter, Randy H. Katz, Scott Shenker, Ion Stoica

Date
NSDI 2007

Novel Idea
The novel idea in X-Trace is to place tracing tags in protocols at all
layers, and to have those tags carry through communication in the wide
area. X-Trace tags can be placed in locations such as TCP options, or
HTTP or e-mail headers, some of which will be automatically preserved,
even by systems which don't understand X-Trace.

Main Results
X-Trace is implemented in several protocols (HTTP, SIP, Email, TCP,
DNS, SQL, etc.) and has been deployed across several large systems
including Hadoop and Coral Cache CDN. The paper discusses how X-Trace
was implemented in a standard 3-tier web application, and in the I3/
Chord overlay network.

Impact
I believe X-Trace's impact is currently small but growing -- because
it requires changes to existing components in order to be fully
useful, there is a significant lag before it can have a large impact.

Evidence
The two examples are well-chosen because they offer different
perspectives on how easy or difficult it can be to implement X-Trace.
In both examples, it was clear how X-Trace can help a programmer or
operator better understand the system.

Prior Work
X-Trace is built on a lot of prior work in tracing, ranging from basic
systems such as traceroute through a lot of distributed tracing
systems such as Project5, Pip, Magpie, and commercial systems such as
Splunk and Openview.

Competitive Work
Some of the prior work can be considered competitive, such as Project
5, and others complementary, such as Pip. Each tracing platform fills
a particular niche and should be chosen based on the programmer's or
operator's needs.

Reproducibility
Yes, the results should be reproducible as the source material has
been released.

Question
Does X-Trace now work with non-tree structures? What changes were
required? (the paper says "metadata described in this work assumes ...
tree structure")

Criticism
I would like to have seen a greater discussion of X-Trace's effect on
the application's performance. What if a different backend (instead of
Postgres) is used? What were the hotspots in the 15% decrease in
throughput? Was a database connection already open for the
application? What is the overhead on an X-traced DBMS?

Ideas for further work
What would an X-Trace of Paxos / ZooKeeper look like?
Can multiple trace reports be correlated, to limit the effect of
accidentally dropped trace packets?

[Dan Rosenberg]

Paper Title
X-Trace: A Pervasive Network Tracing Framework

Authors

Rodrigo Fonseca, et al.

Date
April, 2007

Novel Idea
The authors propose X-Trace, a system that incorporates metadata into network
protocols to allow detailed reporting of path information for use in diagnosing
problems in distributed settings.

Main Result

X-Trace may provide users and network operators important diagnostic data for
troubleshooting.

Impact
Somewhere between "mind-blowing" and "earth-shattering".  </suckup>

Evidence
The authors implemented and deployed X-Trace in several settings, where it
successfully allowed the diagnosis of faults that would otherwise be difficult
to detect.

Prior Work
X-Trace provides an alternative to previous tracing technologies, such as Pip
and Magpie.

Reproducibility
X-Trace is open source and easily available, so reproducing these findings is
simply a matter of obtaining the appropriate hardware.

Criticism
Requiring implementation within each application may limit X-Trace's adoption.
Security could be discussed in further detail, especially in regards to the
fact that allowing clients to perform traces against networks may gain access
to useful information that could be leveraged in an attack.  As JJ mentioned,
tracing may not be appropriate to troubleshoot all networking problems,
especially those that occur during high traffic periods (15% overhead may be
too much in those cases).

Questions/Ideas for Further Work
Does the use of protocol extension fields for metadata break other uses of
those fields (for example, TCP options, which are increasingly used, especially
MSS)?  I would be curious to see further experiments with only partial
reporting implemented, and across multiple AS's.

[Sunil Mallya]

Paper Title
X-Trace: A Pervasive Network Tracing Framework.
Author(s)
Fonseca, Rodrigo, Porter, George, Katz, Randy H., Shenker, Scott, and
Stoica, Ion
Date
NSDI'07

Novel Idea
An extensive framework for tracing the entire system and providing a
comprehensive view, but it differs by the fact that it provides varied
visibility amongst different network levels.

Main Result(s)

The main idea of this paper is to provide a framework to give total
view of the systems on which xtrace is implemented. Xtrace unlike
other systems is capable of tracing multiple applications in the
entire system but requires modification of the protocols on which it
is deployed.
Xtrace has the following 3 good design principles: - trace request is
sent in band, trace data is sent out of band and finally entity that
requests tracing is decoupled from the entity that recieves trace
report. Xtrace works by grouping all the network operations of a
particular task with an indentifier and inserts metadata along with
this task ID into the request packet at each layer in the application
stack. This tracing data is now used to construct a task tree with
casual relationship between its edges and this can be used to identify
any problems in the system. Xtrace was able to find bugs efficiently
and the location of the same in the system it was deployed on. I
really liked the fact that in the paper the authors are candid and
discuss about what xtrace can’t do.

Impact
The fact that xtrace gives different visibility across the system
stack makes it an ideal choice for enterprise tracing as information
would be available specific to the privileges of the entities. I feel
this may set a trend for other tracing applications to abstract out
unprivileged information.

Evidence
Xtrace was deployed and evaluated on a website hosting site and an
overlay network. Xtrace was able to detect non trivial bugs in the
systems that it was implemented on.

Question

- Are there any protocol issues that occurred during evaluation
because of the modifications to protocols or due to insertion of
metadata ?
- How much of an overhead does xtrace introduce to the system being
traced?
- Can Xtrace validate itself? How do we know if xtrace has a technical
snag!! (if possible)?

Criticism

Only critisim that I have is that most of the protocols on which
xtrace was implemented have extended headers or extended fields like
(http, sip), for it to still trace a protocol that doesn’t provide
these extra fields requires reimplementation of these protocols which
may introduce other problems.

Ideas for further work

Xtrace reports can be sent to an automated process which could
dynamically construct the tree and help identify the problems or bugs
while the system is still running. Here we need not wait for the
entire trace report to detect problems.
Ensuring that all trace reports reach the trace sink, so that complete
tracing data is available for the offline tree construction.

[joeyp]

X-Trace: A Pervasive Network Tracing Framework

Fonseca et al
NDSI 2007

The paper presents a framework for tracing distributed systems and
networks across many levels. Part of the goal of the system is to
find errors in accomplishing simple tasks in many different places in
the network. It accomplishes this with instrumentation to collect
relevant tracing data at many levels, so the data can be analyzed
after the fact with respect to the tasks it was tracing and the
different components involved in completing the task.

The results suggest that this type of tracing can help describe a wide
variety of scenarios, even for a single fault scenario. The
distinction here is that undesirable behavior doesn't have to mean
looking for a single bug - there could be dozens of requests for the
same web page that go slowly, with a myriad of different causes. This
work would like to be able to narrow down and identify those causes.

The paper makes two important requirements of tracing data:

1. The trace information should be sent *in-band* with requests - by
adding it to some extra space in protocol metadata. This way,
trace information lives or dies with the request it is tracing.
2. Tracing information should be recorded *out-of-band* from the
requests that are being traced. This is so that (barring faults
in the reporting infrastructure, which are possible), a failure
along a certain protocol will not disrupt logging information
before that point.

These guarantees about the way tracing and recording are designed to
give accurate traces in the event of faults.

The primary difficulty with this work is the extent of the
instrumentation across tools and platforms. This requires significant
effort, and cannot be done in some cases if a system contains
proprietary software that cannot be instrumented. The rest of the
system can traced, however, and it may be possible to narrow down the
problem to the proprietary component, since the tracing data is sent
in band and the components before and after the one in question are
instrumented.

The causal model discussed in the paper might not be gaining much from
the distinction between "next" and "down" edges. If the causal
connection is going down and back up, it should be easy to tell (since
the task id is maintained) that this came back up to the same level of
protocol again. Just the task id, with connecting edges, along with
time information and whatever else the instrumenter wants recorded,
needs to be in the flow.

This is the first paper for which I can actually attest that some of
the results are reproducible. XTrace is written and does, in fact,
appear to work. I even have ideas for future work with it and
Javascript... wonder if I'll ever get the chance.

[Kevin Tierney]

Title: X-Trace: A Pervasive Network Tracing Framework
Author(s): Rodrigo Fonseca George Porter Randy H. Katz Scott Shenker Ion Stoica
Date: NSDI 07

Novelty:
X-Trace introduces a single framework that spans multiple protocols
and devices for tracing the flow of data within a system. X-Trace
reaches all the way to a client's browser (if desired), meaning it is
able to trace data across "administrative domains". X-Trcae provides a
causal tree to a network administrators attempting to diagnose faults
in their systems.

Main result: The authors implement X-Trace across a variety of
protocols and show its utility in a couple small tests. X-Trace is
effective at providing a tree describing the paths of specific packets
and requests/responses through a system.

Prior Work
X-Trace is preceded by SNMP, Cisco Netflows, and a commercial solution
called Splunk.

Question:
Should future protocols include X-Trace like capabilities natively?

Criticism:
The implementation of X-Trace within the protocols and applications on
a network seems rather cumbersome. That is, if applications do not
start coming with X-Trace (or similar) capabilities by default,
X-Trace will not be able to gain wide spread adoption.

Gauging real system administrator's reactions to such a tool would
clearly be time consuming and difficult in a variety of ways, but it
would be interesting to get more perspectives on tools like X-Trace
and others from people who deal with difficult to diagnose faults on a
day to day basis.

Ideas for further work
X-Trace logs could allow for some really interesting data mining, and
could potentially be a really good source of data for a network
intrusion detection system.

[Juexin Wang]

Paper Title  

X-Trace: A Pervasive Network Tracing Framework

 

Author(s)  

Rodrigo Fonseca, George Porter, Randy H. Katz, Scott Shenker, Ion Stoica

Date  
April, 2007

 

Novel Idea  
X-Trace metadata is inserted into a request, which is started by a app/user and propagated through layers and domians.

The metadata log the informations of the task within this path (task tree).  

Tracing can be started and got by different peers along the propagating path (task tree), each peer of them can get information to decide what to do.

 

Main Result(s)  

A  tracing framework that trace multiple applications, at different network layers, and across administrative boundaries,  especially better for complex Internet applications.

 

Impact  

Provide a tracing way that can trace through multi protocols, applications and domains.

The tracing metadata can be acquired by all peers who in the steps.

 

Evidence  

- Trace identified by task identifier.

- Design principles: the trace request should be sent in-band; The collected trace data should be sent out-of-band and that the ntity that requests tracing should be decoupled from the entity that receives the trace reports.

- X-Trace tags all network operations resulting from a particular task with the same task identifier.

- Three usage scenarios are discussed: a simple web request and an accompanying recursive DNS queries, a web hosting service that allows users to post and share photographs and the I3 overlay network. 

- Allows the "involver" to cooperate in solving problems if necessary.

 

 

Prior Work

N/A

 

Competitive work  

X-Trace is different from others for focusing on tracing multiple apps and network layers and ADs.

 

Reproducibility  

 N/A

 

Question  

if we detect a abnormal delay, how to specify what/where cost that, since the reason maybe comes from somewhere else not in the path? The X-Trace only tags network operations, resulting from a particular task, does it means it tag not only the behaviors related to the package, but also all system calls, parallelism pees and data flows?

 

Criticism  

It's said the potential benefits will encourage peers to support the X-trace, but generating a standard for each layer to add in X-trace needs considering more. And in deploying old equipments may can't be modified to fit. 
The tracing result can only show the operations on the nodes within the path, but can't tell the whole system/network's behaviors nor architecture. In a complex (complex in collaborations, not only in scale) system, this information seems give fewer help.

--
J.W
Happy to receive ur message

[yash]

Paper Title: X-Trace: Pervasive Network Tracing Framework

Authors: Rodrigo Fonseca, George Porter, Randy H. Katz, Scott Shenker
and Ion Stoica.

Novel Idea: The paper introduces X-Trace, a network tracing framework
which provides a comprehensive view of the system implementing it-
spanning several protocol layers and applications. It achieves this by
carrying X-trace meta-data in the packets and the meta-data is
propagated to the lower-layers through protocol interfaces.

Impact: Provides a very powerful debugging tool for large Internet
services consisting of distributed components across multiple
administrative domains.

Evidence: The paper describes the implementation of X-trace in a DNS
query lookup and in a photo-sharing application.

Prior Work: Traceroute along with different monitoring suites for HTTP
are considered to be prior work. They fail to give a comprehensive
view.

Reproducibility: Yes, it can be reproduced, there is a java and c++
api implementation.

Questions: The system allows only the concerned AD to view the meta-
data, can encryption be applied to this system and keep the meta-data
globally accessible?

Criticism: The meta-data contents in the packets add-up to the
overhead of the packets, so in big systems, this extra-overhead may
affect the performance.

[James Tavares]

*X-Trace*

Paper Title: X-Trace: A Pervasive Network Tracing Framework

Author(s): Rodrigo Fonseca, Geroge Porter, Randy H. Katz, Scott Shenker,
Ion Stoica

Date: NSDI �07, April 2007.

Novel Idea: X-Trace presents a framework which is able to collect traces
across both network nodes and network layers, capturing the actual path
that a particular request takes through the network via the use of
in-band trace data. Assuming no errors during data collection (which is
an out-of-band process), the precise causal path (directed graph) that
the original request took can be recovered.

Main Result(s): The paper lays out a set of design principles (trace
data in-band, report data out-of-band), discusses the format of X-Trace
messages and the operations performed at each relay point, presents
performance data (see �Evidence�), and outlines a number of typical
usage scenarios including a web request and DNS lookup, a web hosting
service, and an overlay network (I3).

Impact: In the years since its release, the X-Trace framework has found
its way into multiple projects, including Coral CDN, Hadoop, Oasis, and
Thrift.

Evidence: The authors use Wikipedia as a motivating example of a
distributed web service with many layers. A failure in a service of this
size and complexity can be difficult to diagnose. As an example, the
authors discuss how a change made to a Wikipedia page may not appear to
the user who made it due to latency in any one of the caching layers
present in Wikipedia�s architecture, and claim that a pervasive tool
such as X-Trace is necessary to pin-point errors across multiple layers,
protocols, and systems. Some performance information is provided in
section 3.4, including pushDown() / pushNext() time complexity (measured
at less than a microsecond), and overhead in an Apache implementation
(measured at 15%).

Prior/Competitive Work: The authors cite a number of relevant works,
ranging from complimentary works such as Pip to a number of projects
which use alternative trace collection methods such as inference
techniques to reconstruct causal paths. X-Trace differs from the
previous work for the reasons outlined in �Novel Idea� above.

Reproducibility: Performing research on X-Trace is super easy as X-Trace
is an open source project available for free download. Exactly
reproducing the author�s performance experiments on Apache would be
difficult without further consultation as little detail is provided.

Question: 1.) It is clear from the paper that X-Trace can be used as a
fault diagnoses tool. How could X-Trace be expanded to include
performance monitoring? 2.) The 15% performance degradation on Apache
mentioned in section 3.4 seems high for simple operations like
pushDown() and pushNext() � could the performance hit be attributable to
an inefficient logging mechanism?

Criticism: The way I understood the protocol, N layers would require N
distinct, concurrent X-Trace headers in order to capture the directed
graph information. While a good description was given of each of the
fields in the X-Trace header, a more detailed discussion of expected
message size overheads might have been useful. Then again, perhaps it is
not a concern given that its use is typically limited to the rare case
when a problem actually occurs?

Future Work:

1.) 1.) Perhaps it would be possible to introduce authentication
mechanisms into X-Trace as a means of dealing with security concerns. An
�Authenticated X-Trace� could prevent unauthorized X-trace requests,
helping to prevent usage of the X-Trace platform to propagate denial of
service attacks. Of course, the overhead imposed by an authentication
scheme, especially if it depends on public key encryption, could far
outweigh the overhead of X-Trace itself.

2.) 2.) The paper lists scoping as future work; I�m sure this has been
considered, but perhaps a simple TTL field applied to layer depth could
do the trick for dynamic scoping.

[qiao xie]

2009/11/9 qiao xie <qiao....@gmail.com>

Paper Title: X-Trace: A Pervasive Network Tracing Framework

Author: Rodrigo Fonseca

Date: 2007

[Novel Idea]

X-trace can span multiple administrative domains and network layers. X-trace constructs the task tree by injecting metadata into all actions casually related to the original task.

 

[Main Result]

Three design principles:

1. Trace request should be in-band to ensure that trace metadata follow the same datapath.

2. Collected data should be sent out of band to ensure its availability during network failure.

3. Tracing initializer should be decoupled from tracing report receiver for better control.

The propagation of the metadata has little impact on performance but reporting and processing data is challenging.

Three  usage scenarios show that we can locate the fault using the task tree generated by X-trace.

 

 

[Impact]

This framework makes locating faults in a distributed multi-layer network model much easier and more efficient. 

 

[Evidence]

The paper first addressed the need for such a diagnose tool and outlined its design principles. Then it illustrated the metadata structure and progation method followed a few implementation issues. After that there were three interesting case studies where we can see how to use X-trace to diagnose your network.

 

Reproducibility

Yes. It's an opensource project.

 

Question

How is the result from Hadoop testbed?

 

Criticism

Hard to deploy and scale to large networks (A lot of human labor is needed).

 

Ideas for further work

More experiments and a detailed guideline about how to use X-trace to diganose your network.

 

 

2009/11/9 Rodrigo <rodrigo...@gmail.com>

[Xiyang Liu]

Paper Title
X-Trace: A Pervasive Network Tracing Framework

Author(s)
Rodrigo Fonseca, George Porter, Randy H. Katz, Scott Shenker, Ion
Stoica

Date
NSDI’07, April 2007

Novel Idea
X-Trace provides both horizontal and vertical network tracing by
injecting tracing metadata into network protocols. Tracing reports are
gathered from every relevant node to construct task tree. In addition,
tracing reports can be gathered and sent per administrative domain to
guarantee security.

Main Result(s) & Impact
X-Trace was implemented for several commonly used protocols. It was
applied to scenarios involving web request, web hosting site and
overlay networks.

X-Trace provides cross-layer network tracing and implements previous
tracing tools which focuses on specific layer or protocol. It is
suitable for diagnosing networks having complex or variable paths.

Competitive work
X-Trace was compared to several other network tracing and distributed
system diagnosis tools. X-Trace differs with other network tracing
tool that it traces paths across different layers and only evaluate
involved links of the task other than the whole network
infrastructure. It differs with other distributed system diagnosis
tools such as Project 5 and Magpie that it provide accurate casual
path and tracing by adding tracing metadata to protocols.

Reproducibility
Yes. Requests can be traced having X-Trace implemented for related
protocols and installed on every node.

Question & Criticism
Is the performance of metadata propagation and reporting dependent on
how many layers are traced? For example, for a web request involving
http, tcp and ip protocols, the performance of tracing only http and
tcp should be much better than tracing all three protocols because in
the second scenario, every ip packet is traced. If so, the latency and
throughput decrease announced in the paper do not apply to all tracing
patterns.

The implementation of X-Trace for administrative domains was merely
mentioned and evaluated. What extra extension and configuration are
required to implement an automated AD handler, for example, boundary
detection, report authentication and forwarding.

On Nov 9, 6:12 pm, Rodrigo <rodrigo.fons...@gmail.com> wrote:

[小柯]

Paper Title:    X-Trace: A Pervasive Network Tracing Framework

Authors:        Rodrigo Fonseca

                    George Porter
                    Randy H. Katz
                    Scott Shenker
                    Ion Stoica

Date:            2007

Novel Idea:
    To trace the data path in a large-scale, extremely complicated system is a difficult mission. Through inserting little meta-data with a unique ID in every request, X-trace could deal with this problem elegantly. It then propagates the tracing info to ADs or end-users, giving them privacy and convenience whether they want to cooperate or keep it private.

Main Result:
    X-trace is implemented on three different systems and is proven to be workable and won't cause much

overhead.

Impact:
    Similar tools are provided by other research groups, aiming at providing more debug information, not just data path info.

Evidence:
    The author explains the meta-data format, and how it propagates between network layers. Then some implementation details are introduces. Finally, here comes the examples in real world.

Prior Work:

Competitive work:
    Pip

Reproducibility:
    Definitely. Not only just because of all details mentioned, but because the author stayed at Brown.

Question:
    In IP and TCP layer, the packet size is well defined to fit in the minimum layer-2 packet, or the IP packet might be separated into two and cause some overhead.
    I'm not sure if putting X-trace meta-data in packet would cause the size to be bigger and cause this kind of overhead.

Criticism:
  
Ideas for further work:
    X-trace seems to only focus on the data path, so it doesn't provide much other information.
    In the future, it might be extended to provide some system and debug info.

2009/11/9 Rodrigo <rodrigo...@gmail.com>

[Marcelo Martins]

Paper Title "X-Trace: A Pervasive Network Tracing Framework"

Author(s) Rodrigo Fonseca, George Porter, Randy H. Katz, Scott Shenker,
Ion Stoica

Date 4th USENIX NSDI'07, April 2007

Novel Idea

X-Trace is the first cross-layer, cross-application tracing framework
that enables a comprehensive view of the operation of a distributed system.

Main Result(s)

The usage scenarios of Section 4 proves that X-Trace fulfills its
purpose on reporting cross-layer information for distributed,
large-scale, networked applications.

Impact

Not only the work done with X-trace's core, but also the considerable
number of extensions to it (libraries, daemons, browser plug-ins,
support for legacy code) facilitates its deployment and partially
dampens its main liability (need for protocol changes). In addition,
even when X-Trace is partially installed, it is still capable of
providing useful tracing benefits.

Evidence

Fonseca et al. show how diversely X-Trace can be used to diagnose and
investigate problems in different distributed applications. They
describe usage scenarios where X-Trace was successfully deployed (Web
hosting, DNS servers, overlay network) and how much effort was necessary
to enable the tracking tool to serve its purpose.

Prior Work

Network monitoring systems is an area of research which has been
exhaustively explored. Section 7 mentions the most relevant works,
ranging from single-layer analyzers (traceroute, SNMP) to more complex
systems (Splunk, ARM, Pinpoint, Pip, Magpie, etc.)

Competitive work

Works that resemble Quanto the most are the same which were mentioned in
Pip's article. Those are Magpie, Pinpoint and Pip. Nevertheless, a
direct comparison was not provided (not that it would fit either, as the
latter do not provide cross-application support).

Reproducibility

X-trace is now an open-source Sourceforge project that can be downloaded
and enabled in one's installation. The usage scenarios are presented in
increasing difficulty and the first two can be easily replicated, given
access to the machinery. The third scenario requires more expertise with
Chord and I3, and also more modifications to the core of both protocols,
but it is also feasible.

Question

1.) After two years of existence, are there any outside stories of
success using X-Trace?

2.) Despite the preliminary performance results in Section 3.4, it bases
its estimation for a larger load on results of a single operation.
However, reality shows that scalability depends on a series of factors
that are not so easy to predict. Requests from different clients, I/O
read and write requests and other bottleneck components might impact the
performance of X-trace's collection and reporting features. The question
is, by how much?

Criticisms

1.) Worse than DoS attacks is the ability to a malicious node to perform
a man-in-the-middle attack followed by a wormhole attack (using the
optional Destination field), which can provide valuable information to
an interested hacker.

Ideas for further work

1.) One of the main problems of Quanto are the report losses. If a log
message is lost in its way to the reporting destination, the causality
tree cannot be recovered and part of the diagnosis is lost. In order to
guarantee delivery, a secure network should be considered. Other
possibilities would include replication of data, and an overlay network
that provides multiple paths to the sink node.

2.) An obvious criticism is the need of modifying protocols in order to
support X-trace-enabled data. However, if X-trace is implemented into
routing and switching hardware, at least information up to the network
layer can be automatically provided without user interference. This is
not unusual, as Cisco and Juniper's network hardware leverage
proprietary extensions to their equipments.