DTraComp: Comparing distributed execution traces for understanding intermittent latency sources

Problem and motivation

Distributed tracing shows where request latency appears in a service graph, but it usually cannot explain whether the delay arose from CPU contention, blocking, I/O, interrupts, or other kernel activity on the participating machines. Existing tools also make it difficult to compare two groups of similar requests automatically, especially when spans execute in parallel across multiple services and hosts.

Method and contribution

The system uses double instrumentation: HotROD is instrumented with OpenTracing for high-level spans, and the OpenTracing source is instrumented with LTTng-UST while LTTng also collects kernel events. Span events carry start/end times, span and parent identifiers, distributed trace IDs, process IDs, and thread IDs; separate UST and kernel channels share a unified clock. Eclipse Trace Compass synchronizes traces from multiple machines using its packet-based clock synchronization. Users filter requests by duration and self-time, select comparison groups, build per-request and aggregated flame graphs, and compute an Enhanced Differential Flame Graph using the ratio (tB - tA) / tA. The Metric Extractor attributes each span to User, System Call, Waited Blocked, Waited CPU, Interrupted, and Unknown states and reports durations for system calls such as read, write, futex, sched, openat, mmap, and networking calls.

Findings and evidence

The paper evaluates five open-source applications and performance cases: Eclipse Theia, HotROD, TiDB, JFreeChart, and Apache Cassandra. It reports span-attribution differences below 0.1% against Jaeger for the checked examples. For HotROD with 10,000 requests, 10 client threads, and about 51 spans per request, tracing with the selected kernel subset adds less than 9% overhead in snapshot mode and about 10% in standard mode; adding system-call collection raises the reported overhead to below 26% and about 27%, respectively. Metric extraction is reported to scale linearly with trace size, while the differential flame-graph implementation processes traces up to 1.7 GB within seconds and the paper reports filtering/comparison experiments up to 20 GB. The case studies expose TiDB lock contention, CPU contention, Cassandra read-repair resource cost, Eclipse Theia patch effects, and a distributed deadlock. In an exploratory study with eight engineers, localization accuracy was 70.8%, root-cause accuracy was 75%, and median task time was 35 seconds.

Limitations and future directions

Limitations: The evaluation combines five selected applications and structured, pre-captured diagnosis tasks rather than a controlled comparison against alternative tools during live incidents. The user study has only eight participants, 62.5% had prior familiarity with DTraComp, and no inferential tests or controlled baseline condition were used. Clock synchronization, hardware, software versions, network conditions, and manually reproduced application cases can affect the results. The paper also notes that some module evaluations lack an established ground truth, so the measurements support feasibility and diagnostic usefulness rather than universal superiority.

Future work: The paper proposes richer filtering and group-selection controls, anomaly detection for abnormal traces, and automatic grouping or categorization of performance problems using kernel metrics and system-state durations.

Resources

• Code
• Data
• Demo

Sources and identifiers

• Published version published

When to cite this paper

Cite DTraComp when comparing distributed request groups and connecting span-level latency differences to cross-machine kernel and system-call causes.

• Double instrumentation with OpenTracing, LTTng-UST, and LTTng kernel events.
• Eclipse Trace Compass synchronization and span-to-thread/system-state attribution.
• Enhanced Differential Flame Graph comparison of request groups with parallel spans.
• Five application cases, trace-collection overhead, linear scaling, and the eight-engineer exploratory study.

Citation

@article{ezzatiJivan2026dtracompcomparing,
  author = {Maryam Ekhlasi and Fatemeh Faraji Daneshgar and Michel Dagenais and Maxime Lamothe and Naser Ezzati-Jivan and Matthew Khouzam},
  title = {DTraComp: Comparing distributed execution traces for understanding intermittent latency sources},
  year = {2026},
  journal = {Journal of Systems and Software},
  volume = {241},
  pages = {112990},
  eid = {112990},
  publisher = {Elsevier BV},
  issn = {0164-1212},
  doi = {10.1016/j.jss.2026.112990},
  url = {https://doi.org/10.1016/j.jss.2026.112990}
}

Ekhlasi, M., Daneshgar, F. F., Dagenais, M., Lamothe, M., Ezzati-Jivan, N., & Khouzam, M. (2026). DTraComp: Comparing distributed execution traces for understanding intermittent latency sources. Journal of Systems and Software, 241, 112990. https://doi.org/10.1016/j.jss.2026.112990

M. Ekhlasi, F. F. Daneshgar, M. Dagenais, M. Lamothe, N. Ezzati-Jivan, and M. Khouzam, "DTraComp: Comparing distributed execution traces for understanding intermittent latency sources," Journal of Systems and Software, vol. 241, Art. no. 112990, 2026, doi: 10.1016/j.jss.2026.112990