Debugging of Performance Degradation in Distributed Requests Handling Using Multilevel Trace Analysis

Abstract

Root cause identification of performance degradation within distributed systems is often a difficult and time-consuming task, yet it is crucial for maintaining high performance. In this paper, we present an execution trace-driven solution that reduces the efforts required to investigate, debug, and solve performance problems found in multinode distributed systems. The proposed approach employs a unified analysis method to represent trace data collected from the user-space level to the hardware level of involved nodes, allowing for efficient and effective root cause analysis. This solution works by extracting performance metrics and state information from trace data collected at user-space, kernel, and network levels. The multisource trace data is then synchronized and structured in a multidimensional data store, which is designed specifically for this kind of data. A posteriori analysis using a top-down approach is then used to investigate performance problems and detect their root causes. In this paper, we apply this generic framework to analyze trace data collected from the execution of the web server, database server, and application servers in a distributed LAMP (Linux, Apache, MySQL, and PHP) Stack. Using industrial level use cases, we show that the proposed framework can investigate root causes, address unusual latency, and improve base latency by 70%. This is achieved with minimal tracing overhead that does not significantly impact performance, together with O(log n) query response times for efficient analysis.

Source: Official Wiley article abstract; Crossref records CC BY 4.0 for the version of record.

Problem and motivation

Performance degradation in distributed requests can arise at user, kernel, storage, network, or remote-service layers, so single-layer tracing does not provide enough causal context.

Method and contribution

Instrument LAMP components and system layers with LTTng, synchronize hosts using causal network event pairs and convex-hull matching, maintain multilevel state trees, and perform top-down diagnosis from request latency to system call and kernel/storage/network states.

Findings and evidence

The case study localized application-level session/statistics work and disk queue contention from backup activity. The system distinguished cache-hit/miss latency classes and showed that line-level PHP tracing can approximately double response time, while selected UST tracing can approach the no-trace baseline.

Limitations and future directions

Limitations: The evaluation focuses on LAMP and selected instrumented cases; causal synchronization requires suitable event pairs and the available probes determine what can be explained. Network attacks, broader systems, and richer anomaly classes are not evaluated.

Future work: Add network-attack and host-anomaly analysis, broaden probes and workloads, and improve multilevel visualization and causal coverage.

Resources

• Code

Sources and identifiers

• Published version published
• Public Wiley PDF · PDF public_full_text

When to cite this paper

Cite this paper when your work uses or compares top-down multilevel root-cause analysis from distributed request latency to kernel/storage/network state.

• For top-down multilevel root-cause analysis from distributed request latency to kernel/storage/network state.
• For LTTng instrumentation spanning LAMP user space, system calls, kernel, filesystem, page cache, block layer, disk, and network.
• For cross-host causal synchronization using request/response event pairs and convex-hull matching.
• For the MediaWiki/WordPress session-check and `backup.sh` disk-queue case studies, including the tracing-cost boundary.

Citation

@article{ezzatiJivan2021debuggingof,
  author = {Naser Ezzati-Jivan and Houssem Daoud and Michel R. Dagenais},
  title = {Debugging of Performance Degradation in Distributed Requests Handling Using Multilevel Trace Analysis},
  year = {2021},
  journal = {Wireless Communications and Mobile Computing},
  volume = {2021},
  number = {1},
  eid = {8478076},
  publisher = {Wiley},
  issn = {1530-8669, 1530-8677},
  doi = {10.1155/2021/8478076},
  url = {https://doi.org/10.1155/2021/8478076}
}

Ezzati-Jivan, N., Daoud, H., & Dagenais, M. R. (2021). Debugging of Performance Degradation in Distributed Requests Handling Using Multilevel Trace Analysis. Wireless Communications and Mobile Computing, 2021(1), 8478076. https://doi.org/10.1155/2021/8478076

N. Ezzati-Jivan, H. Daoud, and M. R. Dagenais, "Debugging of Performance Degradation in Distributed Requests Handling Using Multilevel Trace Analysis," Wireless Communications and Mobile Computing, vol. 2021, no. 1, 2021, doi: 10.1155/2021/8478076