5g - Quochung.cyou

Part I: Introduction & The RCA Problem Space

Modern mobile networks are complex systems requiring high reliability. However, despite monitoring, faults like hardware failures or software misconfigurations occur. While detecting a fault is straightforward, the main challenge is Root Cause Analysis (RCA) — finding the root cause of the symptoms to help engineers fix them.

This project replicates the paper (Reasoning Language Models for Root Cause Analysis in 5G Wireless Networks) https://arxiv.org/pdf/2507.21974

The Complexity of 5G O&M

Traditionally, RCA relied on expert-defined logical frameworks or “fault trees”. However, these methods don’t scale well with 5G network complexity. While standard machine learning (Decision Trees, SVMs, Neural Networks) has been used, these models often lack the interpretability and reasoning needed for critical infrastructure.

Defining the Objective: RCA as Probabilistic Inference

We treat RCA as a probabilistic inference task. Formally, the goal is to identify the most probable cause from a set of potential root causes , given:

****: Network engineering parameters.
****: User plane observations.
****: Observed symptoms.

The objective is to solve for:

$\hat{c} = \arg max_{c \in C} p (c | U, Y_{t}, s_{t})$

This mathematical formulation provides a framework, but modeling these intricate dependencies in real-world data is notoriously difficult. This implementation explores how domain-adapted, reasoning-enhanced Large Language Models (LLMs) can bridge this gap by providing structured, multi-step diagnostic explanations.

The TeleLogs Framework

To benchmark these capabilities, we utilize TeleLogs, a curated dataset of network troubleshooting scenarios with expert-level annotations. TeleLogs simulates a realistic 5G environment where a User Equipment (UE) moves through a region covered by multiple Base Stations (BSs), providing full visibility into network configurations and performance drops.

Part II: Dataset Architecture & Diagnostic Parameters

To perform effective RCA, the model must synthesize three distinct data streams: configuration parameters, time-series observations, and defined symptoms.

Symptom Definition: The 600 Mbps Threshold

In this study, diagnostic scenarios are centered around a specific symptom ( $s_{t}$ ): a significant degradation in downlink throughput where the performance falls below 600 Mbps. This drop serves as the trigger for the analysis, requiring the model to identify if the cause is environmental, a misconfiguration, or a mobility issue.

Network Engineering Parameters (U)

The dataset provides a comprehensive view of the network topology through static configuration parameters. Key parameters used in our analysis include:

Parameter	Description
gNodeB ID / Cell ID	Unique identifiers for the base station and cell.
Mechanical/Digital Tilt	The vertical angle of the antenna, essential for coverage analysis.
Mechanical/Digital Azimuth	The horizontal direction of the antenna.
Beam Scenario	Specific beamforming configurations that determine vertical beamwidth.
Height & PCI	Physical height of the antenna and the Physical Cell ID.
Beam Scenario	Specific beamforming configurations that determine vertical beamwidth.

User Plane Drive Test Data (Yt)

This represents the dynamic interaction between the user and the network. The model analyzes the following time-series indicators:

Throughput (DL): The primary indicator of service quality.
RSRP & SINR: Measure the signal strength and quality of the serving cell.
Neighboring RSRP: Signal strength from the top-k neighbor cells, used to detect interference or handover opportunities.
Resource Blocks (PRBs): The amount of radio resources allocated to the user.

The Ground Truth: Root Cause Classes (C1–C8)

The implementation evaluates the model’s ability to classify symptoms into one of eight distinct categories:

C1: Excessive downtilt causing weak coverage.
C2: Over-shooting coverage (distance > 1 km).
C3: Better performance available on a neighboring cell.
C4: Interference from non-colocated co-frequency cells.
C5: PCI Mod 30 conflict causing reference signal overlap.
C6: Performance degradation due to frequent handovers.
C7: Misconfigured handover thresholds.
C8: Insufficient PRB allocation.

Part III: Baseline Evaluation

Before fine-tuning, I established a performance baseline using Qwen2.5-1.5B-Instruct.

The Benchmarking Protocol

I ran a zero-shot inference on the TeleLogs Phase 1 test dataset, which contains 864 scenarios. The model was prompted using the standard template to analyze user-plane data and site engineering parameters, then choose the most likely root cause from the eight predefined classes.

Initial Findings

The baseline results showed the model was mostly guessing based on simple patterns:

Total Accuracy: 10.53% (91/864 correct).
The C1 Bias: The model correctly identified C1 (Excessive Downtilt) 42.59% of the time but used it as a “catch-all” for errors. In the top 10 most frequent errors, the model predicted C1 for other classes (C6, C8, C7, C2, C3, C4, and C5) a total of 312 times.
Instruction Following: While the model often produced the correct format (the boxed answer), it lacked the underlying logic to connect the RSRP/SINR drops to specific mobility or interference issues.
Placeholders: For many classes (C2, C4, C5, C8), the accuracy hovered near 3%, indicating that the base model could not differentiate between various types of signal degradation.

analysis report baseline - quochung.cyou PTIT

Part IV: Synthetic Data Generation Phase 1 — Reasoning Traces

To improve reasoning, I started the first phase of synthetic data generation. The goal was to create a dataset that demonstrates the Chain-of-Thought (CoT) required for RCA.

The Teacher Model: Qwen3-32B

I utilized Qwen3-32B as the high-reasoning agent. Its significantly larger parameter count and inherent reasoning capabilities allowed it to act as the “expert engineer”. Unlike the 1.5B model, the 32B model can synthesize the relationship between (Engineering Parameters) and (User Observations) more effectively.

CoT Prompting Strategy

I implemented a customized prompting strategy designed to guide the model through a structured diagnostic trajectory. The prompt forced the model to:

Analyze Data: Explicitly list the throughput drops and serving cell changes.
Eliminate Unlikely Causes: Systematically rule out causes—for example, ruling out C2 (distance) if the serving cell is far away.
Validate via Reflection: Check for specific conflicts, such as the PCI Mod 30 check for interference.

Data Acquisition

Through this multi-agent pipeline, I harvested an initial set of 138 samples. Each sample consisted of the original network log paired with a detailed reasoning trace leading to the correct ground-truth answer. This dataset was intended to teach the 1.5B model how to think, rather than just what to predict.

Part V: Experiment 1 The First LoRA Attempt

With 138 high-quality reasoning traces, I initiated the first fine-tuning stage. The goal was to align the Qwen2.5-1.5B-Instruct model with the structured diagnostic patterns found in the synthetic dataset. Following the general recommendations for domain adaptation in the provided LoRA research , I established a baseline configuration to mirror the paper’s parameters where possible.

Configuration and Training Setup

LoRA Hyperparameters: r = 32, alpha = 32, dropout = 0.1
Learning Rate: 1e-6
Epochs: 10.
Targeting: Applied LoRA across all linear layers (Query, Key, Value, Projection, and MLP) to maximize the model’s capacity to absorb the new domain knowledge.

The Regression Reality: 8.91% Accuracy

The results were a stark contrast to expectations. Instead of improving upon the baseline, the model’s performance dropped to 8.91% (77/864 correct).

Placeholder Issues: A significant issue was the increase in “placeholder” predictions. In the top 10 error categories, 7 were instances where the model outputted a placeholder rather than a valid root cause.
Reduced Instruction Following: The model lost some ability to follow basic formatting instructions. Focusing on the reasoning traces caused the model to struggle with the required \boxed{} format.
Word Count Increase: The mean word count increased from 245 to 651 words, often containing circular logic without a valid conclusion.

Hypothesis: Sample Sparsity and Overfitting

My hypothesis for this failure was sample sparsity. 138 samples are likely insufficient for a 1.5B model to learn both the complex 5G domain rules and the specific “Elimination-based” or “Contradiction-based” prompting strategies used in the dataset. The model likely overfitted to the specific noise of those 138 examples rather than learning the underlying causal relationships.

Part VI: Experiment 2 Scaling Sample Size

To address the sparsity issue identified in Experiment 1, I moved to scale the training data. If 138 samples caused overfitting, perhaps a larger, more diverse dataset would force the model to generalize the reasoning steps.

The 300-Sample Expansion

I re-ran the synthetic generation pipeline with the Qwen3-32B teacher model to reach 300 samples. I also ensured a more balanced distribution across the 8 root cause classes to prevent the model from defaulting to “C1” as it did in the baseline.

Results: Marginal Recovery (9.61%)

Training the 1.5B model on 300 samples yielded a slight recovery but remained below the zero-shot baseline:

Total Accuracy: 9.61% (83/864).
Improved Class Recognition: Accuracy for C2 (18.52%) and C3 (19.44%) saw noticeable jumps compared to the near-zero baseline performance.
Persistent Failures: The “Placeholder” issue remained rampant. Truth C3 and C6 both saw 28 placeholder errors.
Word Count Increase: The mean word count increased to 1318 words, with some responses reaching 12,023 tokens.

Thinking: The Depth vs. Clarity Trade-off

This experiment showed that raw reasoning traces led to verbosity. Without synthesizing these traces, the model produced overly long responses.

Part VII: Synthetic Data Generation Phase 2

Experiment 2 showed that more raw reasoning traces were insufficient. The model mimicked the teacher’s verbosity without learning the logic. To fix this, I updated the synthetic data generation strategy.

The Theoretical Shift: From Traces to Correction

Instead of just saving any correct answer, I implemented a “Reflective Teacher” loop using Qwen3-32B. This approach mirrors the “Aggregator” concept in the paper, designed to synthesize concise, structured explanations.

The Reflection Loop

I introduced a two-tier evaluation for every synthetic sample:

Initial Attempt: The teacher model performs RCA on the raw data.
The “Correction” Prompt: * If Correct: The model is instructed to condense the reasoning into a specific “RCA Reasoning Format” (Data Analysis → Root Cause Analysis → Identification), without give it answer to prevent leakage.

If Incorrect: I forced a Self-Reflection phase. The system prompt would tell the model its assumption was wrong and demand it identify exactly where the logic failed (e.g., “I ignored the PCI Mod 30 check”). This reflection was then added back to the prompt to generate a “perfected” reasoning trace.

Reasoning Optimization

This new pipeline produced 300 samples of refined data. The reasoning traces were focused on key steps, such as checking the distance for C2 or the PRB count for C8.

Part VIII: Experiment 3 Training on Error-Correction Traces

With the 300 “reflected” samples, I reset the training. This experiment was the first true test of whether teaching a model how it failed was more effective than just showing it how to succeed.

Training on this densified dataset finally yielded an increase in accuracy:

Total Accuracy: 12.04% (104/864). While modest, this officially surpassed the zero-shot baseline (10.53%).
Mean Word Count: A massive drop from 1318 to 33.5 words. The model stopped hallucinating long, circular paths and started focusing on the final answer.

Class-Specific Breakthrough: The C1 Dominance

C1 Accuracy: 73.15% (79/108). This was a significant improvement.
Thinking: By learning the rules for C1, the model became proficient at identifying coverage-related drops.
Over-correction: However, the model predicted C1 for almost every other class (e.g., Truth C8 → Pred C1: 85 times).

Analysis: The Learning Capacity Gap

Experiment 3 proved that Data Quality > Data Quantity. However, it also revealed that at a low rank (r=32), the 1.5B model was struggling to hold more than one complex rule at a time. It had “mastered” C1 but was overriding its knowledge of other causes to do so. This set the stage for our exploration into LoRA hyperparameters.

analysis report 104 - quochung.cyou PTIT

Part IX: Theoretical Intermission LoRA Hyperparameter Optimization

The jump to 12% accuracy in Experiment 3 was a proof-of-concept for my self-reflection dataset, but the heavy “C1 bias” suggested the model’s update capacity was saturated.

Raschka’s Principles & The Alpha Heuristic

The Alpha Scaling: A common rule of thumb is setting alpha (the scaling factor) to twice the value of r (the rank), effectively alpha = 2 * r. This ensures the influence of the LoRA weights is balanced against the pre-trained weights.
Layer Coverage: To maximize performance, LoRA should be applied across all layers, including projection and MLP layers, not just the Key and Value matrices. This increases the number of trainable parameters, which is vital for domain-specific tasks like 5G RCA where the model must learn entirely new technical correlations.

The Capacity Problem: Rank vs. Knowledge

In a 1.5B model, a rank of 32 only updates a small fraction of parameters. This might not provide enough capacity to store 8 distinct 5G diagnostic rules. If the rank is too small, the model may only capture dominant patterns like C1 (Downtilt) and miss others.

Part X: Experiment 4 Expanding the Rank (r=128)

Armed with the theory that our model lacked the “memory” to differentiate between classes, I pushed the rank significantly higher.

The Setup: Boosting Rank and Alpha

LoRA Config: r = 128, a = 64
Methodology: I maintained the 300 “reflected” samples from the previous phase but allowed the model more degrees of freedom to store the learned weights.

Accuracy: 16.32%

analysis report 141 - quochung.cyou PTIT

Expanding the rank improved performance:

Total Accuracy: 16.32% (141/864).
Diversification: The C1 bias decreased. The model’s recognition of C4 (Neighbor interference) jumped to 29.63%, and C7 (Handover thresholds) reached 21.30%.
Word Count Consistency: The mean word count remained stable at 960.63 words, indicating that higher rank did not necessarily mean more rambling, but rather more precise reasoning.

Analysis: Reducing Categorical Confusion

In the error logs, we saw a shift. Instead of predicting C1 for everything, the model began confusing similar signal-based issues. For example, Truth C3 (Neighbor throughput) was often confused with C4 (Neighbor interference). This shows progress — the model now understands the problem involves “Neighboring Cells” but is still fine-tuning the specific logic that distinguishes interference from a handover opportunity.

Part XI: Scaling the Dataset 2000 Samples

Recognizing that the 1.5B model’s improvement was driven by better data and increased rank, I increased the training set size. However, scaling synthetic data requires a strict “Teacher-Student” hierarchy to prevent errors.

The “Major Voting” Strategy

I utilized a Qwen2.5-7B-Instruct model, which had better baseline performance, to act as a secondary filter.

The Process: For each question, the 7B model generated multiple reasoning trajectories.
The Filter: Only samples where the 7B model reached the correct answer via majority voting were added.
The Result: This method produced a dataset of 2000 high-quality reasoning traces.

Response-Only Training (SFT Optimization)

To further improve efficiency and focus, I pivoted to training on responses only.

The Goal: By masking the loss for the system and user prompts, the model focuses its entire learning capacity on the reasoning steps and the final \boxed{} identification.
Alignment: This technique prevents the model from wasting parameter updates on memorizing the structure of the input logs and engineering tables, ensuring it prioritizes the causal logic instead.

Part XII: Experiment 5 High-Rank Performance (r=256)

The final phase of my implementation involved pushing the LoRA rank to the maximum sustainable level for a 1.5B model while utilizing the massive 2000-sample dataset. This experiment aimed to replicate the “Reasoning LLM” performance gains described in the paper.

Final Configuration

LoRA Hyperparameters: Rank 256, Alpha 128.
Data Density: 2000 “major-voted” samples from the reflection pipeline.
Optimization: AdamW optimizer with a cosine learning rate scheduler, applied to all linear layers as suggested by Raschka’s research.

Final Result: 21.41% Accuracy

This iteration achieved the highest performance:

Total Accuracy: 21.41% (185/864 correct).
Balanced Learning: The “C1 bias” was significantly mitigated. Accuracy for C7 (Handover Thresholds) reached 37.96%, and C8 (PRB Allocation) surged to 23.15%.
PCI Mod 30 Success: The model finally began correctly identifying C6 (PCI Conflict) at a 29.63% rate, proving it had successfully encoded the mathematical relationship between PCI values and reference signal overlap.

Analysis: The Impact of Scale

The jump from 16.32% to 21.41% demonstrates that for a 1.5B model, the combination of dataset density and rank capacity is significant. By providing 2000 examples, the model found common causal themes across different logs, resulting in better generalization rather than rote memorization.

analysis report 185 - quochung.cyou PTIT

—