Test Quality Analysis

Debtmap provides comprehensive analysis of test code quality, detecting anti-patterns, identifying potentially flaky tests, and providing actionable recommendations for improvement.

Overview

Test quality analysis examines your test suite to identify:

• Assertion patterns - Missing or weak assertions that reduce test effectiveness
• Test complexity - Overly complex tests that are hard to maintain
• Flaky test patterns - Tests that may fail intermittently
• Framework detection - Automatic detection of testing frameworks
• Test type classification - Categorization of tests by type (unit, integration, property, benchmark)

The analysis pipeline uses multiple detectors defined in src/testing/mod.rs:111-115:

#![allow(unused)]
fn main() {
let detectors: Vec<Box<dyn TestingDetector>> = vec![
    Box::new(assertion_detector::AssertionDetector::new()),
    Box::new(complexity_detector::TestComplexityDetector::new()),
    Box::new(flaky_detector::FlakyTestDetector::new()),
];
}

Test Type Classification

Tests are classified into distinct types based on attributes, file paths, and naming patterns. The classification is handled by TestClassifier in src/testing/rust/test_classifier.rs.

Test Types

From src/testing/rust/mod.rs:69-76:

Classification Logic

The classifier checks in order (src/testing/rust/test_classifier.rs:22-44):

1. Benchmark detection: Functions with #[bench] attribute
2. Property test detection: Functions using proptest or quickcheck
3. Integration test path: Files in tests/ directory
4. Default: Unit test

#![allow(unused)]
fn main() {
// Example: Integration test detection
fn is_integration_test_path(&self, path: &Path) -> bool {
    let path_str = path.to_string_lossy();
    path_str.contains("/tests/") || path_str.starts_with("tests/")
}
}

Assertion Pattern Detection

The AssertionDetector (src/testing/assertion_detector.rs) identifies tests with missing or inadequate assertions.

Detected Assertion Types

From src/testing/rust/mod.rs:79-95:

Assertion Macro Recognition

The detector recognizes these macros (src/testing/assertion_detector.rs:227-237):

#![allow(unused)]
fn main() {
fn is_assertion_macro(name: &str) -> bool {
    matches!(
        name,
        "assert" | "assert_eq" | "assert_ne" | "assert_matches"
            | "debug_assert" | "debug_assert_eq" | "debug_assert_ne"
    )
}
}

Tests Without Assertions

Tests flagged as having no assertions receive suggested fixes:

#![allow(unused)]
fn main() {
// From src/testing/assertion_detector.rs:257-278
fn suggest_assertions(analysis: &TestStructureAnalysis) -> Vec<String> {
    if analysis.has_action && !analysis.has_assertions {
        vec![
            "Add assertions to verify the behavior".to_string(),
            "Consider using assert!, assert_eq!, or assert_ne!".to_string(),
        ]
    }
    // ...
}
}

Test Complexity Analysis

The TestComplexityDetector (src/testing/complexity_detector.rs) measures test complexity and suggests simplifications.

Complexity Sources

From src/testing/mod.rs:39-46:

Complexity Scoring

The complexity score is calculated in src/testing/complexity_detector.rs:304-309:

#![allow(unused)]
fn main() {
pub(crate) fn calculate_total_complexity(analysis: &TestComplexityAnalysis) -> u32 {
    analysis.cyclomatic_complexity
        + (analysis.mock_setup_count as u32 * 2)
        + analysis.assertion_complexity
        + (analysis.line_count as u32 / 10)  // Penalty for long tests
}
}

The Rust-specific complexity scoring (src/testing/rust/mod.rs:24-29 in doc comments):

• Conditional statements: +2 per if/match
• Loops: +3 per loop
• Assertions beyond 5: +1 per additional assertion
• Nesting depth > 2: +2 per level
• Tests > 30 lines: +(lines-30)/10

Simplification Recommendations

From src/testing/mod.rs:48-55:

The suggestion logic (src/testing/complexity_detector.rs:320-334):

#![allow(unused)]
fn main() {
pub(crate) fn suggest_simplification(
    analysis: &TestComplexityAnalysis,
    detector: &TestComplexityDetector,
) -> TestSimplification {
    match () {
        _ if analysis.mock_setup_count > detector.max_mock_setups => {
            TestSimplification::ReduceMocking
        }
        _ if analysis.line_count > detector.max_test_length => {
            classify_length_based_simplification(analysis)
        }
        _ if analysis.cyclomatic_complexity > 5 => TestSimplification::ParameterizeTest,
        _ => TestSimplification::SimplifySetup,
    }
}
}

Flaky Test Detection

The FlakyTestDetector (src/testing/flaky_detector.rs) identifies patterns that can cause intermittent test failures.

Flakiness Types

From src/testing/mod.rs:57-65:

Rust-Specific Flakiness Types

From src/testing/rust/mod.rs:98-107:

Reliability Impact Levels

From src/testing/mod.rs:67-73:

• Critical: External dependencies, network calls - high failure probability
• High: Timing dependencies, threading issues - moderate failure probability
• Medium: Random values, filesystem operations - occasional failures
• Low: Minor ordering issues - rare failures

Detection Patterns

The detector uses pattern categories defined in src/testing/flaky_detector.rs:190-284:

Timing Patterns (TimingDependency):

sleep, Instant::now, SystemTime::now, Duration::from, delay,
timeout, wait_for, park_timeout, recv_timeout

Random Patterns (RandomValues):

rand, random, thread_rng, StdRng, SmallRng, gen_range,
sample, shuffle, choose

External Service Patterns (ExternalDependency):

reqwest, hyper, http, Client::new, HttpClient, ApiClient,
database, db, postgres, mysql, redis, mongodb, sqlx, diesel

Filesystem Patterns (FilesystemDependency):

fs::, File::, std::fs, tokio::fs, async_std::fs,
read_to_string, write, create, remove_file, remove_dir

Network Patterns (NetworkDependency):

TcpStream, TcpListener, UdpSocket, connect, bind,
listen, accept, send_to, recv_from

Stabilization Suggestions

Each flaky pattern includes a stabilization suggestion:

#![allow(unused)]
fn main() {
// From src/testing/flaky_detector.rs:156-187
_ if is_timing_function(path_str) => Some(FlakinessIndicator {
    flakiness_type: FlakinessType::TimingDependency,
    impact: ReliabilityImpact::High,
    suggestion: "Replace sleep/timing dependencies with deterministic waits or mocks"
        .to_string(),
}),
_ if is_external_service_call(path_str) => Some(FlakinessIndicator {
    flakiness_type: FlakinessType::ExternalDependency,
    impact: ReliabilityImpact::Critical,
    suggestion: "Mock external service calls for unit tests".to_string(),
}),
}

Timing Classification

The timing classifier (src/testing/timing_classifier.rs) categorizes timing-related operations to assess flakiness risk.

Timing Categories

From src/testing/timing_classifier.rs:31-44:

Only DurationCreation and Unknown are considered non-flaky.

Test Quality Issue Types

The Rust-specific module tracks comprehensive issue types (src/testing/rust/mod.rs:131-140):

Severity Levels

From src/testing/rust/mod.rs:110-116:

• Critical: Fundamental test quality issues
• High: Significant problems affecting reliability
• Medium: Quality concerns worth addressing
• Low: Minor improvements possible

Framework Detection

Debtmap automatically detects testing frameworks to provide context-aware analysis.

Supported Frameworks

From src/testing/rust/mod.rs:54-66:

Multi-Language Support

From src/testing/mod.rs:89-106:

#![allow(unused)]
fn main() {
// Test attribute detection
path_str == "test"
    || path_str == "tokio::test"
    || path_str == "async_std::test"
    || path_str == "bench"
    || path_str.ends_with("::test")
}

The documentation also covers:

• Rust: #[test], #[tokio::test], proptest, rstest, criterion
• Python: pytest, unittest
• JavaScript: jest, mocha

Fast vs Slow Test Detection

Slow tests are identified as a quality issue (src/testing/rust/mod.rs:139):

#![allow(unused)]
fn main() {
RustTestIssueType::SlowTest
}

Detection criteria include:

• Long test functions (> 50 lines by default)
• Timing operations that suggest waiting
• External service calls that may have latency

Configuration

Basic Configuration

[test_quality]
enabled = true
complexity_threshold = 10

Advanced Configuration

From src/testing/complexity_detector.rs:9-13:

[test_quality]
enabled = true

# Maximum allowed test complexity score
complexity_threshold = 10

# Maximum number of mock setups per test
max_mock_setups = 5

# Maximum test function length (lines)
max_test_length = 50

Default values from TestComplexityDetector::new():

• max_test_complexity: 10
• max_mock_setups: 5
• max_test_length: 50

Common Issues and Solutions

Issue: Test Without Assertions

Detection: TestWithoutAssertions anti-pattern

Example of problematic code:

#![allow(unused)]
fn main() {
#[test]
fn test_without_assertion() {
    let result = calculate(10);
    // No assertion!
}
}

Fix:

#![allow(unused)]
fn main() {
#[test]
fn test_with_assertion() {
    let result = calculate(10);
    assert_eq!(result, 20);
}
}

Issue: Timing-Dependent Test

Detection: FlakinessType::TimingDependency

Example of problematic code:

#![allow(unused)]
fn main() {
#[test]
fn test_timing_dependent() {
    let start = Instant::now();
    do_work();
    assert!(start.elapsed() < Duration::from_millis(100));
}
}

Fix:

#![allow(unused)]
fn main() {
#[test]
fn test_deterministic() {
    let result = do_work();
    assert!(result.is_success());
}
}

Issue: Excessive Mocking

Detection: ComplexitySource::ExcessiveMocking

Solution: Consider using real implementations, test doubles, or restructuring to reduce mock count below 5.

Issue: Tests with Loops

Detection: ComplexitySource::LoopInTest

Solution: Use parameterized tests with #[rstest] or property-based testing with proptest instead of loops.

See Also

• Coverage Integration - Combining coverage with test quality analysis
• Complexity Metrics - Understanding complexity scoring
• Configuration - Full configuration reference