Skip to content

CI/CD: Cleaning Test Retry Output

CI pipelines often retry flaky tests multiple times, filling logs with duplicate failure messages. Remove duplicate test failures to see unique issues clearly.

The Problem

Modern CI systems retry failed tests to handle transient failures. This creates verbose logs where the same test failure appears 3-5 times:

  • Obscures real issues - Hard to see unique failures among retries
  • Slows log review - Developers scroll past duplicate failures
  • Wastes storage - Identical error traces repeated multiple times

Input Data

ci-output.log 
Running test suite...
✗ test_authentication [attempt 1/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 1/3]
  AssertionError: Expected 200, got 404
  File "test_api.py", line 142
Retrying failed tests...
✗ test_authentication [attempt 2/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 2/3]
  AssertionError: Expected 200, got 404
  File "test_api.py", line 142
Retrying failed tests...
✗ test_authentication [attempt 3/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 3/3]
  AssertionError: Expected 200, got 404
  File "test_api.py", line 142
All retries exhausted. 2 tests failed.

The test suite retries 2 failed tests 3 times each, producing:

  • test_authentication failure (lines 2-4, 9-11, 15-17) - appears 3×
  • test_user_profile failure (lines 5-7, 12-14, 18-20) - appears 3×

Output Data

expected-output.log 
Running test suite...
✗ test_authentication [attempt 1/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 1/3]
  AssertionError: Expected 200, got 404
  File "test_api.py", line 142
Retrying failed tests...
✗ test_authentication [attempt 2/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 2/3]
✗ test_authentication [attempt 3/3]
  ConnectionError: Database unavailable
  File "test_auth.py", line 89
✗ test_user_profile [attempt 3/3]
  AssertionError: Expected 200, got 404
  File "test_api.py", line 142
All retries exhausted. 2 tests failed.

Result: Duplicate retry attempts removed → see each unique failure once

Solution

uniqseq ci-output.log \ --window-size 3 \ --quiet > output.log

Options:

  • --window-size 3: Match 3-line test failure patterns
  • --quiet: Suppress statistics

from uniqseq import UniqSeq

uniqseq = UniqSeq(
    window_size=3,  # (1)!
)

with open("ci-output.log") as f:
    with open("output.log", "w") as out:
        for line in f:
            uniqseq.process_line(line.rstrip("\n"), out)
        uniqseq.flush_to_stream(out)

  1. Match 3-line test failure sequences

How It Works

Each test failure has a consistent 3-line format:

✗ test_name [attempt N/3]
  Error message
  File location

With --window-size 3, uniqseq detects when this pattern repeats (even with different attempt numbers) and keeps only the first occurrence.

Impact Analysis

Before: 

$ grep "^✗" ci-output.log | wc -l
6    # 6 test failures (3 retries × 2 unique failures)

After: 

$ grep "^✗" output.log | wc -l
2    # 2 unique failures

67% reduction in log size from removing retry duplicates.

Real-World Workflow

See What Was Deduplicated

Use --annotate to show where retries were removed:

uniqseq ci-output.log --window-size 3 --annotate

Output includes markers like: 

[DUPLICATE: Lines 9-11 matched lines 2-4 (sequence seen 2 times)]

Track Flaky Tests

Use --inverse to see ONLY the retry attempts (what was deduplicated):

uniqseq ci-output.log --window-size 3 --inverse > retries-only.log

This shows which tests failed multiple times - potential flaky tests to investigate.

Statistics Output

See deduplication metrics:

uniqseq ci-output.log --window-size 3

Shows: 

┌─────────────────────────────┬──────────┐
│ Total Records Processed     │ 21       │
│ Unique Records (kept)       │ 15       │
│ Duplicate Records (skipped) │ 6        │
│ Reduction                   │ 28.6%    │
└─────────────────────────────┴──────────┘

See Also