Terminal Sessions: Cleaning Script Output

Remove repeated command sequences from terminal session logs to create clean, concise documentation.

The Problem

When demonstrating software workflows, you often run the same commands multiple times during development:

Testing commands - Run same command repeatedly while debugging
Showing retries - Demonstrate retry behavior clutters output
Recording sessions - Terminal session captures include duplicate commands
Creating docs - Need clean examples without repetition

Input Data

verbose-session.log

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying sessions.0001_initial... OK

$ python manage.py test
Creating test database...
Running tests...
..........................
OK (26 tests)
Destroying test database...

$ python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

$ curl http://127.0.0.1:8000/api/users
[{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying sessions.0001_initial... OK

$ python manage.py test
Creating test database...
Running tests...
..........................
OK (26 tests)
Destroying test database...

$ exit

A terminal session where migrate and test commands were run twice:

Lines 1-8: First python manage.py migrate (5 lines)
Lines 24-31: Duplicate python manage.py migrate (5 lines)
Lines 10-16: First python manage.py test (5 lines)
Lines 32-38: Duplicate python manage.py test (5 lines)

Output Data

expected-clean-session.log

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying sessions.0001_initial... OK

$ python manage.py test
Creating test database...
Running tests...
..........................
OK (26 tests)
Destroying test database...

$ python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

$ curl http://127.0.0.1:8000/api/users
[{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]

$ exit

Result: Duplicate command sequences removed → clean session log

Solution

CLIPython

uniqseq verbose-session.log \ --window-size 5 \ --quiet > output.log

Options:

--window-size 5: Match 5-line command sequences
--quiet: Suppress statistics

from uniqseq import UniqSeq

uniqseq = UniqSeq(
    window_size=5,  # (1)!
)

with open("verbose-session.log") as f:
    with open("output.log", "w") as out:
        for line in f:
            uniqseq.process_line(line.rstrip("\n"), out)
        uniqseq.flush_to_stream(out)

Match 5-line sequences (typical command + output length)

How It Works

Terminal commands typically produce multi-line output. By setting --window-size 5, uniqseq detects when a 5-line pattern (command + output) repeats and removes the duplicate.

Choosing Window Size

The window size should match your command's output length:

# Short commands (1-2 lines output): window-size 3
uniqseq session.log --window-size 3

# Medium commands (3-5 lines output): window-size 5
uniqseq session.log --window-size 5

# Long commands (10+ lines output): window-size 10
uniqseq session.log --window-size 10

Real-World Workflows

Create README Examples

Clean up terminal sessions for documentation:

# Record session with script/asciinema
script session.log

# ... run commands, some repeated during testing ...

# Clean for documentation
uniqseq session.log --window-size 5 > clean-session.log

Show What Was Removed

Use --annotate to mark where duplicates were removed:

uniqseq session.log --window-size 5 --annotate > annotated.log

Output includes markers:

[DUPLICATE: Lines 24-31 matched lines 1-8 (sequence seen 2 times)]

Interactive Debugging Cleanup

Remove test command repetitions while debugging:

# You ran: python test.py
# Output: 20 lines of test results
# You ran it 5 times debugging

# Clean to show just once
uniqseq debug-session.log --window-size 20 > clean-debug.log

Aggregate Multiple Sessions

Combine multiple terminal sessions and remove cross-session duplicates:

cat session-01.log session-02.log session-03.log | \
    uniqseq --window-size 5 > combined-clean.log

Tips

Variable Output: If command output varies slightly (timestamps, IDs), use --skip-chars or --hash-transform:

# Skip timestamp prefix in each line
uniqseq session.log --window-size 5 --skip-chars 20

# Normalize timestamps before comparison
uniqseq session.log --window-size 5 \
    --hash-transform "sed 's/[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}//g'"

Preserve Important Repetitions: Use --bypass to keep certain commands even if repeated:

# Keep all ERROR lines even if they repeat
uniqseq session.log --window-size 5 --bypass 'ERROR'