top_drives/docs/ARCHITECTURE.md

# Architecture Documentation

## System Overview

The Top Drives Records Analyzer is a Rust CLI tool for analyzing car performance data in the Top Drives racing game. It consists of a shared library (`src/lib.rs`), a binary entry point (`src/records.rs`), and three internal module trees: `commands/`, `parsing/`, and `output/`.

## Component Architecture

### 1. Shared Library (`src/lib.rs`)

**Thin re-export module** that declares 6 focused sub-modules and re-exports all public types for backward compatibility. External code can use `top_drives::Car` without knowing the internal module layout.

#### Sub-modules

| Module | File | Key Exports |
|--------|------|-------------|
| `types` | `src/types.rs` | `AppError`, `BestCarRecord`, `Car`, `TripletKey` |
| `context` | `src/context.rs` | `AnalysisContext`, `CarCounts`, `compute_car_counts` |
| `user_data` | `src/user_data.rs` | `MyCarsData`, `load_my_cars_data` |
| `translate` | `src/translate.rs` | `translate_track_name`, `translate_track_type` |
| `http` | `src/http.rs` | `get_file` |
| `csv_util` | `src/csv_util.rs` | `compare_cars`, `escape_csv_field` |
| `test_utils` | `src/test_utils.rs` | `make_record`, `make_car` (test helpers) |

#### Error Handling
All public functions return `Result<T, AppError>` where `AppError` is a unified error type:
- `AppError::Io` — File system errors
- `AppError::Network` — HTTP/download errors
- `AppError::Parse` — JSON parsing errors with file context
- `AppError::MissingFile` — Required data file not found
- `AppError::Internal` — Runtime errors (semaphore closed, task join failures)

### 2. Records Binary (`src/records.rs`)

**CLI entry point** that defines command-line arguments and dispatches to command handlers.

#### Key Responsibilities
- **CLI Definition**: `clap` derive-based argument parsing with global path options
- **Command Dispatch**: Routes to the appropriate command handler in `commands/`
- **Error Reporting**: Catches all errors and prints to stderr with exit code 1

#### Global CLI Options
- `--my-cars` — Path to user's car collection export (default: `my_cars.json`)
- `--cars-db` — Path to car database file (default: `cars_repo.json`)
- `--times-dir` — Directory for per-car timing JSON files (default: `times`)
- `--tracks-db` — Path to tracks database file (default: `tracks_repo.json`)
- `--output-dir` — Directory for output CSV files (default: `.`)

### 3. Commands Module (`src/commands/`)

**Command handlers** that orchestrate analysis and output generation.

| Module | Command | Description |
|--------|---------|-------------|
| `fetch.rs` | `fetch` | Downloads car DB, tracks DB, and per-car timing files |
| `best.rs` | `best` | Generates `best.csv` and `very_best.csv` |
| `trash.rs` | `trash` | Generates `trash.csv` (owned cars not best on any track) |
| `wishlist.rs` | `wishlist` | Generates `wishlist.csv`, `add.csv`, `remove.csv` |
| `details.rs` | `details` | Generates `details.csv` with all timing data |
| `track.rs` | `track` | Prints best car for a specific track/condition/RQ |
| `all.rs` | `all` | Generates all output files in a single analysis pass |

#### Shared Helpers (`src/commands/mod.rs`)
- `run_analysis(owned, cars_db_path, times_dir)` — Common analysis setup: loads car DB, parses timing files, computes counts
- `load_my_cars_or_warn(path)` — Loads user data with graceful degradation (returns empty data if file missing)

### 4. Parsing Module (`src/parsing/`)

**Core parsing logic** for car timing data and car database loading.

#### Key Constants
- `MAX_RQ: u8 = 113` — Maximum RQ level used for range analysis
- `MIN_VALID_TIME: f64 = 1.2` — Minimum valid track time threshold
- `TIME_TOLERANCE: f64 = 0.005` — Tolerance for tie detection (5ms)
- `TEST_BOWL_FACTOR: f64 = 2000.0` — Multiplier for Test Bowl time inversion
- `CONCURRENCY_LIMIT: usize = 20` — Maximum concurrent tasks

#### Key Functions
- `parse_car_times()` — Synchronous per-car timing file parser (runs inside `spawn_blocking`)
- `update_best_car()` — Updates `DashMap` with new performance records across RQ range
- `parse_all_times()` — Orchestrates concurrent parsing with semaphore + `spawn_blocking`
- `fetch_times()` — Downloads per-car timing files with bounded concurrency
- `get_car_list()` (`car_list.rs`) — Loads car database via serde deserialization

### 5. Output Module (`src/output/`)

**CSV output utilities** and sorting helpers.

- `write_sorted_csv()` — Generic sorted CSV writer (index-based sorting to avoid cloning)
- `write_best_csv()`, `write_very_best_csv()` — Best car CSV output
- `write_trash_csv()` — Trash car CSV output
- `write_details_csv()` — Comprehensive timing data CSV output
- `write_wishlist_files()` — Wishlist, add, and remove CSV output
- `print_best_car()` — Stdout output for track-specific query
- `compute_ideal_wishlist()` — Ideal wishlist computation
- `compare_records()` — Sorting comparator for `BestCarRecord`

#### Data Flow

```
GitHub Repository / API
        ↓
cars_repo.json + tracks_repo.json  ←  fetch command
        ↓
times/*.json (1000+ car files)     ←  fetch command
        ↓
parse_all_times() → parse_car_times() × N cars (concurrent via spawn_blocking)
        ↓
AnalysisContext (DashMap<TripletKey, Vec<BestCarRecord>>)
        ↓
compute_car_counts() → CarCounts
        ↓
User Collection (my_cars.json) → owned/wishlist sets
        ↓
Output: CSV files / Terminal Results
```

## Data Structures

### Core Types

#### TripletKey
```rust
type TripletKey = (String, u8, String);
//   ↓           ↓        ↓
// Track ID    RQ Level  Surface Condition
```

**Purpose**: Composite key for uniquely identifying track conditions
**Usage**: `DashMap` key for storing best car records

#### Car
```rust
#[derive(Debug, Clone, Deserialize)]
struct Car {
    rid: String,        // Repository ID
    brand: String,      // Manufacturer
    only_name: String,  // Model name (serde rename: "onlyName")
    year: u16,          // Model year
    rq: u8,             // Rarity level
    prize: bool,        // Prize car status (serde default)
    guid: String,       // Game GUID
}
```

#### BestCarRecord
```rust
struct BestCarRecord {
    car_rid: String,   // Car identifier
    brand: String,     // Manufacturer
    only_name: String, // Model name
    year: u16,         // Year
    rq: u8,            // RQ level
    config: String,    // Tune config (111, 233, etc.)
    time: f64,         // Track time (seconds)
    owned: bool,       // User ownership status
    prize: bool,       // Prize car status
    guid: String,      // Game identifier
}
```

## State Management

### AnalysisContext
```rust
struct AnalysisContext {
    best_cars: Arc<DashMap<TripletKey, Vec<BestCarRecord>>>,
}
```

**Thread Safety**: Uses `DashMap` for sharded internal locking, providing high-concurrency access without a global mutex.
**Lifetime**: Created in command handlers, passed through the call chain.
**Access**: Direct `DashMap` methods (`insert`, `get`, `iter`, `entry`).

### CarCounts
```rust
struct CarCounts {
    by_rid: HashMap<String, (BestCarRecord, usize)>,
    guid_to_rid: HashMap<String, String>,
    car_map: HashMap<String, Car>,
    car_list: Vec<Car>,
}
```

**Purpose**: Aggregated per-car win counts computed once from `AnalysisContext`
**Usage**: Shared across all output functions to avoid redundant computation

## Algorithm Details

### RQ Range Analysis

The system analyzes car performance across multiple RQ brackets:

1. **Base RQ**: Car's natural rarity level
2. **RQ Range**: Tests from car's RQ up to `MAX_RQ` (113)
3. **Cross-RQ Competition**: Allows cars to compete above their base level

**Example**: A RQ 85 car might perform well at RQ 90-95 levels against higher-rated cars

### Time Validation

Multiple validation layers ensure data quality:

1. **Minimum Time**: Times below `MIN_VALID_TIME` (1.2s) are discarded
2. **Downvote Filter**: Community-downvoted times are excluded
3. **Tie Detection**: Times within `TIME_TOLERANCE` (5ms) are considered equal
4. **Special Cases**: Test Bowl track uses inverted time calculation (`TEST_BOWL_FACTOR / time`)
5. **Track Name Splitting**: Uses `rsplit_once("_a")` to correctly handle track names containing `_a` (e.g., `airplaneHangars`)

### Concurrent Processing

#### Bounded Concurrency Pattern
- **Semaphore First**: Semaphore permit acquired in async context (avoids deadlocking the blocking thread pool)
- **Blocking Work**: Synchronous parsing runs inside `tokio::task::spawn_blocking` while holding the permit
- **Concurrency Limit**: Maximum `CONCURRENCY_LIMIT` (20) simultaneous operations

#### Synchronization
- **DashMap**: Sharded concurrent HashMap — no global lock needed for parallel parsing
- **Join Handles**: Waits for all parsing tasks to complete
- **Error Collection**: Parse errors are counted and reported; all-cars-failure propagates as an error

## File Organization

### Input Files
```
├── cars_repo.json          # Car database from repository
├── tracks_repo.json        # Track definitions
├── track_names.json        # Track code → name mappings (compiled into binary)
├── my_cars.json            # User's car collection (game export)
└── times/                  # Individual car timing files
    ├── Ferrari_F40_1987.json
    ├── McLaren_F1_1993.json
    └── ...
```

### Output Files
```
├── best.csv              # All cars best on at least one track
├── very_best.csv         # Cars best on ≥ min_count tracks
├── details.csv           # Detailed timing data for all combinations
├── trash.csv             # Owned cars safe to remove
├── wishlist.csv          # Ideal wishlist (very_best unowned, capped)
├── add.csv               # Cars to add to wishlist
└── remove.csv            # Cars to remove from wishlist
```

## Performance Characteristics

- **Startup Time**: Fast (uses cached JSON files)
- **Memory Usage**: Moderate (loads all timing data into memory)
- **Accuracy**: High (comprehensive dataset)
- **Update Frequency**: Manual refresh required
- **Concurrency**: Bounded at 20 simultaneous operations via semaphore
- **Lock Contention**: Minimal (DashMap sharded locking replaces global Mutex)
- **Allocation Optimization**: `update_best_car` avoids cloning on the last RQ iteration

## Error Handling Strategy

### Unified Error Type
All errors are represented as `AppError`:
- **I/O Errors**: File read/write failures
- **Network Errors**: Download failures with descriptive messages
- **Parse Errors**: JSON parsing failures with file name context
- **Missing Files**: Specific variant for absent data files
- **Internal Errors**: Runtime failures (semaphore closed, task join errors)

### Graceful Degradation
- **Missing User Data**: `load_my_cars_or_warn()` returns empty data with a warning
- **Missing Timing Files**: Individual car file failures emit warnings and skip
- **Network Errors**: Fetch errors logged; first error propagated
- **All Parse Failure**: If every car fails to parse, the error is propagated

### User Feedback
- **Progress Indication**: Shows fetching/download progress
- **Error Messages**: Clear descriptions via `AppError` Display impl
- **Warning Messages**: Non-fatal issues reported via stderr

## Extension Points

### Adding New Tracks
1. Update `tracks_repo.json` with new track definitions
2. Add track name mapping in `track_names.json`
3. Add condition mappings in `translate_track_type()` if needed

### Adding New Commands
1. Create new command handler in `src/commands/`
2. Add command variant to the `Commands` enum in `src/records.rs`
3. Add dispatch case in the `run()` function
4. Update documentation

### Adding New Output Formats
1. Create new output function(s) in `src/output/mod.rs`
2. Reuse `CarCounts` and `AnalysisContext` for data access
3. Add CLI flag for format selection if needed

## Security Considerations

### Network Access
- **HTTPS Only**: All external requests use HTTPS
- **Rate Limiting**: Respects server limits with controlled concurrency
- **Atomic Writes**: Downloads write to `.tmp` files first, then rename on success

### Local File Access
- **Read Only**: No modification of game files
- **User Data**: Only reads exported collection data
- **No Persistence**: Analysis results are output-only

## Future Enhancements

### Potential Improvements
1. **Caching Layer**: Cache parsed results between runs
2. **Database Backend**: Replace in-memory HashMap with persistent storage
3. **Configuration Files**: External configuration for analysis parameters
4. **GUI Interface**: Web-based interface for non-technical users
5. **Export Formats**: Multiple output formats (JSON, XML, etc.)

### Scalability Considerations
1. **Pagination**: Handle large datasets with pagination
2. **Incremental Updates**: Only fetch changed data
3. **Compression**: Compress large timing files
4. **Streaming**: Process data without loading everything into memory