refactor: modular architecture with unified error handling and DashMap concurrency

- Break monolithic records.rs into commands/, output/, parsing/ modules
- Add lib.rs with shared types (AppError, Car, BestCarRecord, etc.)
- Replace Arc<Mutex<HashMap>> with DashMap for concurrent access
- Add unified AppError enum with thiserror
- Add configurable CLI paths (--my-cars, --cars-db, --times-dir, --tracks-db)
- Add serde deserialization for Car struct
- Add CSV escaping for fields with commas/quotes/newlines
- Fix track name splitting: split_once -> rsplit_once for _a handling
- Add integration tests, proptest property-based tests, and unit tests
- Add rustfmt.toml configuration
- Update documentation (API, ARCHITECTURE, DATA_STRUCTURES, README)
This commit is contained in:
Jan-Peter von Hunnius 2026-05-08 16:19:01 +02:00
parent 9d57b18a4e
commit a9a46ee551
24 changed files with 3541 additions and 1470 deletions

View file

@ -6,543 +6,267 @@ This document provides comprehensive API documentation for the Top Drives Record
## Core Modules
### Records Module (`src/records.rs`)
### Shared Library (`src/lib.rs`)
Primary analysis engine for processing JSON-based timing data.
Common types and utilities shared across the project.
#### Public Types
##### `Car`
**Core car information structure**
```rust
#[derive(Debug, Clone, Deserialize)]
pub struct Car {
pub rid: String, // Repository identifier
pub brand: String, // Manufacturer/brand name
pub only_name: String, // Model name (without brand)
pub year: u16, // Model year
pub rq: u8, // Rarity level (10-113)
pub prize: bool, // Whether car is prize-only
pub guid: String, // Game-specific card identifier
}
```
**Purpose**: Represents a car entity from the records database.
**Deserialization**: Uses `serde::Deserialize` with `#[serde(rename = "onlyName")]` for the `only_name` field and `#[serde(default)]` for `prize`.
##### `BestCarRecord`
**Best performing car record for a specific track/RQ/condition**
```rust
#[derive(Debug, Clone)]
pub struct BestCarRecord {
pub car_rid: String, // Car identifier
pub brand: String, // Manufacturer
pub only_name: String, // Model name
pub year: u16, // Year
pub rq: u8, // RQ level
pub config: String, // Tune config (111, 233, etc.)
pub time: f64, // Track time in seconds
pub owned: bool, // User ownership status
pub prize: bool, // Prize car status
pub guid: String, // Game identifier
}
```
##### `TripletKey`
**Composite key for unique track conditions**
```rust
pub type TripletKey = (String, u8, String);
// Track ID RQ Level Surface Condition
```
##### `AnalysisContext`
**Mutable analysis state**
```rust
pub struct AnalysisContext {
pub best_cars: Arc<Mutex<HashMap<TripletKey, Vec<BestCarRecord>>>>,
}
```
**Methods**:
- `new()` — Create a new, empty analysis context
- `lock_best_cars()` — Acquire a lock on the best_cars map (returns `Result`, converts poison errors to `AppError`)
##### `CarCounts`
**Aggregated per-car win counts**
```rust
pub struct CarCounts {
pub by_rid: HashMap<String, (BestCarRecord, usize)>,
pub guid_to_rid: HashMap<String, String>,
pub car_map: HashMap<String, Car>,
pub car_list: Vec<Car>,
}
```
##### `MyCarsData`
**Parsed user data from `my_cars.json`**
```rust
#[derive(Debug)]
pub struct MyCarsData {
pub owned: HashSet<String>, // Owned car GUIDs
pub wishlist: HashSet<String>, // Wishlist car GUIDs
}
```
##### `AppError`
**Unified error type**
```rust
#[derive(Debug, thiserror::Error)]
pub enum AppError {
#[error("I/O error: {0}")]
Io(#[from] std::io::Error),
#[error("Network error: {0}")]
Network(String),
#[error("Parse error in {file}: {reason}")]
Parse { file: String, reason: String },
#[error("Missing data file: {0}")]
MissingFile(String),
#[error("Internal error: mutex poisoned — {0}")]
Poisoned(String),
}
```
#### Public Functions
##### `main()`
**Entry point for the records analyzer**
##### `translate_track_name(track_code: &str) -> String`
**Translates internal track identifiers to human-readable names**
```rust
#[tokio::main]
async fn main()
```
- **Input**: Track code (e.g., `"laguna"`)
- **Output**: Track name (e.g., `"Laguna Seca"`)
- **Fallback**: Returns original code if no mapping found
- **Data source**: `track_names.json` (compiled into binary via `include_str!`)
**Purpose**: Command-line interface and operation dispatcher
##### `translate_track_type(track_code: &str) -> &'static str`
**Translates condition codes to surface names**
**Arguments**: Command line arguments
- `args[1]`: Command (`fetch`, `best`, `missing`, `trash`, `details`, `track`, `full_script`)
- **Input**: Condition code (e.g., `"00"`)
- **Output**: Surface name (e.g., `"Dry"`)
- **Fallback**: Returns `"Unknown"` for unrecognized codes
**Commands**:
- `fetch`: Download all data files
- `best`: Show best owned cars
- `missing`: Show best unowned cars
- `trash`: Show removable cars
- `details`: Export all timing data
- `track`: Track-specific analysis
- `full_script`: Run fetch + all analysis commands
##### `get_file(file_url: &str, local_file_path: &str) -> Result<(), AppError>`
**Downloads a file from a URL and saves it locally**
**Example**:
```bash
cargo run --bin records -- best
```
- Removes existing file before downloading
- Returns `AppError::Network` on HTTP failures
- Returns `AppError::Io` on file write failures
##### `print_best_cars(filter_out_owned: bool)`
**Displays ranked list of best performing cars**
##### `load_my_cars_data(path: &str) -> Result<MyCarsData, AppError>`
**Loads and parses user car collection data**
```rust
fn print_best_cars(filter_out_owned: bool)
```
- **Input**: Path to `my_cars.json`
- **Returns**: `MyCarsData` with owned and wishlist GUIDs
- **Errors**: `AppError::MissingFile` if file not found, `AppError::Parse` if JSON invalid
**Purpose**: Aggregates performance data across all tracks and conditions
##### `compare_cars(a: &Car, b: &Car) -> std::cmp::Ordering`
**Sorting comparator for cars**
**Arguments**:
- `filter_out_owned`: If true, excludes owned cars from output
- Sorts by: RQ → Brand → Model name
**Output Format**:
```csv
RQ,Brand,Model,Year,Count
85,McLaren,F1,1993,15
82,Ferrari,250 GTO,1962,12
```
##### `compute_car_counts(ctx: &AnalysisContext, car_list: Vec<Car>) -> Result<CarCounts, AppError>`
**Aggregates per-car win counts from analysis context**
**Algorithm**:
1. Count occurrences of each car across all track/RQ/condition combinations
2. Sort by RQ level, then by car identifier
3. Apply ownership filter if requested
- Returns `CarCounts` with by_rid, guid_to_rid, car_map, and car_list
##### `print_best_times()`
**Exports comprehensive timing data for all combinations**
##### `escape_csv_field(field: &str) -> String`
**Escapes a field for safe CSV inclusion**
```rust
fn print_best_times()
```
**Purpose**: Detailed performance export for external analysis
**Output Format**:
```csv
RQ,TrackID,Condition,CarRQ,Brand,Model,Year,Config,Time
85,laguna,Dry,85,McLaren,F1,1993,233,112.45
```
**Data Volume**: Large output file with all track/condition/RQ combinations
##### `print_best_car(track: &String, condition: &String, rq: &String)`
**Finds best car for specific track conditions**
```rust
fn print_best_car(track: &String, condition: &String, rq: &String)
```
**Purpose**: Targeted analysis for specific racing scenarios
**Arguments**:
- `track`: Track identifier (e.g., "laguna", "monaco")
- `condition`: Surface condition (e.g., "Dry", "Wet")
- `rq`: RQ level limit
**Example**:
```rust
print_best_car(&"laguna".to_string(), &"Dry".to_string(), &"85".to_string());
```
##### `print_trash()`
**Identifies cars safe for removal from collection**
```rust
fn print_trash()
```
**Purpose**: Collection optimization recommendations
**Safety Criteria**:
- Car not among best performers on any track
- RQ level below 80 (preserves high-end cars)
- Not a prize car (may have collection value)
**Output Format**:
```csv
RQ,Brand,Model,Year
45,Toyota,Corolla,1995
```
##### `parse_car_times(car: Car)`
**Processes timing data for a single car**
```rust
async fn parse_car_times(car: Car)
```
**Purpose**: Core parsing logic for individual car performance data
**Process Flow**:
1. Load car's JSON timing file
2. Parse each configuration (111, 233, 323, etc.)
3. Extract timing data for each track/condition
4. Apply quality filters (downvotes, time validation)
5. Update global best cars cache
**Quality Filters**:
- Minimum time: 1.2 seconds
- No downvotes from community
- Special handling for Test Bowl track
##### `parse_all_times()`
**Orchestrates concurrent processing of all cars**
```rust
async fn parse_all_times()
```
**Purpose**: Coordinates parallel processing of car timing files
**Concurrency**: One async task per car file, bounded by a semaphore (limit 20)
**Synchronization**: Waits for all tasks to complete
##### `update_best_car(track_id: &str, car_rq: u8, surface: &str, new_record: BestCarRecord)`
**Updates global best cars database with new performance data**
```rust
fn update_best_car(track_id: &str, car_rq: u8, surface: &str, new_record: BestCarRecord)
```
**Purpose**: Implements core best-car selection algorithm
**RQ Range Analysis**:
- Tests car from base RQ up to `MAX_RQ` (113)
- Compares against existing best times
- Updates records for improved performance
**Logic**:
1. For each RQ level from car_rq to `MAX_RQ` (113):
2. If new time is better → replace record
3. If new time ties → add to list (if not duplicate)
4. If new time is worse → ignore
##### `get_car_list() -> Vec<Car>`
**Loads complete car database from repository file**
```rust
fn get_car_list() -> Vec<Car>
```
**Purpose**: Provides access to all known cars in the database
**Source**: `cars_repo.json` file
**Returns**: Vector of all cars with complete metadata
##### `get_my_car_list() -> Vec<String>`
**Loads user's car collection from game export**
```rust
fn get_my_car_list() -> Vec<String>
```
**Purpose**: Identifies which cars the user owns
**Source**: `my_cars.json` game export
**Filter**: Only `locked: true` cards
**Returns**: Vector of owned car GUIDs
##### `fetch_times()`
**Downloads timing files for all cars with rate limiting**
```rust
async fn fetch_times()
```
**Purpose**: Batch download of car timing data
**Rate Limiting**: 20 concurrent downloads maximum (semaphore)
**Progress**: Shows download progress for each file
##### `fetch_car(rid: String)`
**Downloads timing data for single car**
```rust
async fn fetch_car(rid: String)
```
**Purpose**: Individual car data fetching
**URL Format**: `https://api.topdrivesrecords.com/car/{rid}`
**Storage**: `times/{rid}.json`
##### `get_cars_file()`
**Downloads main car database**
```rust
async fn get_cars_file()
```
**Purpose**: Updates car metadata database
**Source**: GitHub repository cars database
**Destination**: `cars_repo.json`
##### `get_tracks_file()`
**Downloads track definitions**
```rust
async fn get_tracks_file()
```
**Purpose**: Updates track name mappings
**Source**: GitHub repository tracks database
**Destination**: `tracks_repo.json`
##### `get_file(file_url: &str, local_file_path: &str)`
**Generic file download utility**
```rust
async fn get_file(file_url: &str, local_file_path: &str)
```
**Purpose**: Downloads files from remote URLs
**Features**:
- Automatic existing file removal
- Error handling with graceful failure
- Progress indication through errors
### Shared Library (`src/lib.rs`)
Common utilities shared across the project.
##### `translate_track_name(track_code: &str) -> &str`
**Converts track codes to human-readable names**
```rust
fn translate_track_name(track_code: &str) -> &str
```
**Purpose**: User-friendly track name display
**Implementation**: Loads mappings from `track_names.json` at compile time via `include_str!`
**Coverage**: All tracks defined in `track_names.json`
**Fallback**: Returns original code if not found
##### `translate_track_type(track_code: &str) -> &str`
**Converts condition codes to surface descriptions**
```rust
fn translate_track_type(track_code: &str) -> &str
```
**Purpose**: Human-readable surface condition names
**Mappings**:
- `"00"` → `"Dry"`
- `"01"` → `"Wet"`
- `"10"` → `"Dirt"`
- `"11"` → `"Dirt Wet"`
##### `split_prefix_postfix<'a>(input: &'a str, delimiter: &'a str) -> Option<(&'a str, &'a str)>`
**String splitting utility for track ID parsing**
```rust
fn split_prefix_postfix<'a>(input: &'a str, delimiter: &'a str) -> Option<(&'a str, &'a str)>
```
**Purpose**: Parse track IDs with condition suffixes
**Example**:
```rust
split_prefix_postfix("laguna_a00", "_a") // Some(("laguna", "00"))
```
## Data Structures
### Records Module Data Types
#### `Card`
**User's car collection entry**
```rust
struct Card {
cardId: String, // Unique card identifier
locked: bool, // Ownership status
}
```
#### `Car`
**Car metadata from database**
```rust
struct Car {
rid: String, // Repository ID
brand: String, // Manufacturer
only_name: String, // Model name
year: u16, // Model year
rq: u8, // Rarity level
prize: bool, // Prize car status
guid: String, // Game GUID
}
```
#### `BestCarRecord`
**Performance record for best cars**
```rust
struct BestCarRecord {
car_rid: String, // Car identifier
brand: String, // Manufacturer
only_name: String, // Model name
year: u16, // Year
rq: u8, // RQ level
config: String, // Tune configuration
time: f64, // Track time
owned: bool, // User ownership
guid: String, // Game identifier
}
```
#### `TripletKey`
**Composite key for track conditions**
```rust
type TripletKey = (String, u8, String);
// Track ID RQ Level Surface Condition
```
### Constants
#### `MAX_RQ`
```rust
const MAX_RQ: u8 = 113;
```
**Purpose**: Maximum RQ level for range analysis. Replaces former magic number 115 with a named constant.
## Global State
### `BEST_CARS` Cache
**Thread-safe global performance database**
```rust
static BEST_CARS: Lazy<Mutex<HashMap<TripletKey, Vec<BestCarRecord>>>>
```
**Initialization**: One-time setup with `once_cell::Lazy`
**Thread Safety**: Protected by `Mutex` for concurrent access
**Structure**: HashMap with composite keys for efficient lookups
## Error Handling
### Network Errors
- **HTTP Failures**: Logged but don't crash application
- **Invalid JSON**: Graceful fallback to defaults
- **Missing Files**: Clear error messages for user
### Parse Errors
- **Malformed Data**: Skip invalid entries
- **Type Conversion**: Use safe defaults
- **Missing Fields**: Provide sensible fallbacks
### User Input Errors
- **Invalid Commands**: Clear usage instructions
- **Missing Arguments**: Descriptive error messages
- **Out of Range**: Validation with helpful feedback
## Usage Examples
### Basic Analysis Workflow
```rust
// 1. Load user's car collection
let my_cars = get_my_car_list();
// 2. Process all car timing data
parse_all_times().await;
// 3. Generate best cars report
print_best_cars(false);
// 4. Find collection optimization opportunities
print_trash();
```
### Track-Specific Analysis
```rust
// Analyze specific track conditions
let track = "laguna".to_string();
let condition = "Dry".to_string();
let rq = "85".to_string();
print_best_car(&track, &condition, &rq);
```
### Data Fetching
```rust
// Download fresh data
get_cars_file().await;
get_tracks_file().await;
fetch_times().await;
```
## Performance Characteristics
### Time Complexity
- **Parsing**: O(n) where n = number of cars × tracks
- **Best Car Selection**: O(1) HashMap operations
- **Sorting**: O(n log n) for output ordering
### Memory Usage
- **Peak**: ~200-300MB during full analysis
- **Baseline**: ~50MB application overhead
- **Per Car**: ~1-2KB metadata + timing data
### Concurrency
- **Downloads**: 20 concurrent connections max (semaphore bounded)
- **Processing**: One task per car file
- **Thread Safety**: Mutex-protected shared state
## Extension Points
### Adding New Commands
1. Add command parsing in `main()`
2. Implement analysis function
3. Update help documentation
### Adding New Tracks
1. Update `tracks_repo.json` with new track definitions
2. Add track name mapping to `track_names.json`
3. Track names are compiled in at build time
### Custom Analysis Types
1. Create new output formatting functions
2. Add aggregation algorithms
3. Extend data structures as needed
## Security Considerations
### Network Security
- HTTPS-only external requests
- Certificate validation (with fallback)
- No sensitive data transmission
### File System Security
- Read-only access to game files
- No modification of user data
- Safe file path handling
### Input Validation
- Command injection prevention
- Path traversal protection
- Input sanitization
## Troubleshooting API Issues
### Common Problems
**"Function not found"**
- Check module imports
- Verify function signatures
- Ensure proper visibility
**"Type mismatch"**
- Review struct field types
- Check function signatures
- Verify generic parameters
**"Deadlock"**
- Avoid nested Mutex locks
- Check lock ordering
- Use timeout where possible
### Debug Strategies
**Logging**:
```rust
println!("Processing car: {}", car.rid);
eprintln!("Error: {}", error);
```
**Tracing**:
```rust
// Add at function entry
let _span = tracing::span!(Level::INFO, "function_name");
let _enter = _span.enter();
```
**Profiling**:
```bash
cargo flamegraph --bin records -- best
```
## Migration Guide
### Version Compatibility
- **Current**: 0.2.0 API stable
- **Previous**: 0.1.0 fully compatible
- **Future**: Semantic versioning maintained
### Breaking Changes
- Club module (`src/club.rs`) has been removed
- `scraper` dependency removed
- `tracks.txt` removed (replaced by `track_names.json`)
- Magic number 115 replaced with `const MAX_RQ: u8 = 113`
- `translate_track_name()` now loads from `track_names.json` instead of hardcoded match
- Wraps in quotes if field contains comma, quote, newline, or carriage return
- Doubles internal quotes per CSV spec
---
**API Version**: 0.2.0
**Last Updated**: 2026-05-06
**Compatibility**: Rust 1.70.0+
### Records Binary (`src/records.rs`)
Primary analysis engine for processing JSON-based timing data.
#### Constants
| Constant | Value | Description |
|----------|-------|-------------|
| `MAX_RQ` | 113 | Maximum RQ level for range analysis |
| `MIN_VALID_TIME` | 1.2 | Minimum valid track time (seconds) |
| `TIME_TOLERANCE` | 0.005 | Tolerance for tie detection (seconds) |
| `TEST_BOWL_FACTOR` | 2000.0 | Multiplier for Test Bowl time inversion |
| `CONCURRENCY_LIMIT` | 20 | Maximum concurrent tasks |
| `DEFAULT_MY_CARS_PATH` | `"my_cars.json"` | Default user data file path |
| `CARS_DB_PATH` | `"cars_repo.json"` | Car database file path |
| `TIMES_DIR` | `"times"` | Timing data directory |
| `CARS_DB_URL` | GitHub URL | Remote car database URL |
| `TRACKS_DB_URL` | GitHub URL | Remote tracks database URL |
| `CAR_TIMING_API_BASE` | API URL | Base URL for per-car timing data |
#### CLI Commands
| Command | Description | Flags |
|---------|-------------|-------|
| `fetch` | Download latest car and track data | None |
| `best` | Generate `best.csv` and `very_best.csv` | `--min-count N` (default: 10) |
| `trash` | Generate `trash.csv` | None |
| `wishlist` | Generate `wishlist.csv`, `add.csv`, `remove.csv` | `--min-count N`, `--max-wishlist M` |
| `all` | Generate all output files in a single pass | `--min-count N`, `--max-wishlist M` |
| `details` | Generate `details.csv` with all timing data | None |
| `track` | Show best car for specific track/condition/RQ | `<track> <condition> <rq>` |
#### Key Internal Functions
##### `run_analysis(owned: &HashSet<String>) -> Result<(AnalysisContext, CarCounts), AppError>`
**Common analysis setup**
Loads user data, parses all timing files, and computes car counts.
##### `parse_car_times(car: Car, my_cars: &HashSet<String>, ctx: &AnalysisContext)`
**Processes timing data for a single car**
- Runs inside `tokio::task::spawn_blocking` to avoid blocking async runtime
- Reads car JSON file from `times/` directory
- Processes each configuration and track
- Applies quality filters (downvotes, minimum time)
- Handles Test Bowl time inversion
##### `update_best_car(best_cars, track_id, car_rq, surface, new_record)`
**Updates best car records with a new performance entry**
- Iterates RQ range from car's RQ to `MAX_RQ`
- Replaces existing record if new time is faster
- Adds to tie list if within `TIME_TOLERANCE`
- Prevents duplicate `car_rid` entries in tie lists
##### `parse_all_times(my_cars, ctx) -> Vec<Car>`
**Orchestrates concurrent parsing of all car timing files**
- Uses `Semaphore(CONCURRENCY_LIMIT)` for bounded concurrency
- Spawns blocking tasks via `tokio::task::spawn_blocking`
- Returns the full car list for downstream use
##### `compute_ideal_wishlist(counts, owned, min_count, max_wishlist) -> Vec<(BestCarRecord, usize)>`
**Computes the ideal wishlist**
- Filters to cars with count >= `min_count`
- Excludes owned cars
- Sorts by RQ
- Caps at `max_wishlist`
#### Output Functions
##### `write_sorted_csv(path, header, entries, cmp, format_row)`
**Generic sorted CSV writer**
- Sorts by index (avoids cloning entries)
- Writes header row
- Formats each row via closure
##### `write_best_csv(counts)`, `write_very_best_csv(counts, min_count)`
**Generate best car CSV files**
- Format: `RQ,Brand,Model,Year,Count`
- Fields are CSV-escaped
##### `write_trash_csv(counts, owned)`
**Generate trash car CSV file**
- Filters: not best on any track, RQ < 80, not prize car
- Format: `RQ,Brand,Model,Year`
##### `write_wishlist_files(counts, owned, wishlist, min_count, max_wishlist)`
**Generate wishlist, add, and remove CSV files**
- Computes ideal wishlist once
- Derives all three files from it
- Format: `RQ,Brand,Model,Year,Count`
##### `write_details_csv(ctx)`
**Generate comprehensive timing data CSV**
- Format: `RQ,TrackID,Condition,CarRQ,Brand,Model,Year,Config,Time`
- Sorted by RQ then track ID
##### `print_best_car(ctx, track, condition, rq)`
**Print best car for a specific track/condition/RQ to stdout**