Refactor: extract lib.rs into focused sub-modules, add --output-dir CLI option

- Split monolithic src/lib.rs (~640 lines) into 7 sub-modules: types.rs, context.rs, user_data.rs, translate.rs, http.rs, csv_util.rs, test_utils.rs - Add configurable --output-dir CLI option for CSV output files - Implement atomic file writes (write-to-temp + rename) for all CSVs - Change compute_car_counts() to return CarCounts directly (not Result) - Add test-utils feature gate for shared test helpers - Add HTTP mock tests with mockito (6 tests) - Add proptest tests for compute_car_counts (3 property tests) - Update integration tests to use shared test utilities - Update documentation (API.md, ARCHITECTURE.md, DATA_STRUCTURES.md)
2026-05-08 20:21:46 +02:00 · 2026-05-08 20:21:46 +02:00 · 67a79f190a
commit 67a79f190a
parent d2c1418e5e
27 changed files with 1648 additions and 789 deletions
--- a/plans/code-review-v2.md
+++ b/plans/code-review-v2.md
@ -0,0 +1,394 @@
+# Code Review — Top Drives Records Analyzer (v0.3.0)
+
+**Date**: 2026-05-08  
+**Reviewer**: Architect Mode  
+**Scope**: Full codebase review after round-1 refactoring
+
+---
+
+## Executive Summary
+
+The codebase has improved significantly since the initial review. The module split is clean, error handling is consistent, and test coverage is solid. The remaining issues are **design-level improvements** — not bugs or correctness problems. They focus on reducing coupling, improving maintainability, and making the codebase more idiomatic Rust.
+
+**Overall Assessment**: ✅ Good — well-structured, well-documented, well-tested. The suggestions below are quality refinements.
+
+---
+
+## 1. Architecture & Design
+
+### 1.1 ✅ Strengths
+
+- **Clean module separation**: `commands/` → `parsing/` → `output/` pipeline is well-defined
+- **Proper error propagation**: `AppError` enum with `thiserror` is used consistently
+- **Good concurrency model**: `DashMap` + `Semaphore` + `spawn_blocking` is the right approach
+- **Zero global mutable state**: `AnalysisContext` is created in `main()` and threaded through
+- **Excellent documentation**: Module-level doc comments, function docs, and separate docs/ directory
+
+### 1.2 🔧 `lib.rs` is a God Module (Medium)
+
+[`src/lib.rs`](src/lib.rs) at ~650 lines contains **all** shared types and utilities: `AppError`, `Car`, `BestCarRecord`, `AnalysisContext`, `CarCounts`, `MyCarsData`, `compute_car_counts`, `load_my_cars_data`, `compare_cars`, `get_file`, `translate_track_name`, `translate_track_type`, `escape_csv_field`, plus 200 lines of tests.
+
+**Problem**: Any change to any type requires recompiling all downstream modules. The file has 6+ distinct responsibilities.
+
+**Suggestion**: Split into focused sub-modules within `lib.rs`:
+
+```
+src/lib.rs           → re-exports only
+src/types.rs         → Car, BestCarRecord, TripletKey, AppError
+src/context.rs       → AnalysisContext, CarCounts, compute_car_counts
+src/user_data.rs     → MyCarsData, load_my_cars_data
+src/translate.rs     → translate_track_name, translate_track_type
+src/http.rs          → get_file
+src/csv_util.rs      → escape_csv_field, compare_cars
+```
+
+### 1.3 🔧 `output/mod.rs` Has Mixed Responsibilities (Low-Medium)
+
+[`src/output/mod.rs`](src/output/mod.rs) contains:
+- Generic CSV infrastructure (`write_sorted_csv`, `format_counted_row`)
+- Specific output commands (`write_best_csv`, `write_trash_csv`, `write_details_csv`, `write_wishlist_files`)
+- Business logic (`compute_ideal_wishlist`)
+- Stdout output (`print_best_car`)
+- 340 lines of tests
+
+**Suggestion**: Split into:
+- `output/csv.rs` — generic CSV infrastructure + `format_counted_row`
+- `output/wishlist.rs` — wishlist-specific logic (`compute_ideal_wishlist`, `write_wishlist_files`)
+- `output/track_query.rs` — `print_best_car` (stdout output is conceptually different from CSV files)
+
+### 1.4 🔧 `DashMap<Arc<...>>` — Double Arc Wrapping (Low)
+
+In [`src/lib.rs:127`](src/lib.rs:127):
+```rust
+pub best_cars: Arc<DashMap<TripletKey, Vec<BestCarRecord>>>,
+```
+
+Then in [`src/parsing/mod.rs:252`](src/parsing/mod.rs:252):
+```rust
+let best_cars = Arc::clone(&ctx.best_cars);
+```
+
+`DashMap` already uses `Arc` internally for its shard locks. Wrapping it in another `Arc` is redundant — `DashMap` itself is `Clone + Send + Sync` and can be shared by cloning the `DashMap` handle directly.
+
+**Suggestion**: Change to:
+```rust
+pub best_cars: DashMap<TripletKey, Vec<BestCarRecord>>,
+```
+
+And share via `ctx.best_cars.clone()` (which clones only the Arc handle, not the data).
+
+---
+
+## 2. Code Quality
+
+### 2.1 🔧 `update_best_car` String Allocation in Hot Loop (Medium)
+
+In [`src/parsing/mod.rs:184-185`](src/parsing/mod.rs:184):
+```rust
+let track_id_owned = track_id.to_string();
+let surface_owned = surface.to_string();
+
+for rq in car_rq..=MAX_RQ {
+    let key = (track_id_owned.clone(), rq, surface_owned.clone());
+```
+
+For a car with RQ 10, this loop runs **104 times**, cloning two strings per iteration. The `track_id_owned` and `surface_owned` are only needed to construct keys.
+
+**Suggestion**: Use `Cow<str>` or restructure to avoid per-iteration clones:
+```rust
+for rq in car_rq..=MAX_RQ {
+    let key = (track_id.to_string(), rq, surface.to_string());
+```
+This is the same number of allocations but avoids the intermediate owned strings. Or, since `TripletKey = (String, u8, String)`, consider changing the key type to `(Arc<str>, u8, Arc<str>)` to make cloning O(1).
+
+### 2.2 🔧 `compute_car_counts` Returns `Result` But Never Fails (Low)
+
+In [`src/lib.rs:156`](src/lib.rs:156):
+```rust
+pub fn compute_car_counts(ctx: &AnalysisContext, car_list: Vec<Car>) -> Result<CarCounts, AppError> {
+```
+
+This function only does `HashMap` inserts and `DashMap` iteration — it **cannot fail**. The `Result` return is misleading and forces all callers to `.unwrap()`.
+
+**Suggestion**: Return `CarCounts` directly:
+```rust
+pub fn compute_car_counts(ctx: &AnalysisContext, car_list: Vec<Car>) -> CarCounts {
+```
+
+### 2.3 🔧 Hardcoded Output File Paths (Low-Medium)
+
+All CSV output functions hardcode their filenames:
+- [`write_best_csv`](src/output/mod.rs:78) → `"best.csv"`
+- [`write_very_best_csv`](src/output/mod.rs:91) → `"very_best.csv"`
+- [`write_trash_csv`](src/output/mod.rs:110) → `"trash.csv"`
+- [`write_details_csv`](src/output/mod.rs:148) → `"details.csv"`
+- [`write_wishlist_files`](src/output/mod.rs:205) → `"wishlist.csv"`, `"add.csv"`, `"remove.csv"`
+
+**Problem**: Users cannot specify an output directory. All files are written to CWD.
+
+**Suggestion**: Add an `--output-dir` CLI option and pass the output directory through to the write functions. Or at minimum, extract the filenames as constants.
+
+### 2.4 🔧 Magic Number `80` in Trash Filter (Low)
+
+In [`src/output/mod.rs:114`](src/output/mod.rs:114):
+```rust
+.filter(|car| !car.prize && car.rq < 80)
+```
+
+The value `80` is a business rule (minimum RQ for keeping a non-best car) but is not named.
+
+**Suggestion**: Extract as a named constant:
+```rust
+const TRASH_MIN_RQ: u8 = 80;
+```
+
+### 2.5 🔧 `write_details_csv` Doesn't Use `write_sorted_csv` (Low)
+
+[`write_details_csv`](src/output/mod.rs:148) manually creates a file and iterates, while all other output functions use the generic [`write_sorted_csv`](src/output/mod.rs:35). This is because `details.csv` has a different row format, but the pattern could still be unified.
+
+**Suggestion**: Either refactor `write_sorted_csv` to be more flexible, or add a comment explaining why this function is different.
+
+### 2.6 💡 `#[allow(clippy::new_without_default)]` — Just Implement Default (Low)
+
+In [`src/lib.rs:132`](src/lib.rs:132):
+```rust
+#[allow(clippy::new_without_default)]
+pub fn new() -> Self {
+```
+
+Since `AnalysisContext` only contains `Arc<DashMap<...>>` which implements `Default`, the derive is trivial:
+
+**Suggestion**: 
+```rust
+#[derive(Default)]
+pub struct AnalysisContext { ... }
+```
+Then remove the manual `new()` and the allow attribute.
+
+---
+
+## 3. Error Handling
+
+### 3.1 ✅ Strengths
+
+- `AppError` with `thiserror` is clean and comprehensive
+- `load_my_cars_or_warn` correctly distinguishes `MissingFile` from real errors
+- `fetch_times` returns the first error encountered
+- `parse_all_times` detects total failure and propagates
+
+### 3.2 🔧 `writeln!` Errors Silently Propagated via `?` in `write_sorted_csv` (Low)
+
+In [`src/output/mod.rs:49-52`](src/output/mod.rs:49):
+```rust
+writeln!(file, "{header}")?;
+for i in indices {
+    format_row(&entries[i], &mut file)?;
+}
+```
+
+The `?` operator converts `std::io::Error` → `AppError::Io` via the `From` impl. This is correct but means a partial CSV file may be left on disk if writing fails mid-way.
+
+**Suggestion**: Consider writing to a temporary file and renaming on success (same pattern as [`get_file`](src/lib.rs:279)), or at minimum document that partial writes are possible.
+
+---
+
+## 4. Performance
+
+### 4.1 🔧 `parse_all_times` Clones Every Car (Medium)
+
+In [`src/parsing/mod.rs:257`](src/parsing/mod.rs:257):
+```rust
+let car = car.clone();
+```
+
+Every car in the database is cloned to move into the async task. For ~8000+ cars, each with 5+ string fields, this is significant.
+
+**Suggestion**: Use `Arc<Car>` or pass only the needed fields (rid, rq, guid, brand, only_name, year, prize) as a smaller struct.
+
+### 4.2 🔧 `times_dir` Cloned Per Car (Low)
+
+In [`src/parsing/mod.rs:259`](src/parsing/mod.rs:259):
+```rust
+let times_dir = times_dir.to_string();
+```
+
+This allocates a new String for every car. 
+
+**Suggestion**: Use `Arc<str>` or pass as `&str` by restructuring the spawn.
+
+### 4.3 💡 Consider `rayon` for `compute_car_counts` (Low)
+
+[`compute_car_counts`](src/lib.rs:156) iterates all `DashMap` entries sequentially. For large datasets, this could benefit from parallel iteration.
+
+---
+
+## 5. Test Coverage
+
+### 5.1 ✅ Strengths
+
+- **Unit tests** in every module (`lib.rs`, `parsing/mod.rs`, `parsing/car_list.rs`, `output/mod.rs`)
+- **Integration tests** in `tests/integration_tests.rs` covering full pipelines
+- **Property-based tests** in `tests/proptest_tests.rs` for sorting invariants
+- **Edge case tests**: empty input, missing files, downvoted times, Test Bowl inversion
+- Good use of `tempfile::tempdir()` for file-based tests
+
+### 5.2 🔧 Duplicated Test Helpers (Medium)
+
+The `make_record()` and `make_car()` helper functions are duplicated across:
+- [`src/parsing/mod.rs:369`](src/parsing/mod.rs:369)
+- [`src/output/mod.rs:319`](src/output/mod.rs:319)
+- [`tests/integration_tests.rs:17`](tests/integration_tests.rs:17)
+
+**Suggestion**: Create `tests/common/mod.rs` or a `test_utils` module in `lib.rs` with shared helpers:
+
+```rust
+// src/test_utils.rs (gated behind #[cfg(test)])
+pub fn make_record(car_rid: &str, rq: u8, time: f64, guid: &str) -> BestCarRecord { ... }
+pub fn make_car(rid: &str, rq: u8, guid: &str) -> Car { ... }
+```
+
+### 5.3 🔧 Missing Test Coverage for `get_file` (Medium)
+
+[`get_file`](src/lib.rs:275) is the most complex function in `lib.rs` (~75 lines with retry logic, temp file handling, empty response detection). It has **zero tests**.
+
+**Suggestion**: Add tests using a mock HTTP server (e.g., `mockito` crate) or at minimum test the retry/backoff logic.
+
+### 5.4 🔧 Missing Test Coverage for `fetch_times` and `fetch_car` (Medium)
+
+[`fetch_times`](src/parsing/mod.rs:317) and [`fetch_car`](src/parsing/mod.rs:351) have no tests. These contain important concurrency and error-handling logic.
+
+### 5.5 🔧 No Tests for Command Handlers (Low)
+
+None of the command handlers in `src/commands/` have dedicated tests. While they're thin wrappers, testing `load_my_cars_or_warn`'s graceful degradation and `run_analysis`'s orchestration would be valuable.
+
+### 5.6 💡 `proptest` Only Tests `compare_cars` and `escape_csv_field` (Low)
+
+The property-based test suite could be expanded to cover:
+- `update_best_car` — properties like "inserting the same car twice doesn't change the result"
+- `compute_car_counts` — "count is always >= 1 for any car in `best_cars`"
+- `translate_track_name` — "output is never empty for known tracks"
+
+---
+
+## 6. Documentation
+
+### 6.1 ✅ Strengths
+
+- All public functions have doc comments with purpose, arguments, and return values
+- Module-level `//!` doc comments explain each module's role
+- `docs/` directory covers API, architecture, data structures, setup, and user guide
+- `CHANGELOG.md` tracks version history
+
+### 6.2 🔧 `docs/DATA_STRUCTURES.md` May Be Outdated (Low)
+
+Need to verify that `DATA_STRUCTURES.md` reflects the current struct definitions, especially `AnalysisContext` with `DashMap` and the new `CarCounts` struct.
+
+### 6.3 🔧 `docs/DEVELOPMENT_SETUP.md` — Missing `mockito` or Test Dependencies (Low)
+
+If `mockito` or other test dependencies are added, they should be documented.
+
+### 6.4 💡 No `CONTRIBUTING.md` (Low)
+
+The README mentions "Contributing" in the table of contents but there's no contributing guide.
+
+---
+
+## 7. Dependencies
+
+### 7.1 ✅ Strengths
+
+- Tokio features are properly narrowed
+- `thiserror v2`, `dashmap v6`, `clap v4` are all current
+- `proptest` and `tempfile` are correctly in `[dev-dependencies]`
+
+### 7.2 🔧 Consider Adding `mockito` (Low)
+
+For testing `get_file`'s retry logic and HTTP error handling.
+
+### 7.3 💡 Consider Replacing `futures::future::join_all` with `tokio::join!` or `futures::join!` (Low)
+
+In [`src/parsing/mod.rs:274`](src/parsing/mod.rs:274):
+```rust
+let results = join_all(handles).await;
+```
+
+`join_all` buffers all futures in memory. For ~8000 cars this means ~8000 `JoinHandle`s allocated upfront. This is fine for the current scale but could be replaced with `FuturesUnordered` for streaming results.
+
+---
+
+## 8. Summary of Recommended Improvements
+
+### Priority 1 — Design Improvements
+
+| # | Issue | File | Impact |
+|---|-------|------|--------|
+| 1 | Split `lib.rs` into sub-modules | `src/lib.rs` | Reduces coupling, faster compile times |
+| 2 | Remove outer `Arc` from `DashMap` field | `src/lib.rs:127` | Removes unnecessary indirection |
+| 3 | Deduplicate test helpers | Multiple | Reduces maintenance burden |
+| 4 | Add `mockito` + tests for `get_file` | `src/lib.rs` | Covers untested critical path |
+
+### Priority 2 — Code Quality
+
+| # | Issue | File | Impact |
+|---|-------|------|--------|
+| 5 | `compute_car_counts` should return `CarCounts` not `Result` | `src/lib.rs:156` | Removes misleading API |
+| 6 | Extract `TRASH_MIN_RQ` constant | `src/output/mod.rs:114` | Names business rule |
+| 7 | Implement `Default` for `AnalysisContext` | `src/lib.rs:132` | Removes clippy allow |
+| 8 | Add `--output-dir` CLI option | `src/records.rs` | User configurability |
+| 9 | Atomic CSV writes via temp+rename | `src/output/mod.rs` | Prevents partial files |
+
+### Priority 3 — Performance & Polish
+
+| # | Issue | File | Impact |
+|---|-------|------|--------|
+| 10 | Reduce string cloning in `update_best_car` loop | `src/parsing/mod.rs:184` | Performance |
+| 11 | Use `Arc<Car>` or slim struct in `parse_all_times` | `src/parsing/mod.rs:257` | Reduces allocations |
+| 12 | Split `output/mod.rs` into sub-modules | `src/output/` | Maintainability |
+| 13 | Expand proptest coverage | `tests/proptest_tests.rs` | Test confidence |
+| 14 | Add command handler tests | `src/commands/` | Test coverage |
+
+---
+
+## Architecture Diagram (Current)
+
+```mermaid
+graph TD
+    CLI[src/records.rs - CLI Entry Point] --> CMD[src/commands/ - Command Handlers]
+    CMD --> PARSE[src/parsing/ - Core Parsing]
+    CMD --> OUT[src/output/ - CSV Output]
+    PARSE --> LIB[src/lib.rs - Shared Library]
+    CMD --> LIB
+    OUT --> LIB
+    
+    LIB --> TYPES[Types: Car, BestCarRecord, AppError]
+    LIB --> CTX[AnalysisContext with DashMap]
+    LIB --> UTILS[Utilities: translate_*, get_file, escape_csv_field]
+    
+    PARSE --> |spawn_blocking| PARSE
+    PARSE --> |Semaphore limited| FETCH[fetch_times - HTTP Downloads]
+    
+    CMD --> |run_analysis| PARSE
+    CMD --> |compute_car_counts| LIB
+```
+
+## Proposed Architecture (After Refactoring)
+
+```mermaid
+graph TD
+    CLI[src/records.rs - CLI Entry Point] --> CMD[src/commands/ - Command Handlers]
+    CMD --> PARSE[src/parsing/ - Core Parsing]
+    CMD --> OUT[src/output/ - CSV Output]
+    PARSE --> LIB[src/lib.rs - Re-exports only]
+    CMD --> LIB
+    OUT --> LIB
+    
+    LIB --> TYPES[src/types.rs - Car, BestCarRecord, AppError]
+    LIB --> CTX[src/context.rs - AnalysisContext, CarCounts]
+    LIB --> USER[src/user_data.rs - MyCarsData]
+    LIB --> TRANS[src/translate.rs - Track name/type]
+    LIB --> HTTP[src/http.rs - get_file with retry]
+    LIB --> CSV[src/csv_util.rs - escape_csv_field, compare_cars]
+    
+    OUT --> WISHLIST[src/output/wishlist.rs]
+    OUT --> TRACK_Q[src/output/track_query.rs]
+```