top_drives/docs/DATA_STRUCTURES.md

# JSON Data Structures Documentation

## Overview

This document describes the JSON data structures used by the Top Drives Records Analyzer, including their purpose, format, and usage within the application.

## Core Data Files

### 1. Car Database (`cars_repo.json`)

**Purpose**: Master database of all cars in Top Drives with their metadata.

**Source**: Downloaded from `https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/cars_final.json`

**Structure**:
```json
[
  {
    "rid": "Ferrari_F40_1987",
    "brand": "Ferrari",
    "onlyName": "F40",
    "year": 1987,
    "rq": 95,
    "prize": false,
    "guid": "card_ferrari_f40_1987"
  }
]
```

**Field Descriptions**:

| Field | Type | Description |
|-------|------|-------------|
| `rid` | String | Repository identifier (unique) |
| `brand` | String | Car manufacturer/brand name |
| `onlyName` | String | Car model name (without brand) |
| `year` | Number | Model year |
| `rq` | Number | Rarity level (10-113) |
| `prize` | Boolean | Whether car is prize-only |
| `guid` | String | Game-specific card identifier |

**Usage in Application**:
- Loaded by `get_car_list()` function
- Used to identify cars for timing file processing
- Provides metadata for analysis output

### 2. User's Car Collection (`my_cars.json`)

**Purpose**: Contains the user's owned cars exported from the Top Drives game.

**Source**: Exported from Top Drives game (Settings → Account → Export Data)

**Structure**:
```json
{
  "playerDeck": [
    {
      "cardId": "card_ferrari_f40_1987",
      "locked": true,
      "engineMajor": 3,
      "weightMajor": 2,
      "chassisMajor": 3
    },
    {
      "cardId": "card_mclaren_f1_1993",
      "locked": false,
      "engineMajor": 2,
      "weightMajor": 3,
      "chassisMajor": 3
    }
  ]
}
```

**Field Descriptions**:

| Field | Type | Description |
|-------|------|-------------|
| `playerDeck` | Array | List of user's cards |
| `cardId` | String | Unique card identifier |
| `locked` | Boolean | Whether card is owned (true = owned) |
| `engineMajor` | Number | Engine tune level (optional) |
| `weightMajor` | Number | Weight tune level (optional) |
| `chassisMajor` | Number | Chassis tune level (optional) |

**Processing Logic**:
- Only `locked: true` cards are considered owned
- `cardId` values are matched against car database GUIDs
- Used to mark cars as "owned" in analysis results
- Tune config extracted as `"{engineMajor}{weightMajor}{chassisMajor}"` (e.g., `"323"`)
- Cards with incomplete tune data are skipped for the improve command with a warning

### 3. Individual Car Timing Files (`times/{rid}.json`)

**Purpose**: Contains detailed timing data for a specific car across all tracks and conditions.

**Source**: Downloaded from `https://api.topdrivesrecords.com/car/{rid}`

**Example Structure**:
```json
{
  "users": ["Player1", "Player2"],
  "data": {
    "v2c_111": {
      "times": {
        "laguna_a00": {
          "t": 112.45,
          "u": "Player1"
        },
        "monaco_a00": {
          "t": 45.23,
          "u": "Player2",
          "down": ["Player3"]
        }
      },
      "info": {
        "topSpeed": "201",
        "acel": "3.2",
        "hand": "92"
      }
    }
  },
  "rid": "Ferrari_F40_1987"
}
```

**Field Descriptions**:

| Field | Type | Description |
|-------|------|-------------|
| `users` | Array | List of players who submitted times |
| `data` | Object | Timing data organized by configuration |
| `rid` | String | Car repository identifier |

**Configuration Structure** (`data.{config}`):

| Field | Type | Description |
|-------|------|-------------|
| `times` | Object | Track timing data |
| `info` | Object | Car performance stats |

**Track Timing Structure** (`times.{track}_{condition}`):

| Field | Type | Description |
|-------|------|-------------|
| `t` | Number | Time in seconds |
| `u` | String | Username who submitted the time |
| `down` | Array | Usernames who downvoted this time (optional) |
| `d` | String | Date/time of submission (optional) |

**Usage in Application**:
- Processed by `parse_car_times()` function
- Each track entry becomes a `BestCarRecord`
- Downvoted times are filtered out for quality
- Times below 1.2 seconds are considered invalid

### 4. Tracks Database (`tracks_repo.json`)

**Purpose**: Defines all tracks and their properties in Top Drives.

**Source**: Downloaded from `https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/tracks_repo.json`

**Example Structure**:
```json
[
  {
    "id": 1,
    "name": "Laguna Seca",
    "conditions": ["00", "01", "10"],
    "type": "circuit"
  }
]
```

**Field Descriptions**:

| Field | Type | Description |
|-------|------|-------------|
| `id` | Number | Unique track identifier |
| `name` | String | Human-readable track name |
| `conditions` | Array | Supported surface conditions |
| `type` | String | Track category (circuit, drag, rally, etc.) |

**Usage in Application**:
- Provides track name mappings for output
- Used for track-specific analysis queries
- Supports condition filtering

### 5. Track Names Configuration (`track_names.json`)

**Purpose**: Maps internal track codes to human-readable track names.

**Source**: Included in the repository, loaded at compile time via `include_str!`

**Structure**:
```json
{
  "laguna": "Laguna Seca",
  "monaco": "Monaco Streets",
  "testBowl": "Test Bowl",
  "dealsGap": "Deals Gap"
}
```

**Field Descriptions**:

| Field | Type | Description |
|-------|------|-------------|
| *key* | String | Internal track code (e.g., `"laguna"`) |
| *value* | String | Human-readable track name (e.g., `"Laguna Seca"`) |

**Usage in Application**:
- Loaded by `translate_track_name()` in `src/translate.rs`
- Compiled into the binary at build time
- Fallback: Returns original track code if no mapping found

## Application Constants

### `MAX_RQ`

```rust
const MAX_RQ: u8 = 113;
```

**Purpose**: Defines the maximum RQ level used in analysis range calculations.

**Usage**: When evaluating car performance, each car is tested from its base RQ up to `MAX_RQ`. This replaces the former magic number 115, providing a named constant for clarity and easy adjustment.

**Location**: Defined in `src/records.rs`

## Data Processing Pipeline

### 1. Data Acquisition Phase

**Files Downloaded:**
1. `cars_repo.json` - Complete car database
2. `tracks_repo.json` - Track definitions
3. `times/{rid}.json` - Individual car timing data (1000+ files)

**Validation:**
- HTTP response codes checked
- JSON parsing errors handled gracefully
- File corruption detection

### 2. Data Parsing Phase

**Car Database Processing:**
```rust
// Parse cars_repo.json — uses serde derive for deserialization
let cars: Vec<Car> = serde_json::from_str(&content)?;
// Falls back to manual Value-based parsing if serde deserialization fails
// (for backward compatibility with older JSON formats)
```

**Timing Data Processing:**
```rust
// Parse individual car timing file
let json_data: Value = serde_json::from_str(&file_content)?;
if let Some(configurations) = json_data["data"].as_object() {
    for (raw_config, entry) in configurations {
        // Strip version prefix (e.g., "v2c_" from "v2c_233")
        let config = raw_config
            .strip_prefix("v2c_")
            .or_else(|| raw_config.strip_prefix("v1c_"))
            .unwrap_or(raw_config);
        if let Some(times) = entry["times"].as_object() {
            for (track_id, details) in times {
                // Process each track time
            }
        }
    }
}
```

### 3. Data Transformation Phase

**Track ID Parsing:**
- Input: `"laguna_a00"` (track + condition)
- Process: Split on `"_a"` delimiter
- Output: `("laguna", "00")` (track name, condition code)

**Track Name Translation:**
- Track codes are looked up in `track_names.json` (compiled into binary)
- Example: `"laguna"` → `"Laguna Seca"`
- Fallback: Unknown codes returned as-is

**Condition Code Translation:**
- `"00"` → `"Dry"`
- `"01"` → `"Wet"`
- `"10"` → `"Dirt"`
- `"11"` → `"Dirt Wet"`

**Time Validation:**
1. Check for downvotes: `details["down"].as_array()`
2. Validate time range: `time > 1.2`
3. Special handling: Test Bowl inversion `2000.0 / time`

### 4. Data Aggregation Phase

**Best Car Records Creation:**
```rust
BestCarRecord {
    car_rid: car.rid.clone(),
    brand: car.brand.clone(),
    only_name: car.only_name.clone(),
    year: car.year,
    rq: car.rq,
    config: config.clone(),
    normalized_config: normalize_to_final_build(&config),
    time,
    owned: my_cars.contains(&car.guid),
    prize: car.prize,
    guid: car.guid.clone(),
}
```

**RQ Range Analysis:**
- Test each car from its base RQ up to `MAX_RQ` (113)
- Compare against existing best times using `DashMap` entry API
- Update records if better performance found
- Uses `DashMap` for fine-grained concurrent access (no global lock)

## Data Quality Measures

### Validation Rules

1. **Time Thresholds**:
   - Minimum time: 1.2 seconds
   - Test Bowl: Inverted calculation (2000.0 / time)

2. **Community Validation**:
   - Downvoted times excluded
   - Multiple submissions required for reliability

3. **Data Consistency**:
   - Track name normalization
   - Condition code standardization
   - Car identifier validation

### Error Handling

**Network Errors**:
- Failed downloads logged but don't stop processing
- Existing cached data used as fallback
- Clear error messages for user

**Parse Errors**:
- Malformed JSON handled gracefully
- Missing fields use default values
- Invalid times filtered out

**File System Errors**:
- Missing files reported clearly
- Permission issues identified
- Corrupt files detected and skipped

## Performance Considerations

### Memory Usage

**Large Data Structures**:
- `AnalysisContext.best_cars` (`DashMap<TripletKey, Vec<BestCarRecord>>`): Contains all best records across RQ ranges
- Individual timing files: Loaded and parsed sequentially via `spawn_blocking`
- User collection: Small, loaded once

**Optimization Strategies**:
- Concurrent processing of car files with bounded concurrency (semaphore limit of 20)
- Rate-limited downloads (20 concurrent max)
- Streaming JSON parsing where possible

### Processing Speed

**Bottlenecks**:
1. **Network I/O**: 1000+ small files downloaded
2. **JSON Parsing**: Each timing file parsed individually
3. **HashMap Operations**: Frequent insertions with RQ range checking

**Optimizations Applied**:
1. **Bounded Concurrency**: Semaphore limits concurrent operations to 20
2. **Connection Pooling**: Reuse HTTP connections
3. **Early Filtering**: Invalid times rejected quickly

## Data Evolution

### Version Compatibility

**Backwards Compatibility**:
- Old timing files still parseable
- New fields added gracefully
- Deprecated fields handled

**Schema Evolution**:
- New track additions supported
- Condition code extensions possible
- Car metadata fields expandable

### Update Strategy

**Incremental Updates**:
- Only changed cars need re-downloading
- Track database updated periodically
- User collection refreshed as needed

**Data Freshness**:
- Community timing data updates frequently
- Car database relatively stable
- Track definitions rarely change

## Troubleshooting Data Issues

### Common Problems

**Missing Timing Data**:
- **Symptom**: Cars show no performance data
- **Cause**: Timing file not downloaded or corrupted
- **Solution**: Re-run `fetch` command

**Incorrect Ownership**:
- **Symptom**: Wrong cars marked as owned/missing
- **Cause**: Outdated `my_cars.json` export
- **Solution**: Re-export from Top Drives game

**Track Name Issues**:
- **Symptom**: Unknown track codes in output
- **Cause**: Missing mapping in `track_names.json`
- **Solution**: Add new mapping to `track_names.json`

### Data Validation Tools

**Integrity Checks**:
1. **File Count**: Verify expected number of timing files
2. **JSON Validity**: Parse all files for syntax errors
3. **Reference Integrity**: Cross-check car and track references

**Diagnostic Commands**:
```bash
# Check for missing timing files
ls times/ | wc -l  # Should be 1000+

# Validate JSON syntax
find times/ -name "*.json" -exec jq empty {} \;

# Check for common issues
grep -r "No RID" times/
```

## Future Data Structure Enhancements

### Potential Improvements

1. **Compression**: Gzip timing files to reduce storage
2. **Indexing**: Add database indexes for faster queries
3. **Caching**: Cache parsed data to avoid re-processing
4. **Incremental Updates**: Only download changed files

### Schema Extensions

**Additional Fields**:
- Car weight and dimensions
- Performance curves
- Weather condition effects
- Driver skill impact

**New Analysis Types**:
- Historical performance trends
- Meta change impact analysis
- Community submission quality metrics
- Regional performance differences