top_drives/docs/DATA_STRUCTURES.md

JSON Data Structures Documentation

Overview

This document describes the JSON data structures used by the Top Drives Records Analyzer, including their purpose, format, and usage within the application.

Core Data Files

1. Car Database (cars_repo.json)

Purpose: Master database of all cars in Top Drives with their metadata.

Source: Downloaded from https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/cars_final.json

Structure:

[
  {
    "rid": "Ferrari_F40_1987",
    "brand": "Ferrari",
    "onlyName": "F40",
    "year": 1987,
    "rq": 95,
    "prize": false,
    "guid": "card_ferrari_f40_1987"
  }
]

Field Descriptions:

Field	Type	Description
rid	String	Repository identifier (unique)
brand	String	Car manufacturer/brand name
onlyName	String	Car model name (without brand)
year	Number	Model year
rq	Number	Rarity level (10-113)
prize	Boolean	Whether car is prize-only
guid	String	Game-specific card identifier

Usage in Application:

• Loaded by get_car_list() function
• Used to identify cars for timing file processing
• Provides metadata for analysis output

2. User's Car Collection (my_cars.json)

Purpose: Contains the user's owned cars exported from the Top Drives game.

Source: Exported from Top Drives game (Settings → Account → Export Data)

Structure:

{
  "playerDeck": [
    {
      "cardId": "card_ferrari_f40_1987",
      "locked": true
    },
    {
      "cardId": "card_mclaren_f1_1993",
      "locked": false
    }
  ]
}

Field Descriptions:

Field	Type	Description
playerDeck	Array	List of user's cards
cardId	String	Unique card identifier
locked	Boolean	Whether card is owned (true = owned)

Processing Logic:

• Only locked: true cards are considered owned
• cardId values are matched against car database GUIDs
• Used to mark cars as "owned" in analysis results

3. Individual Car Timing Files (times/{rid}.json)

Purpose: Contains detailed timing data for a specific car across all tracks and conditions.

Source: Downloaded from https://api.topdrivesrecords.com/car/{rid}

Example Structure:

{
  "users": ["Player1", "Player2"],
  "data": {
    "111": {
      "times": {
        "laguna_a00": {
          "t": 112.45,
          "u": "Player1"
        },
        "monaco_a00": {
          "t": 45.23,
          "u": "Player2",
          "down": ["Player3"]
        }
      },
      "info": {
        "topSpeed": "201",
        "acel": "3.2",
        "hand": "92"
      }
    }
  },
  "rid": "Ferrari_F40_1987"
}

Field Descriptions:

Field	Type	Description
users	Array	List of players who submitted times
data	Object	Timing data organized by configuration
rid	String	Car repository identifier

Configuration Structure (data.{config}):

Field	Type	Description
times	Object	Track timing data
info	Object	Car performance stats

Track Timing Structure (times.{track}_{condition}):

Field	Type	Description
t	Number	Time in seconds
u	String	Username who submitted the time
down	Array	Usernames who downvoted this time (optional)
d	String	Date/time of submission (optional)

Usage in Application:

• Processed by parse_car_times() function
• Each track entry becomes a BestCarRecord
• Downvoted times are filtered out for quality
• Times below 1.2 seconds are considered invalid

4. Tracks Database (tracks_repo.json)

Purpose: Defines all tracks and their properties in Top Drives.

Source: Downloaded from https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/tracks_repo.json

Example Structure:

[
  {
    "id": 1,
    "name": "Laguna Seca",
    "conditions": ["00", "01", "10"],
    "type": "circuit"
  }
]

Field Descriptions:

Field	Type	Description
id	Number	Unique track identifier
name	String	Human-readable track name
conditions	Array	Supported surface conditions
type	String	Track category (circuit, drag, rally, etc.)

Usage in Application:

• Provides track name mappings for output
• Used for track-specific analysis queries
• Supports condition filtering

5. Track Names Configuration (track_names.json)

Purpose: Maps internal track codes to human-readable track names.

Source: Included in the repository, loaded at compile time via include_str!

Structure:

{
  "laguna": "Laguna Seca",
  "monaco": "Monaco Streets",
  "testBowl": "Test Bowl",
  "dealsGap": "Deals Gap"
}

Field Descriptions:

Field	Type	Description
key	String	Internal track code (e.g., "laguna")
value	String	Human-readable track name (e.g., "Laguna Seca")

Usage in Application:

• Loaded by translate_track_name() in src/translate.rs
• Compiled into the binary at build time
• Fallback: Returns original track code if no mapping found

Application Constants

MAX_RQ

const MAX_RQ: u8 = 113;

Purpose: Defines the maximum RQ level used in analysis range calculations.

Usage: When evaluating car performance, each car is tested from its base RQ up to MAX_RQ. This replaces the former magic number 115, providing a named constant for clarity and easy adjustment.

Location: Defined in src/records.rs

Data Processing Pipeline

1. Data Acquisition Phase

Files Downloaded:

1. cars_repo.json - Complete car database
2. tracks_repo.json - Track definitions
3. times/{rid}.json - Individual car timing data (1000+ files)

Validation:

• HTTP response codes checked
• JSON parsing errors handled gracefully
• File corruption detection

2. Data Parsing Phase

Car Database Processing:

// Parse cars_repo.json — uses serde derive for deserialization
let cars: Vec<Car> = serde_json::from_str(&content)?;
// Falls back to manual Value-based parsing if serde deserialization fails
// (for backward compatibility with older JSON formats)

Timing Data Processing:

// Parse individual car timing file
let json_data: Value = serde_json::from_str(&file_content)?;
if let Some(configurations) = json_data["data"].as_object() {
    for (config, entry) in configurations {
        if let Some(times) = entry["times"].as_object() {
            for (track_id, details) in times {
                // Process each track time
            }
        }
    }
}

3. Data Transformation Phase

Track ID Parsing:

• Input: "laguna_a00" (track + condition)
• Process: Split on "_a" delimiter
• Output: ("laguna", "00") (track name, condition code)

Track Name Translation:

• Track codes are looked up in track_names.json (compiled into binary)
• Example: "laguna" → "Laguna Seca"
• Fallback: Unknown codes returned as-is

Condition Code Translation:

• "00" → "Dry"
• "01" → "Wet"
• "10" → "Dirt"
• "11" → "Dirt Wet"

Time Validation:

1. Check for downvotes: details["down"].as_array()
2. Validate time range: time > 1.2
3. Special handling: Test Bowl inversion 2000.0 / time

4. Data Aggregation Phase

Best Car Records Creation:

BestCarRecord {
    car_rid: car.rid.clone(),
    brand: car.brand.clone(),
    only_name: car.only_name.clone(),
    year: car.year,
    rq: car.rq,
    config: config.clone(),
    time,
    owned: my_cars.contains(&car.guid),
    guid: car.guid.clone(),
}

RQ Range Analysis:

• Test each car from its base RQ up to MAX_RQ (113)
• Compare against existing best times using DashMap entry API
• Update records if better performance found
• Uses DashMap for fine-grained concurrent access (no global lock)

Data Quality Measures

Validation Rules

1. Time Thresholds:

   • Minimum time: 1.2 seconds
   • Test Bowl: Inverted calculation (2000.0 / time)

2. Community Validation:

   • Downvoted times excluded
   • Multiple submissions required for reliability

3. Data Consistency:

   • Track name normalization
   • Condition code standardization
   • Car identifier validation

Error Handling

Network Errors:

• Failed downloads logged but don't stop processing
• Existing cached data used as fallback
• Clear error messages for user

Parse Errors:

• Malformed JSON handled gracefully
• Missing fields use default values
• Invalid times filtered out

File System Errors:

• Missing files reported clearly
• Permission issues identified
• Corrupt files detected and skipped

Performance Considerations

Memory Usage

Large Data Structures:

• AnalysisContext.best_cars (DashMap<TripletKey, Vec<BestCarRecord>>): Contains all best records across RQ ranges
• Individual timing files: Loaded and parsed sequentially via spawn_blocking
• User collection: Small, loaded once

Optimization Strategies:

• Concurrent processing of car files with bounded concurrency (semaphore limit of 20)
• Rate-limited downloads (20 concurrent max)
• Streaming JSON parsing where possible

Processing Speed

Bottlenecks:

1. Network I/O: 1000+ small files downloaded
2. JSON Parsing: Each timing file parsed individually
3. HashMap Operations: Frequent insertions with RQ range checking

Optimizations Applied:

1. Bounded Concurrency: Semaphore limits concurrent operations to 20
2. Connection Pooling: Reuse HTTP connections
3. Early Filtering: Invalid times rejected quickly

Data Evolution

Version Compatibility

Backwards Compatibility:

• Old timing files still parseable
• New fields added gracefully
• Deprecated fields handled

Schema Evolution:

• New track additions supported
• Condition code extensions possible
• Car metadata fields expandable

Update Strategy

Incremental Updates:

• Only changed cars need re-downloading
• Track database updated periodically
• User collection refreshed as needed

Data Freshness:

• Community timing data updates frequently
• Car database relatively stable
• Track definitions rarely change

Troubleshooting Data Issues

Common Problems

Missing Timing Data:

• Symptom: Cars show no performance data
• Cause: Timing file not downloaded or corrupted
• Solution: Re-run fetch command

Incorrect Ownership:

• Symptom: Wrong cars marked as owned/missing
• Cause: Outdated my_cars.json export
• Solution: Re-export from Top Drives game

Track Name Issues:

• Symptom: Unknown track codes in output
• Cause: Missing mapping in track_names.json
• Solution: Add new mapping to track_names.json

Data Validation Tools

Integrity Checks:

1. File Count: Verify expected number of timing files
2. JSON Validity: Parse all files for syntax errors
3. Reference Integrity: Cross-check car and track references

Diagnostic Commands:

# Check for missing timing files
ls times/ | wc -l  # Should be 1000+

# Validate JSON syntax
find times/ -name "*.json" -exec jq empty {} \;

# Check for common issues
grep -r "No RID" times/

Future Data Structure Enhancements

Potential Improvements

1. Compression: Gzip timing files to reduce storage
2. Indexing: Add database indexes for faster queries
3. Caching: Cache parsed data to avoid re-processing
4. Incremental Updates: Only download changed files

Schema Extensions

Additional Fields:

• Car weight and dimensions
• Performance curves
• Weather condition effects
• Driver skill impact

New Analysis Types:

• Historical performance trends
• Meta change impact analysis
• Community submission quality metrics
• Regional performance differences