Added documentation
Added test cases Added comments
This commit is contained in:
parent
7a8d9beedc
commit
32c4d3c10e
10 changed files with 3852 additions and 63 deletions
407
docs/DATA_STRUCTURES.md
Normal file
407
docs/DATA_STRUCTURES.md
Normal file
|
|
@ -0,0 +1,407 @@
|
|||
# JSON Data Structures Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the JSON data structures used by the Top Drives Records Analyzer, including their purpose, format, and usage within the application.
|
||||
|
||||
## Core Data Files
|
||||
|
||||
### 1. Car Database (`cars_repo.json`)
|
||||
|
||||
**Purpose**: Master database of all cars in Top Drives with their metadata.
|
||||
|
||||
**Source**: Downloaded from `https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/cars_final.json`
|
||||
|
||||
**Structure**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"rid": "Ferrari_F40_1987",
|
||||
"brand": "Ferrari",
|
||||
"onlyName": "F40",
|
||||
"year": 1987,
|
||||
"rq": 95,
|
||||
"prize": false,
|
||||
"guid": "card_ferrari_f40_1987"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Field Descriptions**:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `rid` | String | Repository identifier (unique) |
|
||||
| `brand` | String | Car manufacturer/brand name |
|
||||
| `onlyName` | String | Car model name (without brand) |
|
||||
| `year` | Number | Model year |
|
||||
| `rq` | Number | Rarity level (10-113) |
|
||||
| `prize` | Boolean | Whether car is prize-only |
|
||||
| `guid` | String | Game-specific card identifier |
|
||||
|
||||
**Usage in Application**:
|
||||
- Loaded by `get_car_list()` function
|
||||
- Used to identify cars for timing file processing
|
||||
- Provides metadata for analysis output
|
||||
|
||||
### 2. User's Car Collection (`my_cars.json`)
|
||||
|
||||
**Purpose**: Contains the user's owned cars exported from the Top Drives game.
|
||||
|
||||
**Source**: Exported from Top Drives game (Settings → Account → Export Data)
|
||||
|
||||
**Structure**:
|
||||
```json
|
||||
{
|
||||
"playerDeck": [
|
||||
{
|
||||
"cardId": "card_ferrari_f40_1987",
|
||||
"locked": true
|
||||
},
|
||||
{
|
||||
"cardId": "card_mclaren_f1_1993",
|
||||
"locked": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Field Descriptions**:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `playerDeck` | Array | List of user's cards |
|
||||
| `cardId` | String | Unique card identifier |
|
||||
| `locked` | Boolean | Whether card is owned (true = owned) |
|
||||
|
||||
**Processing Logic**:
|
||||
- Only `locked: true` cards are considered owned
|
||||
- `cardId` values are matched against car database GUIDs
|
||||
- Used to mark cars as "owned" in analysis results
|
||||
|
||||
### 3. Individual Car Timing Files (`times/{rid}.json`)
|
||||
|
||||
**Purpose**: Contains detailed timing data for a specific car across all tracks and conditions.
|
||||
|
||||
**Source**: Downloaded from `https://api.topdrivesrecords.com/car/{rid}`
|
||||
|
||||
**Example Structure**:
|
||||
```json
|
||||
{
|
||||
"users": ["Player1", "Player2"],
|
||||
"data": {
|
||||
"111": {
|
||||
"times": {
|
||||
"laguna_a00": {
|
||||
"t": 112.45,
|
||||
"u": "Player1"
|
||||
},
|
||||
"monaco_a00": {
|
||||
"t": 45.23,
|
||||
"u": "Player2",
|
||||
"down": ["Player3"]
|
||||
}
|
||||
},
|
||||
"info": {
|
||||
"topSpeed": "201",
|
||||
"acel": "3.2",
|
||||
"hand": "92"
|
||||
}
|
||||
}
|
||||
},
|
||||
"rid": "Ferrari_F40_1987"
|
||||
}
|
||||
```
|
||||
|
||||
**Field Descriptions**:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `users` | Array | List of players who submitted times |
|
||||
| `data` | Object | Timing data organized by configuration |
|
||||
| `rid` | String | Car repository identifier |
|
||||
|
||||
**Configuration Structure** (`data.{config}`):
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `times` | Object | Track timing data |
|
||||
| `info` | Object | Car performance stats |
|
||||
|
||||
**Track Timing Structure** (`times.{track}_{condition}`):
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `t` | Number | Time in seconds |
|
||||
| `u` | String | Username who submitted the time |
|
||||
| `down` | Array | Usernames who downvoted this time (optional) |
|
||||
| `d` | String | Date/time of submission (optional) |
|
||||
|
||||
**Usage in Application**:
|
||||
- Processed by `parse_car_times()` function
|
||||
- Each track entry becomes a `BestCarRecord`
|
||||
- Downvoted times are filtered out for quality
|
||||
- Times below 1.2 seconds are considered invalid
|
||||
|
||||
### 4. Tracks Database (`tracks_repo.json`)
|
||||
|
||||
**Purpose**: Defines all tracks and their properties in Top Drives.
|
||||
|
||||
**Source**: Downloaded from `https://github.com/TiagoXavi/topdrives-records/raw/refs/heads/main/src/database/tracks_repo.json`
|
||||
|
||||
**Example Structure**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"name": "Laguna Seca",
|
||||
"conditions": ["00", "01", "10"],
|
||||
"type": "circuit"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Field Descriptions**:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `id` | Number | Unique track identifier |
|
||||
| `name` | String | Human-readable track name |
|
||||
| `conditions` | Array | Supported surface conditions |
|
||||
| `type` | String | Track category (circuit, drag, rally, etc.) |
|
||||
|
||||
**Usage in Application**:
|
||||
- Provides track name mappings for output
|
||||
- Used for track-specific analysis queries
|
||||
- Supports condition filtering
|
||||
|
||||
## Data Processing Pipeline
|
||||
|
||||
### 1. Data Acquisition Phase
|
||||
|
||||
**Files Downloaded:**
|
||||
1. `cars_repo.json` - Complete car database
|
||||
2. `tracks_repo.json` - Track definitions
|
||||
3. `times/{rid}.json` - Individual car timing data (1000+ files)
|
||||
|
||||
**Validation:**
|
||||
- HTTP response codes checked
|
||||
- JSON parsing errors handled gracefully
|
||||
- File corruption detection
|
||||
|
||||
### 2. Data Parsing Phase
|
||||
|
||||
**Car Database Processing:**
|
||||
```rust
|
||||
// Parse cars_repo.json
|
||||
let cars_data: Vec<Value> = serde_json::from_str(&file_content)?;
|
||||
for car in cars_data {
|
||||
let car = Car {
|
||||
rid: car["rid"].as_str().unwrap_or(""),
|
||||
brand: car["brand"].as_str().unwrap_or(""),
|
||||
// ... other fields
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
**Timing Data Processing:**
|
||||
```rust
|
||||
// Parse individual car timing file
|
||||
let json_data: Value = serde_json::from_str(&file_content)?;
|
||||
if let Some(configurations) = json_data["data"].as_object() {
|
||||
for (config, entry) in configurations {
|
||||
if let Some(times) = entry["times"].as_object() {
|
||||
for (track_id, details) in times {
|
||||
// Process each track time
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Data Transformation Phase
|
||||
|
||||
**Track ID Parsing:**
|
||||
- Input: `"laguna_a00"` (track + condition)
|
||||
- Process: Split on `"_a"` delimiter
|
||||
- Output: `("laguna", "00")` (track name, condition code)
|
||||
|
||||
**Condition Code Translation:**
|
||||
- `"00"` → `"Dry"`
|
||||
- `"01"` → `"Wet"`
|
||||
- `"10"` → `"Dirt"`
|
||||
- `"11"` → `"Dirt Wet"`
|
||||
|
||||
**Time Validation:**
|
||||
1. Check for downvotes: `details["down"].as_array()`
|
||||
2. Validate time range: `time > 1.2`
|
||||
3. Special handling: Test Bowl inversion `2000.0 / time`
|
||||
|
||||
### 4. Data Aggregation Phase
|
||||
|
||||
**Best Car Records Creation:**
|
||||
```rust
|
||||
BestCarRecord {
|
||||
car_rid: car.rid.clone(),
|
||||
brand: car.brand.clone(),
|
||||
only_name: car.only_name.clone(),
|
||||
year: car.year,
|
||||
rq: car.rq,
|
||||
config: config.clone(),
|
||||
time,
|
||||
owned: my_cars.contains(&car.guid),
|
||||
guid: car.guid.clone(),
|
||||
}
|
||||
```
|
||||
|
||||
**RQ Range Analysis:**
|
||||
- Test each car from its base RQ up to 115
|
||||
- Compare against existing best times
|
||||
- Update records if better performance found
|
||||
|
||||
## Data Quality Measures
|
||||
|
||||
### Validation Rules
|
||||
|
||||
1. **Time Thresholds**:
|
||||
- Minimum time: 1.2 seconds
|
||||
- Test Bowl: Inverted calculation (2000.0 / time)
|
||||
|
||||
2. **Community Validation**:
|
||||
- Downvoted times excluded
|
||||
- Multiple submissions required for reliability
|
||||
|
||||
3. **Data Consistency**:
|
||||
- Track name normalization
|
||||
- Condition code standardization
|
||||
- Car identifier validation
|
||||
|
||||
### Error Handling
|
||||
|
||||
**Network Errors**:
|
||||
- Failed downloads logged but don't stop processing
|
||||
- Existing cached data used as fallback
|
||||
- Clear error messages for user
|
||||
|
||||
**Parse Errors**:
|
||||
- Malformed JSON handled gracefully
|
||||
- Missing fields use default values
|
||||
- Invalid times filtered out
|
||||
|
||||
**File System Errors**:
|
||||
- Missing files reported clearly
|
||||
- Permission issues identified
|
||||
- Corrupt files detected and skipped
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Usage
|
||||
|
||||
**Large Data Structures**:
|
||||
- `BEST_CARS` HashMap: Contains all best records across RQ ranges
|
||||
- Individual timing files: Loaded and parsed sequentially
|
||||
- User collection: Small, loaded once
|
||||
|
||||
**Optimization Strategies**:
|
||||
- Concurrent processing of car files
|
||||
- Rate-limited downloads (20 concurrent max)
|
||||
- Streaming JSON parsing where possible
|
||||
|
||||
### Processing Speed
|
||||
|
||||
**Bottlenecks**:
|
||||
1. **Network I/O**: 1000+ small files downloaded
|
||||
2. **JSON Parsing**: Each timing file parsed individually
|
||||
3. **HashMap Operations**: Frequent insertions with RQ range checking
|
||||
|
||||
**Optimizations Applied**:
|
||||
1. **Concurrency**: One task per car file
|
||||
2. **Connection Pooling**: Reuse HTTP connections
|
||||
3. **Early Filtering**: Invalid times rejected quickly
|
||||
|
||||
## Data Evolution
|
||||
|
||||
### Version Compatibility
|
||||
|
||||
**Backwards Compatibility**:
|
||||
- Old timing files still parseable
|
||||
- New fields added gracefully
|
||||
- Deprecated fields handled
|
||||
|
||||
**Schema Evolution**:
|
||||
- New track additions supported
|
||||
- Condition code extensions possible
|
||||
- Car metadata fields expandable
|
||||
|
||||
### Update Strategy
|
||||
|
||||
**Incremental Updates**:
|
||||
- Only changed cars need re-downloading
|
||||
- Track database updated periodically
|
||||
- User collection refreshed as needed
|
||||
|
||||
**Data Freshness**:
|
||||
- Community timing data updates frequently
|
||||
- Car database relatively stable
|
||||
- Track definitions rarely change
|
||||
|
||||
## Troubleshooting Data Issues
|
||||
|
||||
### Common Problems
|
||||
|
||||
**Missing Timing Data**:
|
||||
- **Symptom**: Cars show no performance data
|
||||
- **Cause**: Timing file not downloaded or corrupted
|
||||
- **Solution**: Re-run `fetch` command
|
||||
|
||||
**Incorrect Ownership**:
|
||||
- **Symptom**: Wrong cars marked as owned/missing
|
||||
- **Cause**: Outdated `my_cars.json` export
|
||||
- **Solution**: Re-export from Top Drives game
|
||||
|
||||
**Track Name Issues**:
|
||||
- **Symptom**: Unknown track codes in output
|
||||
- **Cause**: Outdated track definitions
|
||||
- **Solution**: Update `tracks_repo.json`
|
||||
|
||||
### Data Validation Tools
|
||||
|
||||
**Integrity Checks**:
|
||||
1. **File Count**: Verify expected number of timing files
|
||||
2. **JSON Validity**: Parse all files for syntax errors
|
||||
3. **Reference Integrity**: Cross-check car and track references
|
||||
|
||||
**Diagnostic Commands**:
|
||||
```bash
|
||||
# Check for missing timing files
|
||||
ls times/ | wc -l # Should be 1000+
|
||||
|
||||
# Validate JSON syntax
|
||||
find times/ -name "*.json" -exec jq empty {} \;
|
||||
|
||||
# Check for common issues
|
||||
grep -r "No RID" times/
|
||||
```
|
||||
|
||||
## Future Data Structure Enhancements
|
||||
|
||||
### Potential Improvements
|
||||
|
||||
1. **Compression**: Gzip timing files to reduce storage
|
||||
2. **Indexing**: Add database indexes for faster queries
|
||||
3. **Caching**: Cache parsed data to avoid re-processing
|
||||
4. **Incremental Updates**: Only download changed files
|
||||
|
||||
### Schema Extensions
|
||||
|
||||
**Additional Fields**:
|
||||
- Car weight and dimensions
|
||||
- Performance curves
|
||||
- Weather condition effects
|
||||
- Driver skill impact
|
||||
|
||||
**New Analysis Types**:
|
||||
- Historical performance trends
|
||||
- Meta change impact analysis
|
||||
- Community submission quality metrics
|
||||
- Regional performance differences
|
||||
Loading…
Add table
Add a link
Reference in a new issue