Cache Management¶

Referenced Files in This Document

Table of Contents¶

Introduction
Project Structure
Core Components
Architecture Overview
Detailed Component Analysis
Dependency Analysis
Performance Considerations
Troubleshooting Guide
Conclusion
Appendices

Introduction¶

This document explains the JSON-based persistent caching strategy used by Nexo to store application state, including decisions, history, status information, and Jellyfin integration state. It covers cache file structures, data formats, serialization patterns, invalidation strategies, consistency mechanisms, recovery procedures, performance considerations, confidence scoring cache usage, troubleshooting, backup, and migration guidance.

Project Structure¶

The cache layer is implemented as a set of JSON files under the cache directory. These files are read and written by service routes and core modules that orchestrate scanning, classification, decision-making, and Jellyfin synchronization. The service entry points initialize logging and expose HTTP endpoints that interact with the cache through database adapters and use cases.

graph TB

subgraph "Cache Layer"

C_dec["cache/decisions.json"]

C_hist["cache/history.json"]

C_stat["cache/status.json"]

C_jf["cache/jellyfin_state.json"]

end

subgraph "Service Routes"

R_stat["routes/status.py"]

R_dec["routes/decisions.py"]

R_hist["routes/history.py"]

R_jf["routes/jellyfin.py"]

end

subgraph "Core Modules"

U_conf["core/confidence.py"]

U_use["core/usecases.py"]

U_pipe["core/pipeline.py"]

U_scan["core/scan.py"]

U_match["core/matcher.py"]

U_org["core/organizer.py"]

U_doc["core/document_parser.py"]

U_img["core/image_classifier.py"]

end

subgraph "Database Adapters"

D_init["database/__init__.py"]

D_dec["database/decisions_db.py"]

D_hist["database/history_db.py"]

D_log["database/logs_db.py"]

D_user["database/users_db.py"]

end

subgraph "App Entry"

A_main["service/main.py"]

A_app["service/app.py"]

A_deps["service/deps.py"]

end

subgraph "Config and Utils"

CFG["config.py"]

CST["constants.py"]

UT["utils.py"]

LOG["logger.py"]

end

R_stat --> C_stat

R_dec --> C_dec

R_hist --> C_hist

R_jf --> C_jf

R_dec --> D_dec

R_hist --> D_hist

R_stat --> D_log

R_jf --> D_user

U_pipe --> U_scan

U_pipe --> U_match

U_pipe --> U_org

U_pipe --> U_doc

U_pipe --> U_img

U_pipe --> U_conf

U_pipe --> U_use

U_conf --> C_dec

U_use --> C_dec

U_use --> C_hist

U_use --> C_stat

U_use --> C_jf

A_main --> A_app

A_app --> R_stat

A_app --> R_dec

A_app --> R_hist

A_app --> R_jf

A_app --> D_init

A_app --> LOG

A_app --> CFG

A_app --> CST

A_app --> UT

Diagram sources

Section sources

Core Components¶

Cache files:
decisions.json: Stores user or system decisions for media items.
history.json: Records historical actions and pipeline events.
status.json: Holds current application status and runtime metrics.
jellyfin_state.json: Persists Jellyfin integration state (e.g., connection, last sync metadata).
Service routes:
Status, Decisions, History, and Jellyfin endpoints read/write these files via database adapters and use cases.
Core modules:
Confidence scoring, matching, organization, parsing, and scanning routines update cache state and decisions based on processing results.
Database adapters:
Provide structured access to JSON-backed persistence for decisions, history, logs, and users.

Key responsibilities:

Serialize/deserialize Python objects to/from JSON safely.
Ensure atomic writes to prevent partial updates.
Maintain consistent schema across versions.
Log all cache mutations for auditability.

Section sources

Architecture Overview¶

The cache architecture follows a layered approach:

API layer exposes endpoints for reading/writing cache state.
Use case layer orchestrates business logic and coordinates cache updates.
Persistence layer serializes objects into JSON files with robust error handling.
Configuration and utilities provide paths, constants, and helpers for safe I/O.

sequenceDiagram

participant Client as "Client"

participant Route as "Service Route"

participant UseCase as "Use Case"

participant Adapter as "DB Adapter"

participant FS as "File System"

Client->>Route : "HTTP request"

Route->>UseCase : "Invoke operation"

UseCase->>Adapter : "Read/Write cache"

Adapter->>FS : "Serialize/Deserialize JSON"

FS-->>Adapter : "Data payload"

Adapter-->>UseCase : "Result"

UseCase-->>Route : "Response model"

Route-->>Client : "HTTP response"

Diagram sources

Detailed Component Analysis¶

Cache File Structures and Data Formats¶

decisions.json
Purpose: Persist decisions for media items (e.g., accept/reject, categorization).
Typical structure: Array or map keyed by item identifiers; each entry includes decision metadata such as timestamp, source, and confidence score.
Serialization: Python lists/dicts serialized to JSON arrays/objects.
history.json
Purpose: Record chronological events from scanning, classification, and user interactions.
Typical structure: Array of event records with fields like action type, target ID, result, and timestamps.
Serialization: Append-only pattern with periodic compaction recommended.
status.json
Purpose: Current application status, counters, and runtime flags.
Typical structure: Object with keys for scan progress, last run time, error counts, and feature toggles.
Serialization: Atomic overwrite to avoid partial reads.
jellyfin_state.json
Purpose: Integration state for Jellyfin (connection info, last sync details, tokens if applicable).
Typical structure: Object containing server URL, authentication markers, last sync timestamp, and error summaries.
Serialization: Sensitive fields should be stored securely; consider encryption at rest in future iterations.

Best practices:

Validate schemas on load; reject malformed entries.
Normalize IDs and timestamps consistently.
Keep field names stable across versions; add new fields without breaking old readers.

Section sources

Serialization Patterns and Consistency¶

Atomic writes:
Write to a temporary file and rename to target path to ensure consistency.
Error handling:
Catch IO and JSON errors; log failures and fallback to in-memory state when necessary.
Schema validation:
On load, validate expected keys and types; coerce or drop unknown fields gracefully.
Concurrency:
Avoid concurrent writes to the same file; serialize operations per-file using locks or single-writer threads.

Implementation touchpoints:

Database adapters encapsulate read/write semantics and enforce validation.
Use cases coordinate multiple cache updates within transactions-like sequences.

Section sources

Cache Invalidation Strategies¶

Time-based invalidation:
Status counters reset after successful scans; last sync timestamps updated upon completion.
Event-driven invalidation:
Decisions invalidated when underlying media metadata changes; history appended but not mutated.
TTL policies:
Optional TTL for transient status flags; expired entries pruned during maintenance.
Versioning:
Include cache version metadata; upgrade routines migrate older formats to current schema.

Operational triggers:

Endpoints for decisions, history, and Jellyfin state trigger targeted invalidations.
Pipeline stages may clear stale caches before reprocessing.

Section sources

Recovery Procedures for Corrupted Cache Files¶

Detect corruption:
Attempt to parse JSON; catch exceptions and mark file as corrupted.
Fallback behavior:
Load empty defaults; continue operation with degraded functionality.
Repair workflow:
Backup corrupted file; attempt to salvage valid entries by parsing line-by-line or chunk-by-chunk.
Rebuild indices from history if possible.
Audit trail:
Log recovery steps and outcomes for post-mortem analysis.

Recommended safeguards:

Periodic backups of cache directory.
Checksums or hashes for critical files to detect silent corruption.

Section sources

Confidence Scoring Cache and AI Classification Performance¶

Role:
Confidence scores influence decision thresholds and reduce redundant AI calls by caching results for identical inputs.
Storage:
Scores persisted alongside decisions or in a dedicated cache section within decisions.json.
Update policy:
Recompute when input features change; otherwise reuse cached values.
Impact:
Improves throughput and reduces external API costs; balances accuracy with latency.

Integration points:

Confidence module computes and persists scores.
Decision and matcher modules consult cached scores to accelerate classification.

Section sources

Jellyfin Integration State Management¶

State contents:
Server connectivity, last sync metadata, error summaries, and optional credentials references.
Lifecycle:
Initialize on startup; update on successful sync; persist failures for retry logic.
Consistency:
Ensure state reflects actual Jellyfin interactions; reconcile discrepancies during next sync cycle.

Section sources

Dependency Analysis¶

The following diagram maps key dependencies between routes, adapters, core modules, and cache files.

graph LR

R_dec["routes/decisions.py"] --> D_dec["database/decisions_db.py"]

R_hist["routes/history.py"] --> D_hist["database/history_db.py"]

R_stat["routes/status.py"] --> D_log["database/logs_db.py"]

R_jf["routes/jellyfin.py"] --> D_user["database/users_db.py"]

D_dec --> C_dec["cache/decisions.json"]

D_hist --> C_hist["cache/history.json"]

D_log --> C_stat["cache/status.json"]

D_user --> C_jf["cache/jellyfin_state.json"]

U_conf["core/confidence.py"] --> C_dec

U_use["core/usecases.py"] --> C_dec

U_use --> C_hist

U_use --> C_stat

U_use --> C_jf

U_pipe["core/pipeline.py"] --> U_scan["core/scan.py"]

U_pipe --> U_match["core/matcher.py"]

U_pipe --> U_org["core/organizer.py"]

U_pipe --> U_doc["core/document_parser.py"]

U_pipe --> U_img["core/image_classifier.py"]

Diagram sources

Section sources

Performance Considerations¶

Cache size limits:
Enroll maximum sizes for decisions and history; prune oldest entries when thresholds exceeded.
Cleanup policies:
Periodic compaction of history; remove obsolete status flags; archive old decisions.
Memory optimization:
Stream large payloads; avoid loading entire files into memory when unnecessary.
I/O efficiency:
Batch writes; minimize lock contention; prefer append-only patterns for history.
Indexing:
Maintain lightweight indexes (in-memory) for frequent lookups; rebuild on startup if needed.

[No sources needed since this section provides general guidance]

Troubleshooting Guide¶

Common issues and resolutions:

JSON parse errors:
Symptoms: Endpoint returns 500; logs show JSON decode exceptions.
Actions: Validate file integrity; restore from backup; repair by extracting valid segments.
Stale status:
Symptoms: Dashboard shows outdated counters.
Actions: Trigger status refresh endpoint; verify write permissions; check disk space.
Jellyfin sync failures:
Symptoms: Last sync timestamp not updating; repeated errors.
Actions: Inspect jellyfin_state.json; verify network connectivity; reset credentials reference; retry sync.
Decision inconsistencies:
Symptoms: Conflicting decisions for same item.
Actions: Review history for conflicting events; deduplicate by timestamp; rebuild decision index.

Backup procedures:

Snapshot the entire cache directory before upgrades or major operations.
Store backups offsite or in versioned storage.

Migration strategies:

Increment cache version on schema changes.
Implement migration scripts to transform older formats to current schema.
Test migrations against sample datasets prior to deployment.

Section sources

Conclusion¶

Nexo’s JSON-based cache provides a simple yet effective persistence mechanism for decisions, history, status, and Jellyfin integration state. By enforcing atomic writes, schema validation, and clear invalidation policies, the system maintains consistency and resilience. Confidence scoring caching enhances AI classification performance while reducing redundant computations. Robust troubleshooting, backup, and migration strategies ensure long-term reliability and ease of maintenance.

[No sources needed since this section summarizes without analyzing specific files]

Appendices¶

Appendix A: Cache File Reference¶

decisions.json
Key fields: item_id, decision, timestamp, confidence_score, source.
history.json
Key fields: event_type, target_id, result, timestamp, metadata.
status.json
Key fields: last_scan_time, scan_progress, error_count, feature_flags.
jellyfin_state.json
Key fields: server_url, last_sync_at, auth_ref, error_summary.

Section sources

Appendix B: Operational Checklist¶

Verify cache directory permissions.
Monitor disk usage and enforce cleanup policies.
Schedule regular backups.
Validate cache versions after upgrades.
Review logs for cache-related errors.

Section sources