File Scanning & Monitoring¶

Referenced Files in This Document

Table of Contents¶

Introduction
Project Structure
Core Components
Architecture Overview
Detailed Component Analysis
Dependency Analysis
Performance Considerations
Troubleshooting Guide
Conclusion
Appendices

Introduction¶

This document explains Nexo’s file scanning and monitoring system with a focus on Watchdog integration for real-time file system event detection. It covers how directories are monitored for new files, modifications, and deletions; scan configuration options; directory watching strategies; performance optimization techniques for large media libraries; the event processing pipeline from file system events to queue dispatching; concrete examples of scan configuration and custom watch patterns; and error handling strategies. It also provides guidance on memory management during large scans, handling permission errors, and recovery from file system failures.

Project Structure¶

The scanning and monitoring functionality is implemented across dedicated modules:

Real-time file system monitoring via Watchdog
Scan orchestration and configuration
Application wiring and dependency injection for background tasks

graph TB

subgraph "Monitoring"

WD["Watchdog Observer<br/>src/monitor/watchdog.py"]

end

subgraph "Scanning Core"

SCAN["Scan Orchestration<br/>src/core/scan.py"]

CFG["Configuration<br/>src/config.py"]

end

subgraph "Service Wiring"

MAIN["App Entry / Background Tasks<br/>apps/service/main.py"]

APP["FastAPI App Factory<br/>apps/service/app.py"]

DEPS["Dependencies (Queue, DB, etc.)<br/>apps/service/deps.py"]

end

WD --> SCAN

SCAN --> CFG

MAIN --> WD

MAIN --> SCAN

APP --> MAIN

DEPS --> MAIN

Diagram sources

Section sources

Core Components¶

Watchdog-based observer that detects file system events (create, modify, delete, move) within configured directories and routes them into an internal event pipeline.
Scan orchestrator that interprets events, applies filters and policies, and enqueues work items for downstream processing.
Configuration module that exposes scan settings such as watched paths, inclusion/exclusion patterns, batching behavior, and backoff/retry policies.
Service wiring that initializes the observer, starts background workers, and connects queues to processors.

Key responsibilities:

Event capture and normalization
Path filtering and pattern matching
Queue dispatching and deduplication
Error handling and retry/backoff
Performance tuning for large libraries

Section sources

Architecture Overview¶

The monitoring architecture follows an event-driven design:

Watchdog observes directories and emits normalized events.
The scanner applies policy rules and filters to decide whether to enqueue a task.
A queue decouples producers (observer/scanner) from consumers (processors).
Workers consume queued tasks and perform scanning/indexing operations.

sequenceDiagram

participant FS as "File System"

participant WD as "Watchdog Observer"

participant SC as "Scanner"

participant Q as "Task Queue"

participant WK as "Worker(s)"

participant DB as "Storage/Cache"

FS-->>WD : "Create/Modify/Delete/Move"

WD->>SC : "NormalizedEvent(path, type, meta)"

SC->>SC : "Filter by config/patterns"

SC->>Q : "Enqueue Task(item)"

WK->>Q : "Dequeue Task"

WK->>DB : "Persist/Update Index"

WK-->>SC : "Completion/Ack"

Diagram sources

Detailed Component Analysis¶

Watchdog Integration and Event Capture¶

Uses Watchdog’s Observer to monitor one or more root directories.
Emits standardized events for create, modify, delete, and move operations.
Normalizes metadata (path, event type, timestamps) before forwarding to the scanner.
Supports recursive watching with configurable depth and throttling to avoid event storms.

Operational notes:

Avoid duplicate processing by coalescing rapid successive events for the same path.
Apply include/exclude patterns early to reduce downstream load.
Handle OS-specific quirks (e.g., temporary files, antivirus scans) by ignoring transient changes.

Section sources

watchdog.py

Scanner Orchestration and Policy¶

Interprets normalized events and decides whether to process them based on:
File extensions and MIME hints
Inclusion/exclusion patterns
Directory allowlists/denylists
Rate limits and batch sizes
Enqueues tasks for workers with idempotency keys to prevent duplicates.
Applies backoff and retry for transient failures.

Processing logic highlights:

Deduplicate events per path within a time window.
Batch small bursts of events into a single scan job when appropriate.
Respect user-configured concurrency limits to protect I/O and CPU.

Section sources

scan.py

Configuration Options¶

Common configuration areas relevant to scanning and monitoring:

Watched directories list
Include/exclude patterns (glob-style)
Recursive depth limit
Event coalescing window
Max concurrent workers
Queue capacity and backpressure behavior
Retry/backoff parameters
Logging verbosity and metrics toggles

Example configuration shapes:

Minimal set: watched directories + basic include patterns
Advanced set: full pattern library, concurrency tuning, queue sizing, and retry policies

Section sources

config.py

Service Wiring and Lifecycle¶

Application startup initializes configuration, sets up the queue, and starts the Watchdog observer.
Background workers are launched to consume tasks from the queue.
Graceful shutdown stops observers, drains queues, and persists state.

Integration points:

Dependency injection for queue instances and database connections
Health/status endpoints to report observer and worker states
API hooks to add/remove watched directories at runtime

Section sources

Event Processing Pipeline¶

End-to-end flow from file system change to processed result:

Watchdog captures raw OS events.
Events are normalized and filtered by scanner policies.
Validated events are enqueued with deduplication keys.
Workers dequeue tasks, perform scanning/indexing, and update storage.
Completion status is recorded and optional notifications are emitted.

flowchart TD

Start(["FS Event"]) --> Normalize["Normalize Event"]

Normalize --> Filter{"Matches Policies?"}

Filter -- "No" --> Drop["Drop Event"]

Filter -- "Yes" --> Dedup{"Deduplicated?"}

Dedup -- "Yes" --> Drop

Dedup -- "No" --> Enqueue["Enqueue Task"]

Enqueue --> Worker["Worker Dequeues"]

Worker --> Process["Scan/Index Media"]

Process --> Persist["Persist Results"]

Persist --> End(["Done"])

Diagram sources

Dependency Analysis¶

High-level dependencies among components:

Watchdog depends on OS file system APIs and Watchdog library.
Scanner depends on configuration and queue interfaces.
Service wiring depends on application factory and dependency container.

graph LR

WD["Watchdog Observer"] --> SC["Scanner"]

SC --> CFG["Config"]

SC --> Q["Queue"]

MAIN["Main/Background"] --> WD

MAIN --> SC

APP["App Factory"] --> MAIN

DEPS["Deps Container"] --> MAIN

Diagram sources

Section sources

Performance Considerations¶

Optimization techniques for large media libraries:

Pattern precompilation and efficient glob matching to minimize CPU overhead.
Event coalescing windows to merge rapid changes into fewer tasks.
Batching strategy to group related files (e.g., same folder) into single jobs.
Concurrency controls: cap worker count and queue size to match disk and CPU capabilities.
Lazy loading and streaming where possible to reduce memory pressure.
Excluding known noisy directories (temp, cache, thumbnails) via denylists.
Using filesystem-native features (e.g., case-insensitive matching on Windows) to avoid extra checks.
Periodic compaction or index rebuilds to keep lookups fast.

[No sources needed since this section provides general guidance]

Troubleshooting Guide¶

Common issues and resolutions:

Permission errors:
Ensure the service runs with sufficient privileges to access watched directories.
Use explicit allowlists and exclude inaccessible paths to avoid repeated failures.
Memory spikes during large scans:
Reduce worker concurrency and increase event coalescing window.
Stream file reads and avoid holding large buffers in memory.
File system failures or unresponsive mounts:
Implement exponential backoff and circuit breaker patterns around watcher initialization.
Monitor mount health and restart watchers automatically after recovery.
Duplicate processing:
Verify deduplication keys and ensure consistent path normalization across platforms.
Event storms:
Increase throttle/coalesce thresholds and refine include/exclude patterns.

Section sources

Conclusion¶

Nexo’s scanning and monitoring system combines Watchdog-based event capture with a robust scanner and queue-backed processing pipeline. By configuring precise watch patterns, enabling event coalescing, and tuning concurrency, it scales efficiently over large media libraries while maintaining responsiveness and reliability. Proper error handling, retries, and graceful recovery further ensure stability under adverse conditions.

[No sources needed since this section summarizes without analyzing specific files]

Appendices¶

Example: Scan Configuration¶

Define watched directories and include patterns for supported media types.
Set recursion depth and exclusion lists for temp/thumbnail folders.
Configure coalescing window, max workers, and queue capacity.
Enable logging and metrics for observability.

Section sources

config.py

Example: Custom Watch Patterns¶

Use glob-style patterns to include only desired file extensions.
Exclude vendor or generated content to reduce noise.
Combine multiple patterns for fine-grained control.

Section sources

config.py

Example: Error Handling Strategies¶

Wrap watcher start/stop in try/except with retry and backoff.
Log detailed context for failed events and skip-and-retry semantics.
Persist partial progress to resume after restarts.

Section sources