System Status & Health API¶

Referenced Files in This Document

Table of Contents¶

Introduction
Project Structure
Core Components
Architecture Overview
Detailed Component Analysis
Dependency Analysis
Performance Considerations
Troubleshooting Guide
Conclusion
Appendices

Introduction¶

This document provides detailed API documentation for Nexo’s system monitoring and health check endpoints. It covers:

Health checks for application status, database connectivity, and service availability
System metrics collection including CPU usage, memory consumption, disk space, and processing queue status
Pipeline monitoring endpoints showing scan progress, processing rates, and bottleneck identification
Configuration summary, version information, and dependency status retrieval
Real-time status updates via WebSocket connections for live dashboards
Examples for automation, alerting integration, and performance monitoring setup
Guidance on metrics aggregation, historical data retention, and dashboard integration patterns

The goal is to enable operators and developers to monitor the health and performance of the video discovery pipeline reliably and integrate with external observability systems.

Project Structure¶

The monitoring and health features are implemented primarily under the service layer:

Routes exposing HTTP endpoints for health, metrics, configuration, and pipeline status
Shared configuration and database access utilities
Monitoring utilities and core pipeline components that provide runtime state
Cache-based persistence for status snapshots used by dashboards

graph TB

subgraph "Service Layer"

A["FastAPI App<br/>apps/service/app.py"]

B["Status Routes<br/>apps/service/routes/status.py"]

C["Config Module<br/>apps/shared/config.py"]

D["DB Access<br/>apps/shared/db.py"]

E["Watchdog Monitor<br/>apps/service/monitor/watchdog.py"]

end

subgraph "Core Pipeline"

F["Pipeline Core<br/>apps/service/core/pipeline.py"]

G["Scan Core<br/>apps/service/core/scan.py"]

end

subgraph "Runtime State"

H["Status Cache<br/>cache/status.json"]

end

A --> B

B --> C

B --> D

B --> E

B --> F

B --> G

B --> H

Diagram sources

Section sources

Core Components¶

Health Check Endpoints: Provide liveness/readiness signals, database connectivity checks, and overall service availability.
Metrics Collection: Expose CPU, memory, disk usage, and queue depth indicators.
Pipeline Monitoring: Surface scan progress, throughput, and bottleneck hints.
Configuration Summary: Return a sanitized snapshot of current configuration and version info.
Real-Time Updates: Offer WebSocket streams for live status and events.

These components are wired into the FastAPI application and rely on shared configuration and database modules. The watchdog monitor and core pipeline modules supply runtime state.

Section sources

Architecture Overview¶

The monitoring stack integrates HTTP endpoints with internal state providers and optional real-time channels.

sequenceDiagram

participant Client as "Client"

participant App as "FastAPI App"

participant Routes as "Status Routes"

participant Config as "Config"

participant DB as "Database"

participant Watchdog as "Watchdog Monitor"

participant Pipeline as "Pipeline Core"

participant Cache as "Status Cache"

Client->>App : GET /health

App->>Routes : handle_health()

Routes->>Config : read_config()

Routes->>DB : test_connection()

Routes->>Watchdog : get_system_metrics()

Routes->>Pipeline : get_pipeline_status()

Routes->>Cache : read_status_snapshot()

Routes-->>Client : {status, checks, metrics}

Client->>App : GET /metrics

App->>Routes : handle_metrics()

Routes->>Watchdog : collect_cpu_memory_disk()

Routes->>Pipeline : collect_queue_depth_rates()

Routes-->>Client : {cpu, memory, disk, queue, rates}

Client->>App : WS /ws/status

App->>Routes : open_websocket()

Routes->>Watchdog : subscribe_updates()

Routes->>Pipeline : stream_events()

Routes-->>Client : SSE/WebSocket frames

Diagram sources

Detailed Component Analysis¶

Health Check Endpoints¶

Purpose:

Liveness: Confirm the process is alive and responsive.
Readiness: Validate dependencies (database, external services).
Service Availability: Aggregate component health into a single status.

Key behaviors:

Returns an overall status derived from individual checks.
Includes per-check details such as component name, status, and error messages when applicable.
Supports quick response times suitable for probes and load balancers.

Typical fields:

Overall status
Timestamp
Checks: list of component statuses
Optional diagnostics or version metadata

Operational guidance:

Use liveness for container orchestration restarts.
Use readiness to gate traffic until dependencies are ready.

Section sources

System Metrics Collection¶

Purpose:

Provide operational metrics for CPU, memory, disk, and queue depth.
Enable alerting and capacity planning.

Metrics typically include:

CPU utilization percentage
Memory usage and available memory
Disk usage and free space
Queue depth and processing rates

Data sources:

OS-level metrics via watchdog monitor
Pipeline internals for queue depth and throughput
Optional cache-backed snapshots for historical views

Response considerations:

Keep responses lightweight and fast.
Avoid blocking I/O; prefer non-blocking reads.

Section sources

Pipeline Monitoring Endpoints¶

Purpose:

Show scan progress, processing rates, and identify bottlenecks.

Key capabilities:

Current scan phase and completion percentage
Throughput counters (items processed per time window)
Bottleneck indicators (e.g., slow stages, backpressure)
Historical trends via cached snapshots

Integration points:

Reads from pipeline core for stage states
Aggregates queue depths and processing durations
Optionally persists snapshots to cache for dashboards

Section sources

Configuration Summary, Version, and Dependencies¶

Purpose:

Provide a safe, sanitized view of current configuration.
Include application version and dependency status where relevant.

Behavior:

Excludes sensitive values (e.g., secrets, tokens).
Summarizes key settings affecting monitoring and pipeline behavior.
Reports version and optionally dependency health.

Use cases:

Auditing deployed configuration
Debugging environment differences
Validating feature flags and thresholds

Section sources

Real-Time Status Updates via WebSocket¶

Purpose:

Stream live status and events to dashboards and alerting clients.

Features:

Persistent connection for continuous updates
Event-driven notifications for pipeline milestones and errors
Backpressure handling to avoid overwhelming clients

Implementation notes:

Route handlers manage connection lifecycle
Watchdog and pipeline emit events consumed by the WebSocket handler
Clients can subscribe to specific event types or filters

Section sources

Example Workflows¶

Health Check Automation¶

Periodic HTTP GET to the health endpoint.
Parse overall status and per-check results.
Trigger alerts if readiness fails or critical checks report errors.

Alerting Integration¶

Subscribe to WebSocket events for critical failures.
Correlate with metrics spikes (CPU, memory, queue depth).
Escalate based on thresholds and duration.

Performance Monitoring Setup¶

Poll metrics endpoint at regular intervals.
Store time-series data in a metrics backend.
Build dashboards for CPU, memory, disk, queue depth, and throughput.

[No sources needed since this section provides general guidance]

Dependency Analysis¶

Monitoring endpoints depend on:

Configuration module for settings
Database module for connectivity checks
Watchdog monitor for system metrics
Pipeline core for processing state
Cache for persisted snapshots

graph LR

Routes["Status Routes"] --> Config["Config"]

Routes --> DB["DB Access"]

Routes --> Watchdog["Watchdog Monitor"]

Routes --> Pipeline["Pipeline Core"]

Routes --> Cache["Status Cache"]

Diagram sources

Section sources

Performance Considerations¶

Keep health checks lightweight and idempotent.
Avoid heavy I/O in readiness checks; use timeouts and short-circuit logic.
Debounce metric collection to prevent overhead during high-load periods.
Use caching for expensive aggregations and serve snapshots from cache when appropriate.
Implement rate limiting on WebSocket subscriptions to protect server resources.

[No sources needed since this section provides general guidance]

Troubleshooting Guide¶

Common issues and resolutions:

Health check failures: Inspect per-check diagnostics and error messages. Verify database credentials and network reachability.
Stale metrics: Ensure watchdog is running and has permissions to read system stats.
Pipeline stalls: Review queue depth and processing rates; look for bottleneck indicators in pipeline status.
WebSocket disconnects: Check client-side reconnection logic and server-side subscription limits.

Section sources

Conclusion¶

Nexo’s monitoring and health APIs provide essential visibility into application health, system resources, and pipeline performance. By combining HTTP endpoints with real-time streaming, teams can build robust automation, alerting, and dashboards. Adhering to the recommended practices ensures reliable operation and scalable observability.

[No sources needed since this section summarizes without analyzing specific files]

Appendices¶

Endpoint Reference Summary¶

Health Check: Application status, database connectivity, service availability
Metrics: CPU, memory, disk, queue depth, processing rates
Pipeline Monitoring: Scan progress, throughput, bottleneck hints
Configuration Summary: Sanitized config, version, dependency status
Real-Time Updates: WebSocket for live status and events

[No sources needed since this section provides general guidance]