Metadata Extraction¶

Referenced Files in This Document

Table of Contents¶

Introduction
Project Structure
Core Components
Architecture Overview
Detailed Component Analysis
Dependency Analysis
Performance Considerations
Troubleshooting Guide
Conclusion

Introduction¶

This document explains Nexo’s metadata extraction system with a focus on the pluggable parser architecture that supports multiple file formats through a unified interface. It covers base parser classes, format-specific implementations for documents and images, the registration mechanism, and the end-to-end extraction pipeline including validation, normalization, and error handling. It also provides guidance for implementing custom parsers, extending existing ones, handling errors gracefully, and optimizing performance for batch processing and large files.

Project Structure¶

The metadata extraction logic is primarily implemented under src/core and integrated into the service layer via apps/service. Key modules:

Base interfaces and shared types
Pluggable parser registry and dispatch
Format-specific parsers (documents and images)
Pipeline orchestration and scanning utilities
Service wiring to expose parsing capabilities

graph TB

subgraph "Core"

P["parser.py"]

DP["document_parser.py"]

IP["image_parser.py"]

T["types.py"]

PL["pipeline.py"]

S["scan.py"]

end

subgraph "Service"

A["app.py"]

end

A --> PL

PL --> P

P --> DP

P --> IP

P --> T

S --> P

Diagram sources

Section sources

Core Components¶

Unified Parser Interface: Defines a consistent contract for extracting metadata from any supported file type. Implementations must provide a method to parse a file path or stream and return normalized metadata.
Shared Types: Centralized data structures for parsed metadata, ensuring consistency across parsers and downstream consumers.
Parser Registry: A centralized mapping from file extension or MIME type to a concrete parser implementation, enabling dynamic discovery and dispatch without hard-coded branches.
Format-Specific Parsers:
Document Parser: Handles text-based and structured document formats.
Image Parser: Handles image formats and embedded metadata.
Pipeline Orchestration: Coordinates scanning, selection of the appropriate parser, execution, validation, normalization, and result aggregation.

Key responsibilities:

Registration: Parsers register themselves by supported extensions/MIME types.
Dispatch: The registry selects the correct parser based on input characteristics.
Validation and Normalization: Ensure outputs conform to shared types and are cleaned up for downstream use.
Error Handling: Provide clear, actionable errors and fallbacks when parsing fails.

Section sources

Architecture Overview¶

The metadata extraction system follows a pluggable architecture:

Parsers implement a common interface and self-register with the registry.
The pipeline orchestrates scanning and parsing, delegating to the registry for dispatch.
Outputs are validated against shared types and normalized before being returned to callers.

classDiagram

class Parser {

+parse(file_path) Metadata

+supported_extensions() string[]

+supported_mime_types() string[]

}

class DocumentParser {

+parse(file_path) Metadata

+supported_extensions() string[]

}

class ImageParser {

+parse(file_path) Metadata

+supported_extensions() string[]

}

class ParserRegistry {

-registry dict

+register(parser) void

+get_parser(file_path) Parser

+get_parser_by_mime(mime) Parser

}

class Metadata {

+title string

+description string

+tags string[]

+dimensions object

+duration number

+format string

}

Parser <|-- DocumentParser

Parser <|-- ImageParser

ParserRegistry --> Parser : "dispatches"

Parser --> Metadata : "produces"

Diagram sources

Detailed Component Analysis¶

Base Parser and Registry¶

Base Parser:
Defines the abstract interface for all parsers.
Exposes methods to declare supported extensions and MIME types.
Provides default behaviors where applicable (e.g., logging, basic validation).
Parser Registry:
Maintains mappings from extensions and MIME types to parser instances.
Offers lookup functions to select the best-matching parser for a given file path or content type.
Supports dynamic registration so new parsers can be added without modifying core dispatch logic.

Implementation patterns:

Use a central registry instance initialized at import time.
Decorators or explicit registration calls ensure parsers are discoverable.
Fallback behavior returns a clear “unsupported” result when no parser matches.

Error handling:

Raise specific exceptions for unsupported formats.
Wrap third-party library errors with contextual messages indicating file path and detected format.

Section sources

parser.py

Document Parser¶

Responsibilities:

Extract textual and structural metadata from document-like formats.
Normalize fields such as title, description, tags, and format.
Handle encoding issues and malformed inputs gracefully.

Validation and normalization:

Strip and trim strings.
Deduplicate tags.
Map internal format codes to canonical values.

Error handling:

Catch decoding errors and report them with context.
Return partial metadata when possible, marking missing fields explicitly.

Section sources

document_parser.py

Image Parser¶

Responsibilities:

Read image headers and embedded metadata (EXIF/IPTC/XMP if available).
Infer dimensions, orientation, color space, and other attributes.
Normalize image-specific fields into the shared metadata schema.

Validation and normalization:

Validate dimension ranges and reject corrupted images early.
Normalize date/time fields and coordinate systems.

Error handling:

Detect truncated or unsupported image formats.
Provide informative errors and skip problematic files during batch operations.

Section sources

image_parser.py

Shared Types¶

Purpose:

Define the canonical metadata structure consumed by downstream components.
Enforce field presence and types to reduce runtime checks.

Design considerations:

Use immutable or frozen structures where possible to prevent accidental mutation.
Provide helper constructors or builders to simplify creation and validation.

Section sources

types.py

Pipeline Orchestration¶

Responsibilities:

Coordinate scanning and parsing workflows.
Select the appropriate parser using the registry.
Apply validation and normalization steps consistently.
Aggregate results and handle failures per item without aborting entire batches.

Flow overview:

Input: file paths or streams.
Dispatch: resolve parser via registry.
Parse: execute parser.parse().
Validate: check output against shared types.
Normalize: clean and standardize fields.
Output: return standardized metadata or error records.

sequenceDiagram

participant Caller as "Caller"

participant Pipeline as "Pipeline"

participant Registry as "ParserRegistry"

participant Parser as "Concrete Parser"

participant Types as "Metadata Schema"

Caller->>Pipeline : "extract(file_path)"

Pipeline->>Registry : "get_parser(file_path)"

Registry-->>Pipeline : "Parser instance"

Pipeline->>Parser : "parse(file_path)"

Parser-->>Pipeline : "Raw metadata"

Pipeline->>Types : "validate_and_normalize(raw)"

Types-->>Pipeline : "Normalized metadata"

Pipeline-->>Caller : "Result or error"

Diagram sources

Section sources

pipeline.py

Scanning Utilities¶

Responsibilities:

Discover files in directories or lists.
Filter by supported extensions or MIME types.
Provide progress feedback and resumability hints for long-running scans.

Integration:

Works with the pipeline to feed items for parsing.
Can short-circuit unsupported files early to save resources.

Section sources

scan.py

Service Integration¶

Responsibilities:

Wire parsing capabilities into the application’s API surface.
Expose endpoints or functions that trigger scanning and extraction.
Surface errors and status information to clients.

Section sources

app.py

Dependency Analysis¶

High-level dependencies:

Pipeline depends on the registry and shared types.
Parsers depend on shared types and external libraries for reading formats.
Service layer depends on pipeline and registry to expose functionality.

graph LR

App["apps/service/app.py"] --> Pipe["src/core/pipeline.py"]

Pipe --> Reg["src/core/parser.py"]

Pipe --> Types["src/core/types.py"]

Reg --> Doc["src/core/document_parser.py"]

Reg --> Img["src/core/image_parser.py"]

Scan["src/core/scan.py"] --> Reg

Diagram sources

Section sources

Performance Considerations¶

Batch processing:

Process files concurrently with bounded concurrency to avoid resource exhaustion.
Stream large files where possible instead of loading entirely into memory.
Short-circuit unsupported formats early to minimize overhead.

Memory management:

Avoid retaining references to raw bytes after parsing; release handles promptly.
Use generators or iterators for large directory scans.
Prefer chunked reads for large binary formats.

Caching strategies:

Cache parsed metadata keyed by stable identifiers (e.g., file path plus modification time or checksum).
Invalidate cache entries when underlying files change.
Consider LRU caches for frequently accessed items while bounding memory usage.

Normalization efficiency:

Perform lightweight validation first; defer expensive transformations until necessary.
Deduplicate and normalize once per item to avoid repeated work.

[No sources needed since this section provides general guidance]

Troubleshooting Guide¶

Common issues and resolutions:

Unsupported format:
Symptom: No parser found for the file extension or MIME type.
Action: Register a new parser or extend supported extensions in the registry.
Parsing failure due to corruption:
Symptom: Exceptions raised while reading headers or streams.
Action: Wrap third-party calls with try/catch, log context, and mark the item as failed while continuing the batch.
Missing or invalid metadata fields:
Symptom: Normalization rejects incomplete data.
Action: Provide defaults or null markers for optional fields; validate only required fields strictly.
Encoding problems in text-based documents:
Symptom: Decoding errors or garbled text.
Action: Attempt multiple encodings with fallbacks; record detected encoding for audit.
Large file performance:
Symptom: High memory usage or slow processing.
Action: Enable streaming reads, limit concurrency, and leverage caching for repeated extractions.

Operational tips:

Log detailed context for each parse attempt (file path, detected format, parser used).
Track metrics: success rate, average parse time, and error categories.
Provide a dry-run mode to validate inputs before full processing.

Section sources

Conclusion¶

Nexo’s metadata extraction system uses a pluggable parser architecture to support multiple file formats through a unified interface. The base parser defines a consistent contract, the registry enables dynamic dispatch, and format-specific parsers implement extraction and normalization tailored to their domains. The pipeline coordinates scanning, parsing, validation, and normalization while providing robust error handling. By following the guidance for custom parsers, error handling, and performance optimization, you can extend the system to new formats and scale it effectively for large-scale operations.