Video Downloaders¶
Referenced Files in This Document
Table of Contents¶
Introduction¶
This document explains Nexo’s video downloader implementations, covering multiple strategies: Playwright-based browser automation, stealth modes for anti-detection bypass, cookie-managed authenticated downloads, and simple direct downloads. It also documents authentication mechanisms, proxy support, error handling, retry logic, progress tracking, CAPTCHA handling, rate limiting, and the redecanais wrapper integration. Guidance is included for developing custom downloaders.
Project Structure¶
The downloader subsystem consists of a base strategy and several specialized implementations plus a wrapper for an external library.
graph TB
A["Base Downloader<br/>video_downloader.py"] --> B["Playwright Downloader<br/>video_downloader_playwright.py"]
A --> C["Stealth Downloader<br/>video_downloader_stealth.py"]
A --> D["Simple Downloader<br/>video_downloader_simple.py"]
A --> E["Cookies Downloader<br/>video_downloader_cookies.py"]
A --> F["Custom Solution Downloader<br/>video_downloader_cs.py"]
A --> G["Manual Downloader<br/>video_downloader_manual.py"]
A --> H["User Credentials Downloader<br/>video_downloader_uc.py"]
I["RedeCanais Wrapper<br/>redecanais_wrapper.py"] --> A
[No sources needed since this diagram shows conceptual structure]
Core Components¶
-
Base downloader class defines the common interface and shared behavior used by all strategies.
-
Specialized classes implement site-specific or capability-specific logic:
-
Playwright-based automation for dynamic sites requiring JavaScript rendering.
-
Stealth mode to reduce detection by anti-bot systems.
-
Cookie-based authentication for protected resources.
-
Simple HTTP client for straightforward direct downloads.
-
Manual and user-credentials variants for edge cases and interactive flows.
-
RedeCanais wrapper integrates an external library to assist with certain sources.
Key responsibilities typically include:
-
URL resolution and media link extraction
-
Authentication and session management
-
Proxy configuration
-
Retry and backoff policies
-
Progress reporting
-
Error classification and recovery
Section sources
Architecture Overview¶
The downloader architecture follows a strategy pattern where a common interface is implemented by multiple concrete strategies. The orchestrator selects the appropriate strategy based on source characteristics and configuration.
classDiagram
class BaseDownloader {
+download(url, options) Result
+set_proxy(proxy)
+set_timeout(timeout)
+set_retry_policy(policy)
+on_progress(callback)
}
class PlaywrightDownloader {
+launch_browser()
+navigate_and_extract(url)
+handle_captcha()
}
class StealthDownloader {
+apply_stealth_flags()
+rotate_user_agent()
}
class CookiesDownloader {
+load_cookies(path)
+save_cookies(path)
+authenticate_session()
}
class SimpleDownloader {
+http_get(url, headers)
+stream_download(url, dest)
}
class RedeCanaisWrapper {
+resolve(url)
+get_stream_url()
}
BaseDownloader <|-- PlaywrightDownloader
BaseDownloader <|-- StealthDownloader
BaseDownloader <|-- CookiesDownloader
BaseDownloader <|-- SimpleDownloader
BaseDownloader ..> RedeCanaisWrapper : "uses"
Diagram sources
Detailed Component Analysis¶
Base Downloader Strategy¶
Responsibilities:
-
Define the unified download contract and shared utilities (logging, retries, timeouts).
-
Provide hooks for progress callbacks and error handlers.
-
Centralize configuration parsing and validation.
Typical configuration options:
-
Timeout settings
-
Retry count and backoff strategy
-
Proxy endpoints
-
User-Agent and header overrides
-
Output path and naming templates
Error handling:
-
Classify network vs. server vs. content errors
-
Exponential backoff with jitter
-
Circuit breaker for repeated failures
Progress tracking:
- Emit periodic events with bytes downloaded and estimated time remaining.
Section sources
Playwright-Based Browser Automation¶
Use cases:
-
Sites requiring full JS execution, dynamic token generation, or complex navigation.
-
Sources behind heavy anti-bot protections that still allow headless browsing.
Key behaviors:
-
Launches a headless browser instance
-
Navigates to target URLs and waits for specific DOM elements
-
Extracts final media links from rendered pages
-
Captures screenshots or logs on failure for diagnostics
CAPTCHA handling:
- Detects CAPTCHA prompts and either pauses for manual intervention or falls back to alternative strategies.
Proxy and stealth:
- Can be combined with stealth flags to reduce fingerprinting.
Section sources
Stealth Mode for Anti-Detection Bypass¶
Use cases:
-
High-risk targets with aggressive bot detection.
-
Environments where standard requests are frequently blocked.
Techniques:
-
Randomized User-Agent rotation
-
TLS fingerprint adjustments
-
Request timing randomization
-
Header normalization and spoofing
Integration:
- Extends base strategy and optionally composes with Playwright or HTTP clients.
Section sources
Cookie Management for Authenticated Downloads¶
Use cases:
-
Protected content requiring login sessions.
-
Multi-step authentication flows completed once and reused.
Features:
-
Load cookies from disk or memory
-
Persist updated cookies after login
-
Validate session freshness before use
-
Rotate accounts if available
Flow overview:
sequenceDiagram
participant Client as "Caller"
participant CD as "CookiesDownloader"
participant FS as "Filesystem"
participant Site as "Target Site"
Client->>CD : "download(url, cookies_path)"
CD->>FS : "load_cookies(cookies_path)"
alt "Cookies found"
CD->>Site : "Request with session cookies"
Site-->>CD : "Authenticated response"
CD-->>Client : "Media stream/link"
else "No cookies"
CD->>CD : "Perform login flow"
CD->>FS : "save_cookies(new_cookies)"
CD->>Site : "Request with new session"
Site-->>CD : "Authenticated response"
CD-->>Client : "Media stream/link"
end
Diagram sources
Section sources
Simple Direct Download Method¶
Use cases:
-
Static or CDN-hosted media with predictable URLs.
-
Low-friction sources without heavy protections.
Characteristics:
-
Minimal dependencies
-
Streamed downloads with progress updates
-
Basic retry and timeout controls
Section sources
Custom Solution and Manual/UC Variants¶
-
Custom solution downloader encapsulates site-specific logic not covered by generic strategies.
-
Manual downloader supports interactive steps or human-in-the-loop decisions.
-
User credentials downloader manages per-user auth tokens and profiles.
Section sources
RedeCanais Wrapper Integration¶
Purpose:
- Provides a higher-level API over an external library to resolve streams for supported sources.
Integration points:
-
Wraps library calls into the base downloader contract
-
Normalizes errors and progress signals
-
Adds fallbacks when the library fails
flowchart TD
Start(["Resolve via RedeCanais"]) --> CallLib["Call wrapper.resolve(url)"]
CallLib --> HasResult{"Stream URL found?"}
HasResult --> |Yes| UseDirect["Use SimpleDownloader to fetch"]
HasResult --> |No| Fallback["Fallback to Playwright/Stealth"]
UseDirect --> End(["Return result"])
Fallback --> End
Diagram sources
Section sources
Dependency Analysis¶
-
Base strategy depends on shared utilities for logging, config, and networking.
-
Playwright downloader depends on the Playwright runtime and optional stealth plugins.
-
Stealth downloader may depend on fingerprinting libraries and request shapers.
-
Cookies downloader depends on filesystem or secure storage for persistence.
-
Simple downloader depends on a robust HTTP client with streaming support.
-
RedeCanais wrapper depends on the external library and provides compatibility layers.
graph LR
Base["Base Downloader"] --> Net["HTTP Client"]
Base --> Log["Logger"]
Base --> Conf["Config"]
PW["Playwright Downloader"] --> Base
PW --> PwLib["Playwright Runtime"]
Stealth["Stealth Downloader"] --> Base
Stealth --> FP["Fingerprint Utils"]
Cookies["Cookies Downloader"] --> Base
Cookies --> Store["Cookie Store"]
Simple["Simple Downloader"] --> Base
RCW["RedeCanais Wrapper"] --> Base
RCW --> RCLib["External Library"]
Diagram sources
Section sources
Performance Considerations¶
-
Prefer SimpleDownloader for static links to minimize overhead.
-
Reuse browser instances in Playwright mode; avoid launching per-request.
-
Apply stealth only when necessary to reduce CPU and memory usage.
-
Cache cookies and resolved tokens to avoid repeated logins.
-
Tune retry backoff and timeouts according to target reliability.
-
Stream large media to avoid high memory consumption.
-
Use proxies judiciously; measure latency impact and rotate responsibly.
[No sources needed since this section provides general guidance]
Troubleshooting Guide¶
Common issues and remedies:
-
CAPTCHA encountered:
-
Switch to manual downloader or pause for user input.
-
Reduce stealth aggressiveness if it triggers additional checks.
-
Rate limiting:
-
Increase backoff intervals and add jitter.
-
Rotate proxies and user agents.
-
Session expiration:
-
Refresh cookies and re-authenticate.
-
Validate session health before each request.
-
Network errors:
-
Verify proxy connectivity and DNS.
-
Inspect SSL/TLS handshake and certificate validity.
-
Memory pressure:
-
Stream responses and limit concurrent downloads.
-
Close browser contexts promptly.
Operational tips:
-
Enable detailed logs around request/response cycles.
-
Capture page snapshots or network traces on failure.
-
Implement circuit breakers for unstable sources.
Section sources
Conclusion¶
Nexo’s downloader subsystem offers a flexible set of strategies to handle diverse sources and protection levels. By composing a base strategy with specialized capabilities—browser automation, stealth, cookies, and direct HTTP—you can reliably extract and download media across a wide range of environments. The redecanais wrapper further extends coverage for supported sources. For new integrations, follow the established strategy pattern, implement clear error handling and progress reporting, and prefer minimal dependencies for performance and maintainability.