feat: add headless Chromium fallback to recipe scraper
Build and Deploy / build-and-deploy (push) Has been cancelled
Build and Deploy / build-and-deploy (push) Has been cancelled
Uses Playwright to drive Chromium when the primary requests-based fetch fails (e.g., Cloudflare, JS-required pages). Hybrid approach: try fast path first, fall back to Chromium only on failure. Dockerfile: add playwright==1.52.0 pip package, run playwright install-deps chromium + playwright install chromium to download the browser and all system libraries. scrape-recipe: add fetch_with_chromium() that tries Playwright first, then chromium --headless --dump-dom as a subprocess fallback. Modify main() to catch primary fetch errors and call Chromium fallback before giving up.
This commit is contained in:
+6
-2
@@ -4,11 +4,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates curl python3 python3-pip python3-venv tini && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install recipe-scrapers in a virtualenv to isolate dependencies.
|
||||
# Install recipe-scrapers and Playwright in a virtualenv to isolate dependencies.
|
||||
RUN python3 -m venv /opt/roux-venv && \
|
||||
/opt/roux-venv/bin/pip install --no-cache-dir recipe-scrapers==15.11.0 && \
|
||||
/opt/roux-venv/bin/pip install --no-cache-dir recipe-scrapers==15.11.0 playwright==1.52.0 && \
|
||||
rm -rf /root/.cache/pip
|
||||
|
||||
# Install Playwright system dependencies and Chromium browser.
|
||||
RUN /opt/roux-venv/bin/playwright install-deps chromium && \
|
||||
/opt/roux-venv/bin/playwright install chromium
|
||||
|
||||
# Build timestamp — read at runtime for diagnostics.
|
||||
RUN mkdir -p /build && date -u '+%Y-%m-%d %H:%M UTC' > /build/build-time
|
||||
|
||||
|
||||
Reference in New Issue
Block a user