feat: add headless Chromium fallback to recipe scraper
Build and Deploy / build-and-deploy (push) Has been cancelled

Uses Playwright to drive Chromium when the primary requests-based
fetch fails (e.g., Cloudflare, JS-required pages). Hybrid approach:
try fast path first, fall back to Chromium only on failure.

Dockerfile: add playwright==1.52.0 pip package, run playwright
install-deps chromium + playwright install chromium to download
the browser and all system libraries.

scrape-recipe: add fetch_with_chromium() that tries Playwright
first, then chromium --headless --dump-dom as a subprocess fallback.
Modify main() to catch primary fetch errors and call Chromium
fallback before giving up.
This commit is contained in:
2026-05-20 22:44:15 -04:00
parent 8807ba3851
commit ddb8577c4b
3 changed files with 349 additions and 4 deletions
+6 -2
View File
@@ -4,11 +4,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl python3 python3-pip python3-venv tini && \
rm -rf /var/lib/apt/lists/*
# Install recipe-scrapers in a virtualenv to isolate dependencies.
# Install recipe-scrapers and Playwright in a virtualenv to isolate dependencies.
RUN python3 -m venv /opt/roux-venv && \
/opt/roux-venv/bin/pip install --no-cache-dir recipe-scrapers==15.11.0 && \
/opt/roux-venv/bin/pip install --no-cache-dir recipe-scrapers==15.11.0 playwright==1.52.0 && \
rm -rf /root/.cache/pip
# Install Playwright system dependencies and Chromium browser.
RUN /opt/roux-venv/bin/playwright install-deps chromium && \
/opt/roux-venv/bin/playwright install chromium
# Build timestamp — read at runtime for diagnostics.
RUN mkdir -p /build && date -u '+%Y-%m-%d %H:%M UTC' > /build/build-time