Files

T

Build and Deploy / build-and-deploy (push) Has been cancelled

Details

feat: add headless Chromium fallback to recipe scraper

Uses Playwright to drive Chromium when the primary requests-based
fetch fails (e.g., Cloudflare, JS-required pages). Hybrid approach:
try fast path first, fall back to Chromium only on failure.

Dockerfile: add playwright==1.52.0 pip package, run playwright
install-deps chromium + playwright install chromium to download
the browser and all system libraries.

scrape-recipe: add fetch_with_chromium() that tries Playwright
first, then chromium --headless --dump-dom as a subprocess fallback.
Modify main() to catch primary fetch errors and call Chromium
fallback before giving up.

2026-05-20 22:44:15 -04:00

9.0 KiB

Raw Permalink Blame History

Headless Chromium Scraper Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a Chromium-based fallback to the recipe scraper so sites that block plain HTTP requests (Cloudflare, JS-required pages, etc.) can still be imported.

Architecture: Hybrid approach — try the fast requests path first (current behavior). If it fails with an HTTP error or connection error, fall back to Playwright-driven headless Chromium to fetch the rendered HTML, then pass it to the existing schema.org extraction pipeline. No Haskell-side changes needed.

Tech Stack: Python, Playwright, Chromium, recipe-scrapers (existing), Docker (debian:bookworm-slim)

Files to change

File	Change	Responsibility
`Dockerfile`	Add Chromium + Playwright to the image	Browser rendering
`scripts/scrape-recipe`	Add hybrid fetch: try `requests` first, fall back to Playwright	Fetch + extraction
`docker-compose.yml`	No changes needed (same image)	—
`src/Roux/Server.hs`	No changes needed (calls script same way)	—

Task 1: Add Chromium + Playwright to Dockerfile

Files:

Modify: Dockerfile

Reasoning: The current image only has recipe-scrapers in a venv. We need:

libffi-dev and other Playwright system deps (playwright install chromium downloads its own Chromium)
playwright Python package in the venv
Chromium browser downloaded by playwright install chromium
--no-sandbox flag support (common in Docker environments)

Step 1: Add Playwright deps to apt-get install

Add libffi-dev libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libdbus-1-3 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2 to the apt-get install line.

Step 2: Install Playwright in the venv

Add playwright==1.52.0 to the pip install line (same line as recipe-scrapers).

Step 3: Install Chromium browser

Add a RUN step after pip install:

RUN /opt/roux-venv/bin/playwright install chromium

This downloads Chromium (~200MB) to /root/.cache/ms-playwright/.

Step 4: Add CHROMIUM_PATH env var

Add:

ENV CHROMIUM_PATH="/root/.cache/ms-playwright/chromium-1161/chrome-linux/chrome"

This lets the Python script locate the browser without relying on Playwright's discovery (which can fail in Docker).

Step 5: Build and verify

cd /work/personal/roux/roux-main
docker build -t roux-test .

Expected: Build succeeds, Chromium is downloaded.

Task 2: Modify scrape-recipe to add Chromium fallback

Files:

Modify: scripts/scrape-recipe

Reasoning: The current script calls scrape_html(html=None, org_url=url, online=True) which uses requests under the hood. We wrap this in a try/except. On failure, we use Playwright to launch Chromium headless, get the rendered HTML, and feed it to scrape_html(html=html, org_url=url, online=False).

Key design decisions:

Use Playwright's sync API (sync_playwright) — simpler for a CLI script
Pass --no-sandbox to Chromium (required in Docker)
Set a 30-second timeout for page navigation
Extract <html> tag content from the fully rendered page
Only fall back if the primary request fails (exception from requests)
Step 1: Add fallback import at the top

Add after the existing imports:

import os
import subprocess
import tempfile

No need to import Playwright at module level — only import it inside the fallback path so the script works even if Playwright isn't installed (for local dev without Docker).

Step 2: Add the Chromium fallback function

Add before main():

def fetch_with_chromium(url: str, timeout: int = 30) -> str:
    """Fetch rendered HTML from a URL using headless Chromium via Playwright.
    
    Falls back to chromium --headless --dump-dom as a subprocess if Playwright
    is not available (unlikely in Docker, possible in bare-metal dev).
    """
    html = None
    
    # Try Playwright first (primary path in Docker)
    try:
        from playwright.sync_api import sync_playwright
        
        with sync_playwright() as p:
            browser = p.chromium.launch(
                headless=True,
                args=["--no-sandbox", "--disable-setuid-sandbox"]
            )
            page = browser.new_page()
            page.goto(url, wait_until="networkidle", timeout=timeout * 1000)
            html = page.content()
            browser.close()
        return html
    except ImportError:
        pass  # Playwright not installed, try subprocess approach
    except Exception as e:
        print(f"Playwright failed: {e}", file=sys.stderr)
        print("Falling back to chromium --headless --dump-dom", file=sys.stderr)
    
    # Fallback: chromium --headless --dump-dom
    chromium_path = os.environ.get(
        "CHROMIUM_PATH",
        "/usr/bin/chromium"
    )
    try:
        result = subprocess.run(
            [chromium_path, "--headless", "--dump-dom", url],
            capture_output=True,
            text=True,
            timeout=timeout
        )
        if result.returncode == 0 and result.stdout:
            return result.stdout
        print(f"chromium --dump-dom failed: {result.stderr[:500]}", file=sys.stderr)
    except FileNotFoundError:
        pass
    except Exception as e:
        print(f"chromium --dump-dom error: {e}", file=sys.stderr)
    
    raise RuntimeError("Could not fetch page with Chromium or Playwright")

Step 3: Modify the main fetch logic

Replace the current download block:

if args.html:
    with open(args.html) as f:
        html = f.read()
    scraper = scrape_html(html=html, org_url=args.url, online=False)
else:
    # Download the page and scrape it
    scraper = scrape_html(html=None, org_url=args.url, online=True)

With hybrid logic:

if args.html:
    with open(args.html) as f:
        html = f.read()
    scraper = scrape_html(html=html, org_url=args.url, online=False)
else:
    try:
        # Fast path: use requests via recipe-scrapers
        scraper = scrape_html(html=None, org_url=args.url, online=True)
    except Exception as primary_err:
        print(f"Primary fetch failed ({type(primary_err).__name__}: {primary_err}), trying Chromium...", file=sys.stderr)
        try:
            html = fetch_with_chromium(args.url)
            scraper = scrape_html(html=html, org_url=args.url, online=False)
        except Exception as chrome_err:
            print(f"Chromium fallback also failed: {chrome_err}", file=sys.stderr)
            raise RuntimeError(
                f"Could not fetch '{args.url}'. "
                f"Primary error: {primary_err}. "
                f"Chromium fallback error: {chrome_err}."
            ) from chrome_err

Step 4: Verify the script works with a known site

cd /work/personal/roux/roux-main
docker build -t roux-test .
docker run --rm roux-test /opt/roux-venv/bin/python /usr/local/bin/scrape-recipe --pretty https://cooking.nytimes.com/recipes/12177-fried-rice

Expected: outputs JSON-LD schema.org data.

Step 5: Verify the fallback path

Test that the fallback actually triggers by pointing at a URL that blocks requests but works in a browser. Or simulate by temporarily modifying the script:

docker run --rm roux-test /opt/roux-venv/bin/python -c "
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    b = p.chromium.launch(headless=True, args=['--no-sandbox'])
    page = b.new_page()
    page.goto('https://cooking.nytimes.com/recipes/12177-fried-rice', wait_until='networkidle')
    print(page.title())
    b.close()
"

Expected: "Fried Rice Recipe"

Step 6: Commit

git add Dockerfile scripts/scrape-recipe
git commit -m "feat: add headless Chromium fallback to recipe scraper

Uses Playwright to drive Chromium when the primary requests-based
fetch fails (e.g., Cloudflare, JS-required pages). Hybrid approach:
try fast path first, fall back to Chromium only on failure.
No Haskell-side changes needed."

Task 3: Verify end-to-end import

Files:

No file changes — verification only
Step 1: Build the Docker image

docker build -t roux-test .

Step 2: Run the scraper directly

# Fast path (should succeed with NYT)
docker run --rm roux-test /opt/roux-venv/bin/python /usr/local/bin/scrape-recipe --pretty https://cooking.nytimes.com/recipes/12177-fried-rice 2>&1 | head -5

# Verify the JSON output starts with a schema.org type
docker run --rm roux-test /opt/roux-venv/bin/python /usr/local/bin/scrape-recipe --pretty https://cooking.nytimes.com/recipes/12177-fried-rice 2>&1 | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('@type','no type'))"

Expected for both: valid JSON with @type: "Recipe" or similar.

Step 3: Push to origin

git push origin main

9.0 KiB Raw Permalink Blame History

Headless Chromium Scraper Implementation Plan

Files to change

Task 1: Add Chromium + Playwright to Dockerfile

Task 2: Modify scrape-recipe to add Chromium fallback

Task 3: Verify end-to-end import

9.0 KiB

Raw Permalink Blame History