feat: add images to all 50 recipes with frontmatter (Task 2)
Downloaded recipe images from source URLs (NYT Cooking, Love & Lemons, Food Network, etc.) via og:image scraping and DuckDuckGo search + page scraping for recipes without direct source URLs.
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/alain-ducase-bread.webp
|
||||
---
|
||||
|
||||
title: Alain Ducasse's Gougères
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/arugula-pesto-chicken-with-pecans-roasted-butternut.webp
|
||||
---
|
||||
|
||||
title: Arugula Pesto Chicken with Pecans & Roasted Butternut
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/arugula-pesto-wheat-berries-recipe.webp
|
||||
---
|
||||
title: Arugula Pesto Wheat Berries Recipe
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/asparagus-salad.webp
|
||||
---
|
||||
title: Asparagus Salad
|
||||
prep_time: 20 minutes
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/authentic-cacio-e-pepe-recipe.webp
|
||||
---
|
||||
title: Authentic Cacio e Pepe Recipe
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/butternut-turkey-chili-with-sweet-peppers-spiced-pepitas-queso-fresco.webp
|
||||
---
|
||||
title: BUTTERNUT & TURKEY CHILI with sweet peppers, spiced pepitas & queso fresco
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/baked-chile-relleno-with-black-beans.webp
|
||||
---
|
||||
title: Baked Chile Relleno with Black Beans
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/bang-bang-tofu.webp
|
||||
---
|
||||
title: Bang Bang Tofu
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/barbecue-chicken-with-crispy-pan-fried-okra-cheddar-biscuits.webp
|
||||
---
|
||||
title: Barbecue Chicken with Crispy Pan-Fried Okra & Cheddar Biscuits
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/black-bean-and-corn-salad.webp
|
||||
---
|
||||
title: Black Bean and Corn Salad
|
||||
servings: 4
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/blackened-shrimp-and-gouda-grits.webp
|
||||
---
|
||||
title: Blackened Shrimp & Gouda Grits
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/butternut-turkey-chili-2.webp
|
||||
---
|
||||
title: Butternut and Turkey Chili
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/cabbage-salad.webp
|
||||
---
|
||||
title: Cabbage Salad
|
||||
servings: 6
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chana-chickpea-salad.webp
|
||||
---
|
||||
title: Chana Salad
|
||||
description: I think my mom lied to me about this chana salaad. She said it was junk food so I would eat it, but now that I'm older I know better. After all, it's chickpeas and vegetables! But I'm glad my mom said whatever it took at the time to get me to eat it, because I love this salad. It is perfect for summer and is easy to make. Serve it as a side salad or as a main dish for a vegan meal. Adjust the cayenne as desired for your heat preference.
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chicken-and-pumpkin-gnocchi-with-ricotta.webp
|
||||
---
|
||||
title: Chicken & Pumpkin Gnocchi with Ricotta
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chickpea-butter-curry.webp
|
||||
---
|
||||
title: Chickpea Butter Curry
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chickpea-butter-curry-2.webp
|
||||
---
|
||||
title: Chickpea Butter Curry
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chinese-orange-shrimp.webp
|
||||
---
|
||||
title: Chinese Orange Shrimp
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/chocolate-mousse-cake-with-raspberries.webp
|
||||
---
|
||||
title: Chocolate Mousse Cake with Raspberries
|
||||
servings: 24
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/cilantro-pesto-shrimp-over-orzo.webp
|
||||
---
|
||||
title: Cilantro Pesto Shrimp Over Orzo
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/crispy-roasted-chickpeas-garbanzo-beans.webp
|
||||
---
|
||||
title: Crispy Roasted Chickpeas (Garbanzo Beans)
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/crispy-roasted-chickpeas-garbanzo-beans-2.webp
|
||||
---
|
||||
title: Crispy Roasted Chickpeas (Garbanzo Beans)
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/dried-fava-soup.webp
|
||||
---
|
||||
title: Dried Fava Soup with Mint and Guajillo Chiles
|
||||
prep_time: 10 minutes
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/easy-lemon-rosemary-white-bean-soup.webp
|
||||
---
|
||||
title: EASY LEMON ROSEMARY WHITE BEAN SOUP
|
||||
servings: 4
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/easy-lemon-rosemary-white-bean-soup-2.webp
|
||||
---
|
||||
title: EASY LEMON ROSEMARY WHITE BEAN SOUP
|
||||
servings: 4
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/gaja-s-bulgogi-lettuce-wraps.webp
|
||||
---
|
||||
title: Gaja's Bulgogi Lettuce Wraps
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/gochugaru-honey-salmon.webp
|
||||
---
|
||||
title: Gochugaru Honey Salmon
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/indian-spiced-chicken-2.webp
|
||||
---
|
||||
title: Indian Spiced Chicken with Creamy Tomato Sauce & Spinach
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/kale-and-chestnut-stew.webp
|
||||
---
|
||||
title: Kale & Chestnut Stew with Butternut Squash, Wild Rice & Parsley–Walnut Pesto
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/kkanpoong-tofu.webp
|
||||
---
|
||||
title: Spicy & Crunchy Garlic Tofu (Kkanpoong Tofu)
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/korean-shrimp-tacos.webp
|
||||
---
|
||||
title: Korean Shrimp Tacos
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/kung-pao-shrimp-with-sichuan-roasted-green-beans.webp
|
||||
---
|
||||
title: Kung Pao Shrimp with Sichuan Roasted Green Beans
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/lactation-cookies.webp
|
||||
---
|
||||
title: Lactation cookies
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/lactation-cookies-2.webp
|
||||
---
|
||||
title: Lactation Cookies
|
||||
servings: 12
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/moroccan-spiced-lentil-chickpea-soup.webp
|
||||
---
|
||||
title: Moroccan Spiced Lentil & Chickpea Soup
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/moroccan-spiced-lentil-and-chickpea-soup.webp
|
||||
---
|
||||
title: Moroccan Spiced Lentil & Chickpea Soup
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/orange-rolls.webp
|
||||
---
|
||||
title: Orange Rolls
|
||||
servings: 8
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/palak-paneer.webp
|
||||
---
|
||||
title: Palak Paneer
|
||||
servings: 4
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/pecan-brown-butter-trout.webp
|
||||
---
|
||||
title: Pecan-Brown Butter Trout
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/roasted-pumpkin-sage.webp
|
||||
---
|
||||
title: Roasted Pumpkin Sage Soup
|
||||
description: Pumpkin season is in full swing, and this pumpkin sage soup is the perfect first course for any meal. Totally optional, but I love using hollowed out pumpkins as a bowl for a beautiful presentation.
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/saag-paneer.webp
|
||||
---
|
||||
title: Saag Paneer
|
||||
servings: 1
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/spaghetti-carbonara.webp
|
||||
---
|
||||
title: Spaghetti Carbonara with Bacon & Crisp Lettuce Salad
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/spaghetti-squash-w-chickpeas-and-kale.webp
|
||||
---
|
||||
title: Spaghetti Squash w/ Chickpeas & Kale
|
||||
servings: 3
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/spanakopita.webp
|
||||
---
|
||||
title: Skillet Spanakopita (Greek Spinach Feta Pie)
|
||||
servings: 8
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/spicy-shrimp-paella.webp
|
||||
---
|
||||
title: Spicy Shrimp Paella
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/thai-veggie-pad-see-ew.webp
|
||||
---
|
||||
title: Thai Veggie Pad See Ew
|
||||
servings: 2
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/warm-up-with-a-quick-lemony-chicken-soup.webp
|
||||
---
|
||||
title: Warm Up With a Quick Lemony Chicken Soup
|
||||
servings: 6
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/west-african-peanut-chicken.webp
|
||||
---
|
||||
title: West African Peanut Chicken
|
||||
servings: 2
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
slug="$1"
|
||||
url="$2"
|
||||
|
||||
echo "[$slug] Fetching $url" >&2
|
||||
html=$(curl -sL --max-time 15 "$url" 2>/dev/null) || { echo "[$slug] FETCH FAILED" >&2; echo ""; exit 0; }
|
||||
|
||||
# Try multiple og:image patterns
|
||||
img_url=$(echo "$html" | grep -oP 'property="og:image"\s+content="[^"]*"' | sed 's/.*content="//;s/"//' | head -1)
|
||||
|
||||
if [ -z "$img_url" ]; then
|
||||
img_url=$(echo "$html" | grep -oP 'name="og:image"\s+content="[^"]*"' | sed 's/.*content="//;s/"//' | head -1)
|
||||
fi
|
||||
|
||||
if [ -z "$img_url" ]; then
|
||||
img_url=$(echo "$html" | grep -oP 'content="[^"]*"[^>]*property="og:image"' | sed 's/.*content="//;s/".*//' | head -1)
|
||||
fi
|
||||
|
||||
if [ -n "$img_url" ]; then
|
||||
echo "[$slug] Found: $img_url" >&2
|
||||
echo "$img_url"
|
||||
else
|
||||
echo "[$slug] No og:image found" >&2
|
||||
fi
|
||||
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Final attempt: search DDG text, scrape recipe pages for actual images."""
|
||||
|
||||
import csv
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from ddgs import DDGS
|
||||
|
||||
WORKDIR = Path("/home/jbrechtel/recipes/.worktrees/recipe-images")
|
||||
HEADERS = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
}
|
||||
SKIP_DOMAINS = {"pinterest", "facebook", "instagram", "twitter", "youtube", "tiktok"}
|
||||
MIN_IMAGE_SIZE = 10000 # 10KB minimum for a recipe photo
|
||||
|
||||
|
||||
def search_url(query: str) -> str | None:
|
||||
"""Search and return first relevant recipe page URL."""
|
||||
try:
|
||||
results = list(DDGS().text(query, max_results=8))
|
||||
except Exception as e:
|
||||
print(f" Search error: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
for r in results:
|
||||
url = r.get("href", "")
|
||||
domain = url.split("/")[2] if "//" in url else ""
|
||||
if any(s in domain for s in SKIP_DOMAINS):
|
||||
continue
|
||||
# Prefer recipe sites
|
||||
if any(kw in domain for kw in ["recipe", "cook", "food", "kitchen", "nytimes", "loveandlemons"]):
|
||||
return url
|
||||
|
||||
# Return first non-wikipedia result
|
||||
for r in results:
|
||||
url = r.get("href", "")
|
||||
if "wikipedia.org" not in url:
|
||||
return url
|
||||
return None
|
||||
|
||||
|
||||
def find_recipe_images(url: str) -> list[str]:
|
||||
"""Fetch a page and find actual recipe/food images."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=15, allow_redirects=True)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException:
|
||||
return []
|
||||
|
||||
soup = BeautifulSoup(resp.text, "html.parser")
|
||||
images = []
|
||||
|
||||
# 1. Check JSON-LD for recipe image
|
||||
for script in soup.find_all("script", type="application/ld+json"):
|
||||
try:
|
||||
data = json.loads(script.string)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
continue
|
||||
# Handle list or single
|
||||
items = data if isinstance(data, list) else [data]
|
||||
for item in items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("@type") == "Recipe":
|
||||
img = item.get("image")
|
||||
if isinstance(img, str):
|
||||
images.append(img)
|
||||
elif isinstance(img, list):
|
||||
for i in img:
|
||||
if isinstance(i, str):
|
||||
images.append(i)
|
||||
elif isinstance(i, dict):
|
||||
u = i.get("contentUrl") or i.get("url", "")
|
||||
if u:
|
||||
images.append(u)
|
||||
elif isinstance(img, dict):
|
||||
u = img.get("contentUrl") or img.get("url", "")
|
||||
if u:
|
||||
images.append(u)
|
||||
|
||||
# 2. Check og:image
|
||||
for meta in soup.find_all("meta"):
|
||||
content = meta.get("content", "")
|
||||
if (meta.get("property") == "og:image" or meta.get("name") == "og:image" or meta.get("name") == "twitter:image"):
|
||||
if content and "logo" not in content.lower():
|
||||
images.append(content)
|
||||
|
||||
# 3. Check all img tags for large, relevant images
|
||||
for img in soup.find_all("img"):
|
||||
src = img.get("src") or img.get("data-src") or ""
|
||||
if not src.startswith("http"):
|
||||
continue
|
||||
src_lower = src.lower()
|
||||
if any(kw in src_lower for kw in ["logo", "avatar", "icon", "banner", "pixel", "spacer", "thumb"]):
|
||||
continue
|
||||
# Check for recipe-related class/id
|
||||
parent_classes = ""
|
||||
parent = img.parent
|
||||
if parent:
|
||||
parent_classes = " ".join(parent.get("class", []) or []) + " " + (parent.get("id") or "")
|
||||
if any(kw in parent_classes.lower() for kw in ["logo", "nav", "header", "footer", "menu", "icon"]):
|
||||
continue
|
||||
images.append(src)
|
||||
|
||||
# Deduplicate
|
||||
seen = set()
|
||||
unique = []
|
||||
for img in images:
|
||||
if img not in seen:
|
||||
seen.add(img)
|
||||
unique.append(img)
|
||||
|
||||
return unique
|
||||
|
||||
|
||||
def download_image(url: str, dest: Path) -> bool:
|
||||
"""Download image, validate it's a real image."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=30, stream=True)
|
||||
resp.raise_for_status()
|
||||
ct = resp.headers.get("content-type", "")
|
||||
if not ct.startswith("image/"):
|
||||
return False
|
||||
content = resp.content
|
||||
if len(content) < MIN_IMAGE_SIZE:
|
||||
return False
|
||||
dest.write_bytes(content)
|
||||
return True
|
||||
except requests.RequestException:
|
||||
return False
|
||||
|
||||
|
||||
def get_title(filepath: Path) -> str:
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
m = re.search(r"^title:\s*(.+)$", content, re.MULTILINE)
|
||||
if m:
|
||||
return m.group(1).strip()
|
||||
return filepath.stem.replace("_", " ").replace("-", " ")
|
||||
|
||||
|
||||
def add_image_field(filepath: Path, slug: str):
|
||||
url = f"https://roux.roo.lol/recipe-images/{slug}.webp"
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
if re.search(r"^image:", content, re.MULTILINE):
|
||||
content = re.sub(r"^image:.*$", f"image: {url}", count=1, flags=re.MULTILINE, string=content)
|
||||
else:
|
||||
content = re.sub(r"^---$", f"image: {url}\n---", count=1, flags=re.MULTILINE, string=content)
|
||||
filepath.write_text(content, encoding="utf-8")
|
||||
print(f" Updated {filepath.name}", file=sys.stderr)
|
||||
|
||||
|
||||
def get_source_url(filepath: Path) -> str | None:
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
m = re.search(r"^source:\s*(.+)$", content, re.MULTILINE)
|
||||
if m:
|
||||
val = m.group(1).strip()
|
||||
if val.startswith("http"):
|
||||
return val
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
csv_file = WORKDIR / "docs" / "no-image-has-fm.csv"
|
||||
|
||||
recipes = []
|
||||
with open(csv_file) as f:
|
||||
reader = csv.reader(f, delimiter="|")
|
||||
for row in reader:
|
||||
if len(row) >= 2:
|
||||
recipes.append((row[0].strip(), row[1].strip()))
|
||||
|
||||
success = []
|
||||
failed = []
|
||||
|
||||
for filename, slug in recipes:
|
||||
filepath = WORKDIR / filename
|
||||
if not filepath.exists():
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
|
||||
# Skip if already has image AND file exists
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
has_field = bool(re.search(r"^image:", content, re.MULTILINE))
|
||||
has_file = dest.exists()
|
||||
if has_field and has_file and dest.stat().st_size > 10000:
|
||||
print(f"[{slug}] Already done, skipping", file=sys.stderr)
|
||||
success.append(slug)
|
||||
continue
|
||||
|
||||
title = get_title(filepath)
|
||||
query = f"{title} recipe"
|
||||
|
||||
print(f"[{slug}] Searching: {query}", file=sys.stderr)
|
||||
page_url = search_url(query)
|
||||
if not page_url:
|
||||
print(f" No search results", file=sys.stderr)
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
print(f" Page: {page_url}", file=sys.stderr)
|
||||
time.sleep(0.3)
|
||||
|
||||
img_urls = find_recipe_images(page_url)
|
||||
if not img_urls:
|
||||
print(f" No images found on page", file=sys.stderr)
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
downloaded = False
|
||||
for img_url in img_urls[:5]:
|
||||
print(f" Trying: {img_url[:80]}...", file=sys.stderr)
|
||||
if download_image(img_url, dest):
|
||||
print(f" Downloaded ({dest.stat().st_size} bytes)", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
downloaded = True
|
||||
break
|
||||
time.sleep(0.2)
|
||||
|
||||
if downloaded:
|
||||
success.append(slug)
|
||||
else:
|
||||
failed.append(slug)
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
print(f"\n=== Results ===", file=sys.stderr)
|
||||
print(f"Success: {len(success)}", file=sys.stderr)
|
||||
print(f"Failed: {len(failed)}", file=sys.stderr)
|
||||
if failed:
|
||||
print(f"Failed: {', '.join(failed)}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/tmp/recipe-venv/bin/python3
|
||||
"""Find images for remaining 6 recipes."""
|
||||
import re, sys, time
|
||||
from pathlib import Path
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from ddgs import DDGS
|
||||
|
||||
WORKDIR = Path("/home/jbrechtel/recipes/.worktrees/recipe-images")
|
||||
HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"}
|
||||
|
||||
recipes = [
|
||||
("Arugula Pesto Chicken with Pecans & Roasted Butternut.cook", "arugula-pesto-chicken-with-pecans-roasted-butternut", "Arugula Pesto Chicken Pecans Roasted Butternut recipe garnishandgather"),
|
||||
("Bang Bang Tofu.cook", "bang-bang-tofu", "Bang Bang Tofu recipe"),
|
||||
("Chana Chickpea Salad.cook", "chana-chickpea-salad", "Chana Chickpea Salad recipe"),
|
||||
("Gaja's Bulgogi Lettuce Wraps.cook", "gaja-s-bulgogi-lettuce-wraps", "Gaja Bulgogi Lettuce Wraps recipe garnishandgather"),
|
||||
("Pecan-Brown Butter Trout.cook", "pecan-brown-butter-trout", "Pecan Brown Butter Trout recipe"),
|
||||
("West African Peanut Chicken.cook", "west-african-peanut-chicken", "West African Peanut Chicken recipe"),
|
||||
]
|
||||
|
||||
for file, slug, query in recipes:
|
||||
fp = WORKDIR / file
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
if dest.exists() and dest.stat().st_size > 10000:
|
||||
print(f"[{slug}] Already done")
|
||||
continue
|
||||
|
||||
print(f"[{slug}] Searching: {query}")
|
||||
try:
|
||||
results = list(DDGS().text(query, max_results=5))
|
||||
except Exception as e:
|
||||
print(f" Search error: {e}")
|
||||
continue
|
||||
|
||||
found = False
|
||||
for r in results:
|
||||
url = r.get("href", "")
|
||||
if not url or any(s in url for s in ["pinterest","facebook","instagram"]):
|
||||
continue
|
||||
print(f" Fetching: {url}")
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=15)
|
||||
resp.raise_for_status()
|
||||
except Exception as e:
|
||||
print(f" Failed: {e}")
|
||||
continue
|
||||
|
||||
soup = BeautifulSoup(resp.text, "html.parser")
|
||||
img_urls = []
|
||||
|
||||
# og:image
|
||||
for meta in soup.find_all("meta"):
|
||||
c = meta.get("content", "")
|
||||
p = (meta.get("property") or "").lower()
|
||||
n = (meta.get("name") or "").lower()
|
||||
if ("og:image" in p or "og:image" in n or "twitter:image" in n) and c and "logo" not in c.lower():
|
||||
img_urls.append(c)
|
||||
|
||||
# img tags
|
||||
for img in soup.find_all("img"):
|
||||
src = img.get("src") or img.get("data-src") or ""
|
||||
if src.startswith("http") and "logo" not in src.lower() and "icon" not in src.lower():
|
||||
img_urls.append(src)
|
||||
|
||||
for img_url in img_urls[:5]:
|
||||
try:
|
||||
r2 = requests.get(img_url, headers=HEADERS, timeout=15)
|
||||
r2.raise_for_status()
|
||||
ct = r2.headers.get("content-type", "")
|
||||
if ct.startswith("image/") and len(r2.content) > 10000:
|
||||
dest.write_bytes(r2.content)
|
||||
print(f" Downloaded ({len(r2.content)} bytes): {img_url[:60]}")
|
||||
found = True
|
||||
break
|
||||
except:
|
||||
pass
|
||||
if found:
|
||||
break
|
||||
time.sleep(0.3)
|
||||
|
||||
if found:
|
||||
# Add image field
|
||||
url_val = f"https://roux.roo.lol/recipe-images/{slug}.webp"
|
||||
content = fp.read_text(encoding="utf-8")
|
||||
if re.search(r"^image:", content, re.MULTILINE):
|
||||
content = re.sub(r"^image:.*$", f"image: {url_val}", count=1, flags=re.MULTILINE, string=content)
|
||||
else:
|
||||
content = re.sub(r"^---$", f"image: {url_val}\n---", count=1, flags=re.MULTILINE, string=content)
|
||||
fp.write_text(content, encoding="utf-8")
|
||||
print(f" Updated image field")
|
||||
else:
|
||||
print(f" FAILED")
|
||||
time.sleep(0.5)
|
||||
|
||||
print("\nDone!")
|
||||
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Scrape og:image from recipe source URLs, download images, update .cook files."""
|
||||
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
WORKDIR = Path("/home/jbrechtel/recipes/.worktrees/recipe-images")
|
||||
HEADERS = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
|
||||
|
||||
def get_source_url(filepath: Path) -> str | None:
|
||||
"""Extract the source URL from a .cook file's YAML frontmatter."""
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
match = re.search(r"^source:\s*(.+)$", content, re.MULTILINE)
|
||||
if not match:
|
||||
return None
|
||||
val = match.group(1).strip()
|
||||
# Only return actual URLs
|
||||
if val.startswith("http://") or val.startswith("https://"):
|
||||
return val
|
||||
return None
|
||||
|
||||
|
||||
def fetch_og_image(url: str, timeout: int = 20) -> str | None:
|
||||
"""Fetch a page and extract og:image URL."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException as e:
|
||||
print(f" FETCH FAILED: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
soup = BeautifulSoup(resp.text, "html.parser")
|
||||
|
||||
# Try og:image meta tag
|
||||
for meta in soup.find_all("meta"):
|
||||
if meta.get("property") == "og:image" and meta.get("content"):
|
||||
return meta["content"]
|
||||
if meta.get("name") == "og:image" and meta.get("content"):
|
||||
return meta["content"]
|
||||
|
||||
# Try twitter:image
|
||||
for meta in soup.find_all("meta"):
|
||||
if meta.get("name") == "twitter:image" and meta.get("content"):
|
||||
return meta["content"]
|
||||
|
||||
# Try finding the first large image (common in recipe sites)
|
||||
for img in soup.find_all("img"):
|
||||
src = img.get("src") or ""
|
||||
if "logo" in src.lower() or "avatar" in src.lower() or "icon" in src.lower():
|
||||
continue
|
||||
if src.startswith("http") and any(kw in src.lower() for kw in ["upload", "wp-content", "images"]):
|
||||
return src
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def download_image(url: str, dest: Path) -> bool:
|
||||
"""Download an image from URL to destination path."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=30, stream=True)
|
||||
resp.raise_for_status()
|
||||
dest.write_bytes(resp.content)
|
||||
return True
|
||||
except requests.RequestException as e:
|
||||
print(f" DOWNLOAD FAILED: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def add_image_field(filepath: Path, slug: str):
|
||||
"""Add or update the image field in a .cook file's YAML frontmatter."""
|
||||
url = f"https://roux.roo.lol/recipe-images/{slug}.webp"
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
|
||||
# If image field already exists, replace it
|
||||
if re.search(r"^image:", content, re.MULTILINE):
|
||||
content = re.sub(r"^image:.*$", f"image: {url}", content, count=1, flags=re.MULTILINE)
|
||||
else:
|
||||
# Insert before closing ---
|
||||
content = re.sub(r"^---$", f"image: {url}\n---", content, count=1, flags=re.MULTILINE)
|
||||
|
||||
filepath.write_text(content, encoding="utf-8")
|
||||
print(f" Updated image field: {filepath.name}", file=sys.stderr)
|
||||
|
||||
|
||||
def process_recipe(filepath: Path, slug: str) -> bool:
|
||||
"""Process a single recipe: find image, download, update frontmatter."""
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
|
||||
# Skip if already downloaded
|
||||
if dest.exists():
|
||||
print(f"[{slug}] Image already exists, skipping download", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
|
||||
# Try to get source URL and scrape image
|
||||
src_url = get_source_url(filepath)
|
||||
if src_url:
|
||||
print(f"[{slug}] Scraping {src_url}", file=sys.stderr)
|
||||
img_url = fetch_og_image(src_url)
|
||||
if img_url:
|
||||
print(f" Found: {img_url}", file=sys.stderr)
|
||||
if download_image(img_url, dest):
|
||||
print(f" Downloaded to {dest.name}", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
else:
|
||||
print(f" Download failed", file=sys.stderr)
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
csv_file = WORKDIR / "docs" / "no-image-has-fm.csv"
|
||||
if not csv_file.exists():
|
||||
print(f"CSV not found: {csv_file}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
recipes = []
|
||||
with open(csv_file) as f:
|
||||
reader = csv.reader(f, delimiter="|")
|
||||
for row in reader:
|
||||
if len(row) >= 2:
|
||||
recipes.append((row[0].strip(), row[1].strip()))
|
||||
|
||||
print(f"Processing {len(recipes)} recipes...", file=sys.stderr)
|
||||
|
||||
success = []
|
||||
failed = []
|
||||
|
||||
for filename, slug in recipes:
|
||||
filepath = WORKDIR / filename
|
||||
if not filepath.exists():
|
||||
print(f"[{slug}] File not found: {filepath}", file=sys.stderr)
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
ok = process_recipe(filepath, slug)
|
||||
if ok:
|
||||
success.append(slug)
|
||||
else:
|
||||
failed.append(slug)
|
||||
|
||||
# Be polite to servers
|
||||
time.sleep(0.5)
|
||||
|
||||
print(f"\n=== Results ===", file=sys.stderr)
|
||||
print(f"Success: {len(success)}", file=sys.stderr)
|
||||
print(f"Failed: {len(failed)}", file=sys.stderr)
|
||||
if failed:
|
||||
print(f"Failed slugs: {', '.join(failed)}", file=sys.stderr)
|
||||
|
||||
# Write failed slugs to file for retry
|
||||
if failed:
|
||||
with open(WORKDIR / "docs" / "failed-images.csv", "w") as f:
|
||||
for slug in failed:
|
||||
f.write(f"{slug}\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,233 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search the web for recipe images, download them, and update .cook files.
|
||||
|
||||
Uses DuckDuckGo search to find recipe pages, then scrapes og:image.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from duckduckgo_search import DDGS
|
||||
|
||||
WORKDIR = Path("/home/jbrechtel/recipes/.worktrees/recipe-images")
|
||||
HEADERS = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||
}
|
||||
|
||||
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp", ".gif"}
|
||||
|
||||
|
||||
def fetch_og_image(url: str) -> str | None:
|
||||
"""Fetch a page and extract og:image URL."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=15, allow_redirects=True)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException:
|
||||
return None
|
||||
|
||||
soup = BeautifulSoup(resp.text, "html.parser")
|
||||
|
||||
# Try og:image meta tag (property or name)
|
||||
for meta in soup.find_all("meta"):
|
||||
prop = (meta.get("property") or "").lower()
|
||||
name = (meta.get("name") or "").lower()
|
||||
content = meta.get("content", "")
|
||||
if "og:image" in prop and content:
|
||||
return content
|
||||
if "og:image" in name and content:
|
||||
return content
|
||||
if "twitter:image" in name and content:
|
||||
return content
|
||||
|
||||
# Try schema.org JSON-LD
|
||||
for script in soup.find_all("script", type="application/ld+json"):
|
||||
import json
|
||||
try:
|
||||
data = json.loads(script.string)
|
||||
if isinstance(data, dict):
|
||||
img = data.get("image")
|
||||
if isinstance(img, str) and img.startswith("http"):
|
||||
return img
|
||||
if isinstance(img, dict):
|
||||
url = img.get("contentUrl") or img.get("url", "")
|
||||
if url.startswith("http"):
|
||||
return url
|
||||
if isinstance(img, list) and img:
|
||||
first = img[0]
|
||||
if isinstance(first, str):
|
||||
return first
|
||||
if isinstance(first, dict):
|
||||
url = first.get("contentUrl") or first.get("url", "")
|
||||
if url.startswith("http"):
|
||||
return url
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def search_and_find_image(query: str) -> str | None:
|
||||
"""Search DuckDuckGo for a recipe and find og:image."""
|
||||
try:
|
||||
results = list(DDGS().text(query, max_results=5))
|
||||
except Exception as e:
|
||||
print(f" Search error: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
for r in results:
|
||||
url = r.get("href", "")
|
||||
if not url or "pinterest" in url or "facebook" in url or "instagram" in url:
|
||||
continue
|
||||
print(f" Trying: {url}", file=sys.stderr)
|
||||
img = fetch_og_image(url)
|
||||
if img:
|
||||
print(f" Found image: {img}", file=sys.stderr)
|
||||
return img
|
||||
time.sleep(0.3)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def download_image(url: str, dest: Path) -> bool:
|
||||
"""Download an image from URL to destination path."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=30, stream=True)
|
||||
resp.raise_for_status()
|
||||
dest.write_bytes(resp.content)
|
||||
return True
|
||||
except requests.RequestException as e:
|
||||
print(f" Download failed: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def add_image_field(filepath: Path, slug: str):
|
||||
"""Add or update the image field in a .cook file's YAML frontmatter."""
|
||||
url = f"https://roux.roo.lol/recipe-images/{slug}.webp"
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
|
||||
if re.search(r"^image:", content, re.MULTILINE):
|
||||
content = re.sub(r"^image:.*$", f"image: {url}", count=1, flags=re.MULTILINE, string=content)
|
||||
else:
|
||||
content = re.sub(r"^---$", f"image: {url}\n---", count=1, flags=re.MULTILINE, string=content)
|
||||
|
||||
filepath.write_text(content, encoding="utf-8")
|
||||
print(f" Updated {filepath.name}", file=sys.stderr)
|
||||
|
||||
|
||||
def get_source_url(filepath: Path) -> str | None:
|
||||
"""Extract source URL from .cook frontmatter."""
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
match = re.search(r"^source:\s*(.+)$", content, re.MULTILINE)
|
||||
if match:
|
||||
val = match.group(1).strip()
|
||||
if val.startswith("http://") or val.startswith("https://"):
|
||||
return val
|
||||
return None
|
||||
|
||||
|
||||
def process_recipe(filename: str, slug: str) -> bool:
|
||||
"""Try to find and download an image for a recipe."""
|
||||
filepath = WORKDIR / filename
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
|
||||
if dest.exists():
|
||||
print(f"[{slug}] Already exists", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
|
||||
# Strategy 1: Try source URL if available
|
||||
src_url = get_source_url(filepath)
|
||||
if src_url:
|
||||
print(f"[{slug}] Trying source URL: {src_url}", file=sys.stderr)
|
||||
img = fetch_og_image(src_url)
|
||||
if img:
|
||||
if download_image(img, dest):
|
||||
print(f" Downloaded from source", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
|
||||
# Strategy 2: Search by recipe name
|
||||
# Get the title from frontmatter
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
title_match = re.search(r"^title:\s*(.+)$", content, re.MULTILINE)
|
||||
if title_match:
|
||||
query = title_match.group(1).strip() + " recipe"
|
||||
else:
|
||||
query = filename.replace(".cook", "").replace("_", " ").replace("-", " ") + " recipe"
|
||||
|
||||
print(f"[{slug}] Searching: {query}", file=sys.stderr)
|
||||
img = search_and_find_image(query)
|
||||
if img and download_image(img, dest):
|
||||
print(f" Downloaded from search", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
|
||||
# Strategy 3: Try with "og:image" or "recipe image" variations
|
||||
query2 = query + " image"
|
||||
print(f"[{slug}] Searching (alt): {query2}", file=sys.stderr)
|
||||
img = search_and_find_image(query2)
|
||||
if img and download_image(img, dest):
|
||||
print(f" Downloaded from alt search", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
# First process the no-image-has-fm list
|
||||
csv_file = WORKDIR / "docs" / "no-image-has-fm.csv"
|
||||
if not csv_file.exists():
|
||||
print(f"CSV not found: {csv_file}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
recipes = []
|
||||
with open(csv_file) as f:
|
||||
reader = csv.reader(f, delimiter="|")
|
||||
for row in reader:
|
||||
if len(row) >= 2:
|
||||
recipes.append((row[0].strip(), row[1].strip()))
|
||||
|
||||
success = []
|
||||
failed = []
|
||||
|
||||
for filename, slug in recipes:
|
||||
filepath = WORKDIR / filename
|
||||
if not filepath.exists():
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
if dest.exists() and re.search(r"^image:", filepath.read_text(), re.MULTILINE):
|
||||
print(f"[{slug}] Already processed, skipping", file=sys.stderr)
|
||||
success.append(slug)
|
||||
continue
|
||||
|
||||
ok = process_recipe(filename, slug)
|
||||
if ok:
|
||||
success.append(slug)
|
||||
else:
|
||||
failed.append(slug)
|
||||
|
||||
# Be polite
|
||||
time.sleep(0.5)
|
||||
|
||||
print(f"\n=== Results ===", file=sys.stderr)
|
||||
print(f"Success: {len(success)}", file=sys.stderr)
|
||||
print(f"Failed: {len(failed)}", file=sys.stderr)
|
||||
|
||||
if failed:
|
||||
with open(WORKDIR / "docs" / "failed-images.csv", "w") as f:
|
||||
for s in failed:
|
||||
f.write(f"{s}\n")
|
||||
print(f"Failed: {', '.join(failed)}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search DuckDuckGo Images for recipe photos, download, update .cook files."""
|
||||
|
||||
import csv
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from duckduckgo_search import DDGS
|
||||
|
||||
WORKDIR = Path("/home/jbrechtel/recipes/.worktrees/recipe-images")
|
||||
HEADERS = {
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
|
||||
}
|
||||
|
||||
|
||||
def download_image(url: str, dest: Path) -> bool:
|
||||
"""Download an image from URL to destination path."""
|
||||
try:
|
||||
resp = requests.get(url, headers=HEADERS, timeout=30, stream=True)
|
||||
resp.raise_for_status()
|
||||
# Validate it's actually an image
|
||||
content_type = resp.headers.get("content-type", "")
|
||||
if not content_type.startswith("image/"):
|
||||
print(f" Not an image (content-type: {content_type})", file=sys.stderr)
|
||||
return False
|
||||
dest.write_bytes(resp.content)
|
||||
return True
|
||||
except requests.RequestException as e:
|
||||
print(f" Download failed: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def add_image_field(filepath: Path, slug: str):
|
||||
"""Add or update the image field in a .cook file."""
|
||||
url = f"https://roux.roo.lol/recipe-images/{slug}.webp"
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
|
||||
if re.search(r"^image:", content, re.MULTILINE):
|
||||
content = re.sub(r"^image:.*$", f"image: {url}", count=1, flags=re.MULTILINE, string=content)
|
||||
else:
|
||||
content = re.sub(r"^---$", f"image: {url}\n---", count=1, flags=re.MULTILINE, string=content)
|
||||
|
||||
filepath.write_text(content, encoding="utf-8")
|
||||
print(f" Updated {filepath.name}", file=sys.stderr)
|
||||
|
||||
|
||||
def search_images(query: str, max_results: int = 5):
|
||||
"""Search DuckDuckGo Images for a recipe and return image URLs."""
|
||||
try:
|
||||
results = list(DDGS().images(query, max_results=max_results))
|
||||
return [r["image"] for r in results if r.get("image")]
|
||||
except Exception as e:
|
||||
print(f" Image search error: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
def get_title(filepath: Path) -> str:
|
||||
"""Extract title from .cook frontmatter."""
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
match = re.search(r"^title:\s*(.+)$", content, re.MULTILINE)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
# Fall back to filename
|
||||
name = filepath.stem
|
||||
return name.replace("_", " ").replace("-", " ")
|
||||
|
||||
|
||||
def main():
|
||||
csv_file = WORKDIR / "docs" / "failed-images.csv"
|
||||
# Also check which are actually missing
|
||||
existing_csv = WORKDIR / "docs" / "no-image-has-fm.csv"
|
||||
|
||||
recipes = []
|
||||
with open(existing_csv) as f:
|
||||
reader = csv.reader(f, delimiter="|")
|
||||
for row in reader:
|
||||
if len(row) >= 2:
|
||||
recipes.append((row[0].strip(), row[1].strip()))
|
||||
|
||||
# Filter to those without an image file or without image field
|
||||
to_process = []
|
||||
for filename, slug in recipes:
|
||||
filepath = WORKDIR / filename
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
has_image_file = dest.exists()
|
||||
has_image_field = False
|
||||
if filepath.exists():
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
has_image_field = bool(re.search(r"^image:", content, re.MULTILINE))
|
||||
|
||||
if not has_image_file or not has_image_field:
|
||||
to_process.append((filename, slug))
|
||||
|
||||
print(f"Processing {len(to_process)} remaining recipes...", file=sys.stderr)
|
||||
|
||||
success = []
|
||||
failed = []
|
||||
|
||||
for filename, slug in to_process:
|
||||
filepath = WORKDIR / filename
|
||||
dest = WORKDIR / "recipe-images" / f"{slug}.webp"
|
||||
|
||||
if not filepath.exists():
|
||||
failed.append(slug)
|
||||
continue
|
||||
|
||||
title = get_title(filepath)
|
||||
queries = [
|
||||
f"{title} recipe",
|
||||
title,
|
||||
]
|
||||
|
||||
found = False
|
||||
for query in queries:
|
||||
print(f"[{slug}] Searching images: {query}", file=sys.stderr)
|
||||
urls = search_images(query)
|
||||
for img_url in urls:
|
||||
# Skip tiny icons, logos
|
||||
if any(kw in img_url.lower() for kw in ["logo", "icon", "avatar", "merriam", "wikipedia"]):
|
||||
continue
|
||||
print(f" Trying: {img_url[:80]}...", file=sys.stderr)
|
||||
if download_image(img_url, dest):
|
||||
print(f" Downloaded!", file=sys.stderr)
|
||||
add_image_field(filepath, slug)
|
||||
found = True
|
||||
break
|
||||
time.sleep(0.2)
|
||||
if found:
|
||||
break
|
||||
time.sleep(0.5)
|
||||
|
||||
if found:
|
||||
success.append(slug)
|
||||
else:
|
||||
failed.append(slug)
|
||||
|
||||
print(f"\n=== Results ===", file=sys.stderr)
|
||||
print(f"Success: {len(success)}", file=sys.stderr)
|
||||
print(f"Failed: {len(failed)}", file=sys.stderr)
|
||||
if failed:
|
||||
print(f"Failed: {', '.join(failed)}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,3 +1,4 @@
|
||||
image: https://roux.roo.lol/recipe-images/cauliflower-milanese.webp
|
||||
---
|
||||
title: Cauliflower Milanese
|
||||
source: https://cooking.nytimes.com/recipes/1026145-cauliflower-milanese
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
Alain-Ducase-Bread.cook|alain-ducase-bread
|
||||
Arugula Pesto Chicken with Pecans & Roasted Butternut.cook|arugula-pesto-chicken-with-pecans-roasted-butternut
|
||||
Arugula Pesto Wheat Berries Recipe.cook|arugula-pesto-wheat-berries-recipe
|
||||
Asparagus Salad.cook|asparagus-salad
|
||||
Authentic Cacio e Pepe Recipe.cook|authentic-cacio-e-pepe-recipe
|
||||
Baked Chile Relleno with Black Beans.cook|baked-chile-relleno-with-black-beans
|
||||
Bang Bang Tofu.cook|bang-bang-tofu
|
||||
Barbecue Chicken with Crispy Pan-Fried Okra & Cheddar Biscuits.cook|barbecue-chicken-with-crispy-pan-fried-okra-cheddar-biscuits
|
||||
Black Bean and Corn Salad.cook|black-bean-and-corn-salad
|
||||
Blackened Shrimp and Gouda Grits.cook|blackened-shrimp-and-gouda-grits
|
||||
@@ -0,0 +1,10 @@
|
||||
Butternut Turkey Chili.cook|butternut-turkey-chili-2
|
||||
BUTTERNUT & TURKEY CHILI with sweet peppers, spiced pepitas & queso fresco.cook|butternut-turkey-chili-with-sweet-peppers-spiced-pepitas-queso-fresco
|
||||
Cabbage_Salad.cook|cabbage-salad
|
||||
cauliflower-milanese.cook|cauliflower-milanese
|
||||
Chana Chickpea Salad.cook|chana-chickpea-salad
|
||||
Chicken and Pumpkin Gnocchi with Ricotta.cook|chicken-and-pumpkin-gnocchi-with-ricotta
|
||||
Chickpea Butter Curry.cook|chickpea-butter-curry
|
||||
Chickpea_Butter_Curry.cook|chickpea-butter-curry-2
|
||||
Chinese Orange Shrimp.cook|chinese-orange-shrimp
|
||||
Chocolate_Mousse_Cake_with_Raspberries.cook|chocolate-mousse-cake-with-raspberries
|
||||
@@ -0,0 +1,10 @@
|
||||
Cilantro Pesto Shrimp Over Orzo.cook|cilantro-pesto-shrimp-over-orzo
|
||||
Crispy Roasted Chickpeas (Garbanzo Beans).cook|crispy-roasted-chickpeas-garbanzo-beans
|
||||
Crispy_Roasted_Chickpeas_(Garbanzo_Beans).cook|crispy-roasted-chickpeas-garbanzo-beans-2
|
||||
Dried Fava Soup.cook|dried-fava-soup
|
||||
Easy Lemon Rosemary White Bean Soup.cook|easy-lemon-rosemary-white-bean-soup
|
||||
Easy_Lemon_Rosemary_White_Bean_Soup.cook|easy-lemon-rosemary-white-bean-soup-2
|
||||
Gaja's Bulgogi Lettuce Wraps.cook|gaja-s-bulgogi-lettuce-wraps
|
||||
Gochugaru Honey Salmon.cook|gochugaru-honey-salmon
|
||||
Indian Spiced Chicken.cook|indian-spiced-chicken-2
|
||||
Kale and Chestnut Stew.cook|kale-and-chestnut-stew
|
||||
@@ -0,0 +1,10 @@
|
||||
Kkanpoong Tofu.cook|kkanpoong-tofu
|
||||
Korean Shrimp Tacos.cook|korean-shrimp-tacos
|
||||
Kung Pao Shrimp with Sichuan Roasted Green Beans.cook|kung-pao-shrimp-with-sichuan-roasted-green-beans
|
||||
Lactation cookies.cook|lactation-cookies
|
||||
Lactation_cookies.cook|lactation-cookies-2
|
||||
Moroccan_Spiced_Lentil_and_Chickpea_Soup.cook|moroccan-spiced-lentil-and-chickpea-soup
|
||||
Moroccan Spiced Lentil & Chickpea Soup.cook|moroccan-spiced-lentil-chickpea-soup
|
||||
Orange Rolls.cook|orange-rolls
|
||||
Palak Paneer.cook|palak-paneer
|
||||
Pecan-Brown Butter Trout.cook|pecan-brown-butter-trout
|
||||
@@ -0,0 +1,10 @@
|
||||
Roasted Pumpkin Sage.cook|roasted-pumpkin-sage
|
||||
Saag Paneer.cook|saag-paneer
|
||||
sheet-pan-miso-chicken-with-radishes-and-lime.cook|sheet-pan-miso-chicken-with-radishes-and-lime
|
||||
Spaghetti Carbonara.cook|spaghetti-carbonara
|
||||
Spaghetti_Squash_w_Chickpeas_and_Kale.cook|spaghetti-squash-w-chickpeas-and-kale
|
||||
Spanakopita.cook|spanakopita
|
||||
Spicy Shrimp Paella.cook|spicy-shrimp-paella
|
||||
Thai_Veggie_Pad_See_Ew.cook|thai-veggie-pad-see-ew
|
||||
Warm Up With a Quick Lemony Chicken Soup.cook|warm-up-with-a-quick-lemony-chicken-soup
|
||||
West African Peanut Chicken.cook|west-african-peanut-chicken
|
||||
@@ -0,0 +1,5 @@
|
||||
arugula-pesto-chicken-with-pecans-roasted-butternut|Arugula Pesto Chicken with Pecans & Roasted Butternut
|
||||
asparagus-salad|Asparagus Salad loveandlemons asparagus salad
|
||||
baked-chile-relleno-with-black-beans|Baked Chile Relleno with Black Beans
|
||||
bang-bang-tofu|Bang Bang Tofu
|
||||
barbecue-chicken-with-crispy-pan-fried-okra-cheddar-biscuits|Barbecue Chicken with Crispy Pan-Fried Okra & Cheddar Biscuits
|
||||
@@ -0,0 +1,5 @@
|
||||
blackened-shrimp-and-gouda-grits|Blackened Shrimp and Gouda Grits
|
||||
butternut-turkey-chili-2|Butternut Turkey Chili
|
||||
butternut-turkey-chili-with-sweet-peppers-spiced-pepitas-queso-fresco|BUTTERNUT & TURKEY CHILI with sweet peppers, spiced pepitas & queso fresco
|
||||
chana-chickpea-salad|Chana Chickpea Salad
|
||||
chicken-and-pumpkin-gnocchi-with-ricotta|Chicken and Pumpkin Gnocchi with Ricotta
|
||||
@@ -0,0 +1,5 @@
|
||||
chickpea-butter-curry|Chickpea Butter Curry
|
||||
chickpea-butter-curry-2|Chickpea Butter Curry
|
||||
chinese-orange-shrimp|Chinese Orange Shrimp
|
||||
chocolate-mousse-cake-with-raspberries|Chocolate Mousse Cake with Raspberries
|
||||
cilantro-pesto-shrimp-over-orzo|Cilantro Pesto Shrimp Over Orzo
|
||||
@@ -0,0 +1,5 @@
|
||||
crispy-roasted-chickpeas-garbanzo-beans|Crispy Roasted Chickpeas (Garbanzo Beans)
|
||||
crispy-roasted-chickpeas-garbanzo-beans-2|Crispy Roasted Chickpeas (Garbanzo Beans)
|
||||
dried-fava-soup|Dried Fava Soup 101cookbooks
|
||||
easy-lemon-rosemary-white-bean-soup|Easy Lemon Rosemary White Bean Soup
|
||||
easy-lemon-rosemary-white-bean-soup-2|Easy Lemon Rosemary White Bean Soup
|
||||
@@ -0,0 +1,5 @@
|
||||
gaja-s-bulgogi-lettuce-wraps|Gaja's Bulgogi Lettuce Wraps
|
||||
gochugaru-honey-salmon|Gochugaru Honey Salmon
|
||||
indian-spiced-chicken-2|Indian Spiced Chicken
|
||||
kale-and-chestnut-stew|Kale and Chestnut Stew garnishandgather
|
||||
korean-shrimp-tacos|Korean Shrimp Tacos
|
||||
@@ -0,0 +1,5 @@
|
||||
kung-pao-shrimp-with-sichuan-roasted-green-beans|Kung Pao Shrimp with Sichuan Roasted Green Beans
|
||||
lactation-cookies|Lactation cookies
|
||||
lactation-cookies-2|Lactation cookies
|
||||
moroccan-spiced-lentil-and-chickpea-soup|Moroccan Spiced Lentil and Chickpea Soup
|
||||
moroccan-spiced-lentil-chickpea-soup|Moroccan Spiced Lentil and Chickpea Soup
|
||||
@@ -0,0 +1,5 @@
|
||||
palak-paneer|Palak Paneer
|
||||
pecan-brown-butter-trout|Pecan Brown Butter Trout
|
||||
saag-paneer|Saag Paneer
|
||||
spaghetti-carbonara|Spaghetti Carbonara
|
||||
spaghetti-squash-w-chickpeas-and-kale|Spaghetti Squash with Chickpeas and Kale loveandlemons
|
||||
@@ -0,0 +1,5 @@
|
||||
spanakopita|Spanakopita
|
||||
spicy-shrimp-paella|Spicy Shrimp Paella
|
||||
thai-veggie-pad-see-ew|Thai Veggie Pad See Ew
|
||||
warm-up-with-a-quick-lemony-chicken-soup|Lemony Chicken Soup
|
||||
west-african-peanut-chicken|West African Peanut Chicken
|
||||
@@ -0,0 +1,27 @@
|
||||
arugula-pesto-chicken-with-pecans-roasted-butternut
|
||||
asparagus-salad
|
||||
baked-chile-relleno-with-black-beans
|
||||
bang-bang-tofu
|
||||
barbecue-chicken-with-crispy-pan-fried-okra-cheddar-biscuits
|
||||
blackened-shrimp-and-gouda-grits
|
||||
butternut-turkey-chili-2
|
||||
chana-chickpea-salad
|
||||
chicken-and-pumpkin-gnocchi-with-ricotta
|
||||
chickpea-butter-curry
|
||||
chickpea-butter-curry-2
|
||||
chocolate-mousse-cake-with-raspberries
|
||||
cilantro-pesto-shrimp-over-orzo
|
||||
crispy-roasted-chickpeas-garbanzo-beans-2
|
||||
easy-lemon-rosemary-white-bean-soup
|
||||
gaja-s-bulgogi-lettuce-wraps
|
||||
kale-and-chestnut-stew
|
||||
korean-shrimp-tacos
|
||||
kung-pao-shrimp-with-sichuan-roasted-green-beans
|
||||
moroccan-spiced-lentil-chickpea-soup
|
||||
palak-paneer
|
||||
pecan-brown-butter-trout
|
||||
spaghetti-squash-w-chickpeas-and-kale
|
||||
spanakopita
|
||||
spicy-shrimp-paella
|
||||
thai-veggie-pad-see-ew
|
||||
west-african-peanut-chicken
|
||||
|
|
After Width: | Height: | Size: 140 KiB |
|
After Width: | Height: | Size: 26 KiB |
|
After Width: | Height: | Size: 40 KiB |
|
After Width: | Height: | Size: 1.0 MiB |
|
After Width: | Height: | Size: 242 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 200 KiB |
|
After Width: | Height: | Size: 428 KiB |
|
After Width: | Height: | Size: 772 KiB |
|
After Width: | Height: | Size: 62 KiB |
|
After Width: | Height: | Size: 274 KiB |
|
After Width: | Height: | Size: 626 KiB |
|
After Width: | Height: | Size: 783 KiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 162 KiB |
|
After Width: | Height: | Size: 106 KiB |
|
After Width: | Height: | Size: 222 KiB |
|
After Width: | Height: | Size: 167 KiB |
|
After Width: | Height: | Size: 118 KiB |
|
After Width: | Height: | Size: 169 KiB |
|
After Width: | Height: | Size: 1.0 MiB |
|
After Width: | Height: | Size: 544 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 15 KiB |
|
After Width: | Height: | Size: 72 KiB |
|
After Width: | Height: | Size: 123 KiB |
|
After Width: | Height: | Size: 139 KiB |
|
After Width: | Height: | Size: 45 KiB |
|
After Width: | Height: | Size: 637 KiB |
|
After Width: | Height: | Size: 89 KiB |