Files
spine/docs/plans/2026-06-20-spine-org-data-model.md

738 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Spine Data Model & Index Page — Implementation Plan
**Goal:** Build the model layer (`spine-books`) and index page view so the app reads a real `spine.org` file and displays books grouped by status.
**Architecture:** Model/view split. `spine-books` parses `spine.org` into plists via `org-element` + `org-entry-get`. `spine-render-index` takes those plists and an optional selected ID, returns HTML matching mockup A's class structure. A new `/` handler replaces the `/hello` smoke test. A `spine-skip-server-start` guard lets tests load `spine.el` without starting the HTTP server.
**Tech Stack:** Emacs Lisp, `org-element` (built-in), `org-entry-get` (built-in), `ert` (built-in testing).
---
## File structure
| File | Responsibility |
|------|---------------|
| `sample-books.org` | Test data: 5 books across all statuses |
| `spine.el` | Modified: new functions, handler, skip-server-start guard |
| `test/spine-books-test.el` | ERT tests for `spine-books` |
---
### Task 1: Create sample test data
**Files:**
- Create: `sample-books.org`
- [ ] **Step 1: Write `sample-books.org`**
Five books covering all four statuses, plus an entry with minimal properties.
```org
#+TODO: WANT(w) READING(r) | READ(d) ABANDONED(a)
* WANT The Left Hand of Darkness
:PROPERTIES:
:AUTHOR: Ursula K. Le Guin
:ISBN: 978-0441478125
:COVER: covers/left-hand.jpg
:FORMAT: ebook
:REC_BY: Alex
:REC_NOTE: Le Guin at her most human
:RATING:
:ADDED: [2026-04-01]
:ID: 9a2b-lhd
:END:
* WANT Dune
:PROPERTIES:
:AUTHOR: Frank Herbert
:ISBN: 978-0441172719
:COVER: covers/dune.jpg
:FORMAT: hardcover
:REC_BY: Everyone
:REC_NOTE:
:RATING:
:ADDED: [2026-05-10]
:ID: 7f3c-dun
:END:
* WANT Piranesi
:PROPERTIES:
:AUTHOR: Susanna Clarke
:ISBN:
:COVER:
:FORMAT:
:REC_BY:
:REC_NOTE:
:RATING:
:ADDED: [2026-06-01]
:ID: 5e8d-pir
:END:
* READING Use of Weapons
:PROPERTIES:
:AUTHOR: Iain M. Banks
:ISBN: 978-0316029193
:COVER: covers/use-of-weapons.jpg
:FORMAT: audiobook
:REC_BY: Priya
:REC_NOTE: If you liked Player of Games, this one will wreck you
:RATING:
:ADDED: [2026-02-20]
:ID: 8c1e-uow
:END:
:LOGBOOK:
- State "READING" from "WANT" [2026-03-01]
:END:
- [2026-03-02] The two-track structure is doing something I can't name yet.
- [2026-03-07] Zakalwe's competence reads as a wound.
- [2026-03-09] The chapter numbering. Oh.
* READ Ancillary Justice
:PROPERTIES:
:AUTHOR: Ann Leckie
:ISBN: 978-0316246620
:COVER: covers/ancillary.jpg
:FORMAT: ebook
:REC_BY:
:REC_NOTE:
:RATING: 5
:ADDED: [2025-11-15]
:ID: 3a1b-aj
:END:
:LOGBOOK:
- State "READ" from "READING" [2026-01-10]
- State "READING" from "WANT" [2025-11-20]
:END:
- [2025-11-22] The pronoun game is a genuinely fresh take on identity.
- [2025-12-05] Justice of Toren's multi-body perspective is unsettling.
- [2026-01-08] The tea set. Perfect ending.
```
This file exercises:
- **WANT books**: 3 (Left Hand, Dune, Piranesi) — tests group header, count, different property densities
- **READING books**: 1 (Use of Weapons) — tests notes extraction, recommendation trail
- **READ books**: 1 (Ancillary Justice) — tests rating, LOGBOOK
- **ABANDONED books**: 0 — tests empty group omission
- **Minimal properties**: Piranesi has no ISBN, cover, format, recommendation, or notes
- [ ] **Step 2: Verify file exists**
Run: `cat sample-books.org | head -3`
Expected: `#+TODO: WANT(w) READING(r) | READ(d) ABANDONED(a)`
- [ ] **Step 3: Commit**
```bash
git add sample-books.org
git commit -m "test: add sample-books.org test data"
```
---
### Task 2: Add server-start guard to spine.el
**Files:**
- Modify: `spine.el`
- [ ] **Step 1: Add `spine-skip-server-start` variable and wrap server start**
Insert after the template rendering section (after line 44, before `;; --- handlers`):
```elisp
(defvar spine-skip-server-start nil
"When non-nil, do not start the HTTP server.
Set by test harnesses that only need the model functions.")
```
Change the server start block (lines 5358) to:
```elisp
;; --- start ------------------------------------------------------------
(unless spine-skip-server-start
(setq httpd-host spine-host)
(setq httpd-port spine-port)
(httpd-start)
(message "Spine listening on http://%s:%d"
(or spine-host "localhost") spine-port))
```
- [ ] **Step 2: Verify batch load with guard**
Run: `emacs --quick --batch --eval "(setq spine-skip-server-start t)" --load spine.el`
Expected: exits cleanly, no "Spine listening" message, no errors.
- [ ] **Step 3: Verify server still works normally without guard**
Run the normal `scripts/run` flow and curl `/hello``/hello` still exists at this point since the handler hasn't been replaced yet.
```bash
scripts/run 8080
sleep 3
curl -s http://localhost:8080/hello
scripts/stop
```
Expected: HTML containing `<h1>Hello from Emacs.</h1>`.
- [ ] **Step 4: Commit**
```bash
git add spine.el
git commit -m "feat: add spine-skip-server-start guard for testing"
```
---
### Task 3: Implement spine-books model function
**Files:**
- Modify: `spine.el`
- [ ] **Step 1: Add `cl-lib` require**
At the top of `spine.el`, after `(require 'ht)`, add:
```elisp
(require 'cl-lib)
```
`cl-lib` is built into Emacs and provides `cl-return-from`.
- [ ] **Step 2: Add `spine-org-file` config variable**
After `spine-port` and `spine-host` defvars (after line 27), add:
```elisp
(defvar spine-org-file "~/spine/books.org"
"Path to the spine.org data file.")
```
- [ ] **Step 3: Add `spine--h` HTML escaping helper**
After `spine-render`, add:
```elisp
(defun spine--h (text)
"HTML-escape TEXT."
(when text
(let ((s text))
(setq s (replace-regexp-in-string "&" "&amp;" s))
(setq s (replace-regexp-in-string "<" "&lt;" s))
(setq s (replace-regexp-in-string ">" "&gt;" s))
(setq s (replace-regexp-in-string "\"" "&quot;" s))
s)))
```
- [ ] **Step 4: Add `spine--extract-notes` helper**
After `spine--h`, add:
```elisp
(defun spine--extract-notes (hl-begin hl-end)
"Extract reading notes from headline body between HL-BEGIN and HL-END.
Returns list of (date-string text-string) pairs."
(let ((notes nil))
(save-excursion
(goto-char hl-begin)
;; Skip past PROPERTIES and LOGBOOK drawers
(while (re-search-forward
"^[ \t]*:\\(PROPERTIES\\|LOGBOOK\\):" hl-end t)
(re-search-forward "^[ \t]*:END:" hl-end t))
;; Collect note lines: "- [YYYY-MM-DD] text"
(while (re-search-forward
"^- \\[\\([0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\}\\)\\] \\(.*\\)"
hl-end t)
(push (list (match-string 1) (match-string 2)) notes)))
(nreverse notes)))
```
- [ ] **Step 5: Add `spine-books` function**
After `spine--extract-notes`, add:
```elisp
(defun spine-books ()
"Return a list of plists, one per top-level headline in `spine-org-file'.
Returns nil if the file is missing or cannot be parsed."
(unless (file-exists-p spine-org-file)
(message "spine-books: file not found: %s" spine-org-file)
(cl-return-from spine-books nil))
(condition-case err
(with-temp-buffer
(insert-file-contents spine-org-file)
(org-mode)
(let ((books nil))
(org-element-map (org-element-parse-buffer 'headline) 'headline
(lambda (hl)
(when (= (org-element-property :level hl) 1)
(let ((pos (org-element-property :begin hl)))
(push (list :id (save-excursion
(goto-char pos)
(org-entry-get nil "ID"))
:title (org-element-property :title hl)
:status (org-element-property :todo-keyword hl)
:author (save-excursion
(goto-char pos)
(org-entry-get nil "AUTHOR"))
:isbn (save-excursion
(goto-char pos)
(org-entry-get nil "ISBN"))
:cover (save-excursion
(goto-char pos)
(org-entry-get nil "COVER"))
:format (save-excursion
(goto-char pos)
(org-entry-get nil "FORMAT"))
:rec_by (save-excursion
(goto-char pos)
(org-entry-get nil "REC_BY"))
:rec_note (save-excursion
(goto-char pos)
(org-entry-get nil "REC_NOTE"))
:rating (save-excursion
(goto-char pos)
(org-entry-get nil "RATING"))
:added (save-excursion
(goto-char pos)
(org-entry-get nil "ADDED"))
:tags (org-element-property :tags hl)
:notes (spine--extract-notes
(org-element-property :begin hl)
(org-element-property :end hl)))
books)))))
(nreverse books)))
(error
(message "spine-books: failed to parse %s: %s"
spine-org-file (error-message-string err))
nil)))
```
- [ ] **Step 6: Verify batch load still works**
Run: `emacs --quick --batch --eval "(setq spine-skip-server-start t)" --load spine.el`
Expected: exits cleanly.
- [ ] **Step 7: Verify spine-books returns data from sample file**
Run:
```bash
emacs --quick --batch \
--eval "(setq spine-skip-server-start t spine-org-file \"sample-books.org\")" \
--load spine.el \
--eval "(progn (let ((books (spine-books))) (message \"Found %d books\" (length books))))"
```
Expected: `Found 5 books` in output.
- [ ] **Step 8: Commit**
```bash
git add spine.el
git commit -m "feat: add spine-books model function"
```
---
### Task 4: Test spine-books with ERT
**Files:**
- Create: `test/spine-books-test.el`
- [ ] **Step 1: Create `test/` directory**
```bash
mkdir -p test
```
- [ ] **Step 2: Write `test/spine-books-test.el`**
```elisp
;;; spine-books-test.el — ERT tests for spine-books
(require 'ert)
(require 'cl-lib)
;; Load spine.el without starting the server
(setq spine-skip-server-start t)
(setq spine-org-file (expand-file-name "sample-books.org"
(file-name-directory load-file-name)))
(load-file (expand-file-name "spine.el"
(file-name-directory load-file-name)))
(ert-deftest spine-books-parses-sample ()
"spine-books returns correct plists for sample-books.org."
(let ((books (spine-books)))
(should books)
(should (= (length books) 5))))
(ert-deftest spine-books-first-book-has-all-fields ()
"Use of Weapons entry has all expected fields."
(let* ((books (spine-books))
(uow (cl-find "8c1e-uow" books
:key (lambda (b) (plist-get b :id))
:test #'equal)))
(should uow)
(should (equal (plist-get uow :title) "Use of Weapons"))
(should (equal (plist-get uow :status) "READING"))
(should (equal (plist-get uow :author) "Iain M. Banks"))
(should (equal (plist-get uow :isbn) "978-0316029193"))
(should (equal (plist-get uow :cover) "covers/use-of-weapons.jpg"))
(should (equal (plist-get uow :format) "audiobook"))
(should (equal (plist-get uow :rec_by) "Priya"))
(should (equal (plist-get uow :rec_note) "If you liked Player of Games, this one will wreck you"))
(should (equal (plist-get uow :tags) '("scifi" "literary")))
(should (equal (plist-get uow :notes)
'(("2026-03-02" "The two-track structure is doing something I can't name yet.")
("2026-03-07" "Zakalwe's competence reads as a wound.")
("2026-03-09" "The chapter numbering. Oh."))))))
(ert-deftest spine-books-missing-properties-are-nil ()
"Piranesi entry has nil for non-present properties."
(let* ((books (spine-books))
(pir (cl-find "5e8d-pir" books
:key (lambda (b) (plist-get b :id))
:test #'equal)))
(should pir)
(should (equal (plist-get pir :title) "Piranesi"))
(should (equal (plist-get pir :status) "WANT"))
(should (equal (plist-get pir :author) "Susanna Clarke"))
;; ISBN, cover, format, rec_by, rec_note should all be nil or ""
(should-not (plist-get pir :isbn))
(should-not (plist-get pir :cover))
(should-not (plist-get pir :format))
(should-not (plist-get pir :rec_by))
(should-not (plist-get pir :rec_note))
;; Notes should be empty
(should-not (plist-get pir :notes))))
(ert-deftest spine-books-read-book-has-rating ()
"Ancillary Justice has rating 5."
(let* ((books (spine-books))
(aj (cl-find "3a1b-aj" books
:key (lambda (b) (plist-get b :id))
:test #'equal)))
(should aj)
(should (equal (plist-get aj :status) "READ"))
(should (equal (plist-get aj :rating) "5"))
(should (= (length (plist-get aj :notes)) 3))))
(ert-deftest spine-books-empty-file-returns-empty-list ()
"Empty Org file returns empty list, not nil."
(let* ((tmp-file (make-temp-file "spine-test-" nil ".org"))
(spine-org-file tmp-file))
(unwind-protect
(let ((books (spine-books)))
(should (listp books))
(should (= (length books) 0)))
(delete-file tmp-file))))
(ert-deftest spine-books-missing-file-returns-nil ()
"Non-existent file returns nil."
(let ((spine-org-file "/tmp/spine-nonexistent-12345.org"))
(should-not (spine-books))))
```
- [ ] **Step 3: Run the tests**
```bash
emacs --quick --batch \
--directory "$(pwd)" \
--load test/spine-books-test.el \
-f ert-run-tests-batch-and-exit
```
Expected: all 6 tests pass.
- [ ] **Step 4: Commit**
```bash
git add test/spine-books-test.el
git commit -m "test: add ERT tests for spine-books"
```
---
### Task 5: Implement view functions
**Files:**
- Modify: `spine.el`
- [ ] **Step 1: Add `spine-render-index` function**
After `spine-books`, add:
```elisp
(defun spine-render-index (books &optional selected-id)
"Render the index page HTML for BOOKS, expanding the book matching SELECTED-ID."
(let ((order '("WANT" "READING" "READ" "ABANDONED"))
(grouped (make-hash-table :test 'equal)))
;; Group books by status
(dolist (book books)
(let* ((status (or (plist-get book :status) "WANT")))
(push book (gethash status grouped))))
;; Render
(with-temp-buffer
(insert "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n")
(insert "<meta charset=\"utf-8\">\n")
(insert "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n")
(insert "<title>Spine</title>\n</head>\n<body>\n")
(insert "<div class=\"frame\">\n")
(insert "<div class=\"titlebar\">Spine</div>\n")
;; Each group in order
(dolist (status order)
(let ((group-books (nreverse (gethash status grouped))))
(when group-books
(insert (format "<div class=\"group\">%s (%d)</div>\n"
status (length group-books)))
(dolist (book group-books)
(let ((selected (equal (plist-get book :id) selected-id)))
(insert (format "<div class=\"row%s\">\n"
(if selected " selected" "")))
;; Format glyph
(let ((fmt (plist-get book :format)))
(insert (format "<span class=\"glyph\">%s</span>\n"
(cond ((equal fmt "hardcover") "[H]")
((equal fmt "ebook") "[E]")
((equal fmt "audiobook") "[A]")
(t "")))))
;; Title + author
(insert "<span class=\"title\">")
(insert (spine--h (plist-get book :title)))
(let ((author (plist-get book :author)))
(when author
(insert (format " <span class=\"muted\">%s</span>"
(spine--h author)))))
(insert "</span>\n")
;; Tags
(let ((tags (plist-get book :tags)))
(when tags
(insert (format "<span class=\"tags\">%s</span>\n"
(mapconcat (lambda (t) (concat ":" t ":"))
tags " ")))))
;; Recommendation trail
(let ((rec-by (plist-get book :rec_by))
(rec-note (plist-get book :rec_note)))
(when (and rec-by (> (length rec-by) 0))
(insert
(format "<span class=\"trail\">%s%s</span>\n"
(spine--h rec-by)
(if (and rec-note (> (length rec-note) 0))
(concat ": " (spine--h rec-note))
"")))))
;; Status pill
(let ((status (or (plist-get book :status) "WANT")))
(insert (format "<span class=\"pill %s\">%s</span>\n"
(downcase status) status)))
;; Expanded detail
(when selected
(insert "<div class=\"detail\">\n")
(dolist (note (plist-get book :notes))
(insert (format
"<div class=\"logline\"><span class=\"logdate\">[%s]</span> %s</div>\n"
(spine--h (car note))
(spine--h (cadr note)))))
(let ((rating (plist-get book :rating)))
(when (and rating (> (length rating) 0))
(insert (format "<div>Rating: %s/5</div>\n"
(spine--h rating)))))
(let ((isbn (plist-get book :isbn)))
(when (and isbn (> (length isbn) 0))
(insert (format "<div>ISBN: %s</div>\n"
(spine--h isbn)))))
(let ((fmt (plist-get book :format)))
(when (and fmt (> (length fmt) 0))
(insert (format "<div>Format: %s</div>\n"
(spine--h fmt)))))
(let ((added (plist-get book :added)))
(when (and added (> (length added) 0))
(insert (format "<div>Added: %s</div>\n"
(spine--h added)))))
(insert "</div>\n"))
(insert "</div>\n"))))))
(insert "</div>\n</body>\n</html>\n")
(buffer-string))))
```
- [ ] **Step 2: Add `spine-render-empty-state` function**
After `spine-render-index`, add:
```elisp
(defun spine-render-empty-state ()
"Render a page indicating no books file was found."
(concat "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n"
"<meta charset=\"utf-8\">\n"
"<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n"
"<title>Spine</title>\n</head>\n<body>\n"
"<div class=\"frame\">\n"
"<div class=\"titlebar\">Spine</div>\n"
"<div class=\"row\">No books yet.</div>\n"
"</div>\n</body>\n</html>\n"))
```
- [ ] **Step 3: Verify batch load**
Run: `emacs --quick --batch --eval "(setq spine-skip-server-start t)" --load spine.el`
Expected: exits cleanly, no errors.
- [ ] **Step 4: Commit**
```bash
git add spine.el
git commit -m "feat: add spine-render-index and empty-state view functions"
```
---
### Task 6: Wire up the index handler
**Files:**
- Modify: `spine.el`
- [ ] **Step 1: Replace `/hello` handler with `/` index handler**
Replace the `hello` handler (lines 4851) with:
```elisp
(defservlet index text/html ()
(let ((books (spine-books)))
(if books
(insert (spine-render-index books
(ht-get (ht-from-query-string (httpd-query-string)) "id")))
(insert (spine-render-empty-state)))))
```
- [ ] **Step 2: Verify batch load**
Run: `emacs --quick --batch --eval "(setq spine-skip-server-start t)" --load spine.el`
Expected: exits cleanly.
- [ ] **Step 3: Commit**
```bash
git add spine.el
git commit -m "feat: wire up / index handler replacing /hello"
```
---
### Task 7: Smoke test end-to-end
**Files:**
- No changes — verification only.
- [ ] **Step 1: Start server with sample data**
```bash
emacs --quick \
--eval "(setq spine-org-file \"$(pwd)/sample-books.org\")" \
--load spine.el &
EMACS_PID=$!
sleep 5
```
- [ ] **Step 2: Verify index page returns HTML with real data**
```bash
curl -s http://localhost:8080/ | head -20
```
Expected: HTML document with:
- `<title>Spine</title>`
- `<div class="group">WANT (3)</div>`
- `<div class="group">READING (1)</div>`
- `<div class="group">READ (1)</div>`
- No `ABANDONED` group (zero books)
- "Use of Weapons" visible in output
- Recommendation trail "Priya: If you liked..."
- [ ] **Step 3: Verify expanded detail via ?id=**
```bash
curl -s "http://localhost:8080/?id=8c1e-uow" | grep -A5 "detail"
```
Expected: expanded detail section with reading notes for Use of Weapons:
- `[2026-03-02] The two-track structure...`
- `[2026-03-07] Zakalwe's competence...`
- `[2026-03-09] The chapter numbering. Oh.`
- [ ] **Step 4: Verify bogus ?id= does not crash**
```bash
curl -s "http://localhost:8080/?id=nonexistent"
```
Expected: normal index page, no expanded detail, HTTP 200.
- [ ] **Step 5: Verify empty state when file is missing**
```bash
kill $EMACS_PID 2>/dev/null
sleep 1
emacs --quick \
--eval "(setq spine-org-file \"/tmp/nonexistent-books.org\")" \
--load spine.el &
EMACS_PID=$!
sleep 5
curl -s http://localhost:8080/
```
Expected: HTML containing `No books yet.`
- [ ] **Step 6: Stop server**
```bash
kill $EMACS_PID 2>/dev/null
```
- [ ] **Step 7: Verify with scripts/run**
```bash
# Override spine-org-file via the run script (pass it as an extra eval)
SPINE_ORG="$(pwd)/sample-books.org"
emacs --quick --daemon=spine \
--eval "(setq spine-org-file \"$SPINE_ORG\")" \
--load spine.el
sleep 3
curl -s http://localhost:8080/ | grep -c "WANT"
# Expected: 1 (the group header "WANT (3)")
emacsclient --socket-name=spine --eval '(kill-emacs)'
```
The output should contain `1`.
---
### Task 8: Final verification and cleanup
**Files:**
- No changes — verification only.
- [ ] **Step 1: Run full ERT test suite**
```bash
emacs --quick --batch \
--directory "$(pwd)" \
--load test/spine-books-test.el \
-f ert-run-tests-batch-and-exit
```
Expected: all 6 tests pass.
- [ ] **Step 2: Verify git status is clean**
```bash
git status
```
Expected: `nothing to commit, working tree clean`
- [ ] **Step 3: Review commit log**
```bash
git log --oneline
```
Expected: ~7 commits covering sample data, server guard, model, tests, view, handler.