Add orville-sqlite MVP design spec

2026-05-29 22:51:44 -04:00
commit bbbe1beb09
4 changed files with 355 additions and 0 deletions
@@ -0,0 +1,6 @@
+[submodule "orville"]
+	path = orville
+	url = https://github.com/flipstone/orville.git
+[submodule "flipstone-orville"]
+	path = flipstone-orville
+	url = https://github.com/flipstone/orville.git
@@ -0,0 +1,347 @@
+# orville-sqlite MVP Design
+
+**Date**: 2026-05-29  
+**Status**: Draft  
+**Scope**: Minimal Viable Product — type-safe SQLite API modeled after [Flipstone Orville](https://github.com/flipstone/orville/)
+
+---
+
+## Overview
+
+`orville-sqlite` is a Haskell library providing a type-safe SQLite API modeled after the Flipstone Orville PostgreSQL library. It maps Haskell data types to SQLite tables with compile-time schema guarantees.
+
+The MVP covers three layers:
+1. **FieldDefinition & SqlType** — type-safe column mapping
+2. **SqlMarshaller & TableDefinition** — record-to-table mapping with primary keys
+3. **AutoMigration** — automatic schema creation and column migration
+4. **Execution** — basic CRUD operations
+
+### What's NOT in MVP
+
+- Foreign keys
+- Indexes (beyond implicit primary key indexes)
+- Composite primary keys
+- Table comments
+- Plans (N+1 query prevention)
+- Batch operations
+- Upsert
+- `findEntitiesBy` (filter on non-PK columns)
+- `updateFields` (partial update)
+- `marshallPartial`, `marshallQualifyFields`, `prefixMarshaller`
+- `AnnotatedSqlMarshaller` / `marshallResultFromSql`
+- `SyntheticField`
+- Nested record marshalling
+- Raw SQL execution (`executeAndDecode`)
+
+---
+
+## Module Structure
+
+Single package `orville-sqlite`. Flat module hierarchy under `Orville.SQLite.*`.
+
+```
+src/
+  Orville/SQLite.hs
+  Orville/SQLite/Monad.hs
+  Orville/SQLite/RawSql.hs
+  Orville/SQLite/SqlType.hs
+  Orville/SQLite/FieldDefinition.hs
+  Orville/SQLite/SqlMarshaller.hs
+  Orville/SQLite/TableDefinition.hs
+  Orville/SQLite/AutoMigration.hs
+  Orville/SQLite/Execution.hs
+```
+
+**Dependencies**: `base >=4.17 && <5`, `direct-sqlite`, `text`, `bytestring`
+
+**Comparison to Orville PostgreSQL**: Drops ~60+ submodules. No Expression sub-hierarchy, no Execution sub-hierarchy, no extensions/sequences/triggers/indexes/windows/locking.
+
+---
+
+## Component Designs
+
+### 1. SqlType (`Orville.SQLite.SqlType`)
+
+SQLite has 5 storage classes: NULL, INTEGER, REAL, TEXT, BLOB. The `SqlType` encodes how Haskell values convert to/from SQLite values.
+
+```haskell
+data SqlType a = SqlType
+  { sqlTypeName   :: String           -- "INTEGER", "TEXT", "BLOB", "REAL"
+  , sqlTypeToSql  :: a -> SqlValue    -- Haskell → SQLite value
+  , sqlTypeFromSql :: SqlValue -> Either String a  -- SQLite value → Haskell
+  }
+```
+
+Pre-built instances:
+- `integerType :: SqlType Int64`
+- `textType :: SqlType Text`
+- `realType :: SqlType Double`
+- `blobType :: SqlType ByteString`
+
+Helper: `convertSqlType :: (a -> b) -> (b -> a) -> SqlType a -> SqlType b` for wrapping custom types.
+
+### 2. FieldDefinition (`Orville.SQLite.FieldDefinition`)
+
+A GADT parameterized by nullability. Maps a Haskell type to a named SQL column.
+
+```haskell
+data Nullability = NotNull | Nullable
+
+data FieldDefinition (nullability :: Nullability) a where
+  FieldDefinition ::
+    { fieldName :: String
+    , fieldSqlType :: SqlType a
+    } -> FieldDefinition 'NotNull a
+  NullableFieldDefinition ::
+    { nullableFieldName :: String
+    , nullableFieldSqlType :: SqlType a
+    } -> FieldDefinition 'Nullable a
+```
+
+Construction functions:
+- `integerField :: String -> FieldDefinition 'NotNull Int64`
+- `textField :: String -> FieldDefinition 'NotNull Text`
+- `realField :: String -> FieldDefinition 'NotNull Double`
+- `blobField :: String -> FieldDefinition 'NotNull ByteString`
+- `nullableField :: FieldDefinition 'NotNull a -> FieldDefinition 'Nullable a`
+- `convertField :: (a -> b) -> (b -> a) -> FieldDefinition null a -> FieldDefinition null b`
+
+### 3. SqlMarshaller (`Orville.SQLite.SqlMarshaller`)
+
+A GADT combining field definitions into record-level marshalling using applicative syntax. This is the core type-safe mapping between Haskell records and SQLite tables.
+
+```haskell
+data SqlMarshaller writeEntity readEntity where
+  MarshallPure   :: readEntity -> SqlMarshaller writeEntity readEntity
+  MarshallApply  :: SqlMarshaller writeEntity (a -> b)
+                 -> SqlMarshaller writeEntity a
+                 -> SqlMarshaller writeEntity b
+  MarshallNest   :: (writeEntity -> a)
+                 -> SqlMarshaller a readEntity
+                 -> SqlMarshaller writeEntity readEntity
+  MarshallField  :: FieldDefinition nullability a -> SqlMarshaller a a
+```
+
+Combinators:
+- `marshallField :: (writeEntity -> a) -> FieldDefinition 'NotNull a -> SqlMarshaller writeEntity a` — read+write column
+- `marshallReadOnlyField :: FieldDefinition nullability a -> SqlMarshaller writeEntity a` — for DB-populated columns (e.g., auto-increment PK). Excluded from INSERT/UPDATE, included in SELECT.
+- `marshallMaybe :: FieldDefinition 'Nullable a -> SqlMarshaller writeEntity (Maybe a)` — wraps nullable column as `Maybe`, handles SQL NULL.
+
+Internal operations (not exposed as public API):
+- Collect derived columns for SELECT
+- Encode writeEntity to `[(String, SqlValue)]` for INSERT/UPDATE
+- Decode a result row into readEntity
+- Enumerate `FieldDefinition`s for AutoMigration schema comparison
+
+Usage pattern:
+```haskell
+data Person = Person { firstName :: Text, lastName :: Text, age :: Int64 }
+
+personMarshaller :: SqlMarshaller Person Person
+personMarshaller =
+  Person
+    <$> marshallField firstName firstNameField
+    <*> marshallField lastName  lastNameField
+    <*> marshallField age       ageField
+```
+
+### 4. TableDefinition (`Orville.SQLite.TableDefinition`)
+
+Ties a table name, primary key, and marshaller together.
+
+```haskell
+data PrimaryKey writeEntity key where
+  PrimaryKey ::
+    (writeEntity -> key) ->               -- accessor to extract key from writeEntity
+    FieldDefinition 'NotNull key ->        -- column definition
+    PrimaryKey writeEntity key
+
+data TableDefinition key writeEntity readEntity = TableDefinition
+  { tableName :: String
+  , tablePrimaryKey :: PrimaryKey writeEntity key
+  , tableMarshaller :: SqlMarshaller writeEntity readEntity
+  }
+```
+
+The `PrimaryKey` includes an accessor function so that `updateEntity` and `deleteEntity` can extract the key value from the `writeEntity` for the WHERE clause, even when the PK field is mapped as read-only in the marshaller.
+
+Construction:
+- `primaryKey :: (writeEntity -> key) -> FieldDefinition 'NotNull key -> PrimaryKey writeEntity key`
+- `mkTableDefinition :: String -> PrimaryKey writeEntity key -> SqlMarshaller writeEntity readEntity -> TableDefinition key writeEntity readEntity`
+- `mkTableDefinitionWithoutKey :: String -> SqlMarshaller w r -> TableDefinition () w r`
+
+### 5. AutoMigration (`Orville.SQLite.AutoMigration`)
+
+Compares expected schema (from `TableDefinition`) against actual SQLite schema (via `PRAGMA table_info`) and generates DDL.
+
+#### API
+
+```haskell
+data MigrationOptions = MigrationOptions
+  { runSchemaChanges :: Bool    -- True = apply changes, False = plan only
+  }
+
+defaultOptions :: MigrationOptions
+defaultOptions = MigrationOptions { runSchemaChanges = True }
+
+data SchemaItem where
+  SchemaTable :: TableDefinition key writeEntity readEntity -> SchemaItem
+
+autoMigrateSchema :: MigrationOptions -> [SchemaItem] -> OrvilleM ()
+
+generateMigrationPlan :: [SchemaItem] -> OrvilleM [MigrationStep]
+
+executeMigrationPlan :: [MigrationStep] -> OrvilleM ()
+
+data MigrationStep
+  = CreateTable String [(String, String, Bool)] (String, String)
+    -- table name, [(colName, colType, notNull)], (pkName, pkType)
+  | AddColumn String String String Bool
+    -- table name, colName, colType, notNull
+  | DropColumn String String
+    -- table name, colName
+```
+
+Explicit column drop opt-in (mirroring Orville):
+```haskell
+dropColumns :: [String] -> TableDefinition key w r -> SchemaItem
+```
+
+#### Schema Comparison Logic
+
+1. Query `PRAGMA table_info(<tableName>)` which returns `(cid, name, type, notnull, dflt_value, pk)`
+2. **Table missing** → generate `CREATE TABLE IF NOT EXISTS` with all expected columns and primary key constraint
+3. **Column missing in DB but expected** → generate `ALTER TABLE ... ADD COLUMN`. The column must be nullable. If the ColumnDefinition specifies NOT NULL, error with guidance suggesting multi-step migration.
+4. **Column in DB but not expected** → ignored (kept). Columns are only dropped with explicit `dropColumns` opt-in.
+5. **Column type or nullability mismatch** → error with guidance explaining the mismatch and suggesting multi-step migration approach.
+
+#### Error Handling for Incompatible Changes
+
+When a column type or nullability changes in a way SQLite can't handle via ALTER TABLE:
+- Error message includes: table name, column name, expected vs actual type/nullability
+- Suggestion: "Consider a multi-step migration: add a new nullable column, backfill data, then use dropColumns to remove the old column"
+
+#### SQLite DDL Constraints (Informing Design)
+
+- `ALTER TABLE` can: rename table, add column (to end), rename column, drop column
+- `ALTER TABLE` cannot: change column type, change NOT NULL constraint, remove column without explicit drop
+- Added columns must be nullable or have a default value
+
+### 6. Execution (`Orville.SQLite.Execution`)
+
+Basic CRUD operations running in `OrvilleM`.
+
+#### API
+
+```haskell
+-- INSERT
+insertEntity :: TableDefinition key w r -> w -> OrvilleM ()
+
+-- SELECT
+findEntity :: TableDefinition key w r -> key -> OrvilleM (Maybe r)
+findAll :: TableDefinition key w r -> OrvilleM [r]
+
+-- UPDATE
+updateEntity :: TableDefinition key w r -> w -> OrvilleM ()
+
+-- DELETE
+deleteEntity :: TableDefinition key w r -> key -> OrvilleM ()
+```
+
+#### How Operations Work
+
+**insertEntity**: Extracts `(colName, sqlValue)` pairs from writeEntity via marshaller (excluding read-only fields), builds `INSERT INTO table (cols) VALUES (vals)`, binds values as parameters, executes via `direct-sqlite`.
+
+**findEntity**: Uses marshaller to generate `SELECT cols FROM table WHERE pk = ?`, binds the key value, decodes the single result row. Returns `Nothing` if no row found.
+
+**findAll**: Generates `SELECT cols FROM table`, decodes all rows.
+
+**updateEntity**: Generates `UPDATE table SET col=val, ... WHERE pk = val`. Only non-read-only fields from the marshaller are included in the SET clause. The PK value for the WHERE clause is extracted from writeEntity via `PrimaryKey`'s accessor function (which works even when the PK is mapped as read-only in the marshaller for auto-increment columns).
+
+**deleteEntity**: `DELETE FROM table WHERE pk = ?` — the PK value is extracted from the provided key argument via the `PrimaryKey`'s `FieldDefinition` for encoding.
+
+#### Error Handling
+
+Uses `MarshallError` for decoding failures — identifies which column failed and why (type mismatch, out of range, etc.).
+
+### 7. Monad (`Orville.SQLite.Monad`)
+
+A simple reader monad wrapping a `direct-sqlite` `Database` handle.
+
+```haskell
+newtype OrvilleM a = OrvilleM (ReaderT Database IO a)
+  deriving (Functor, Applicative, Monad, MonadIO, MonadReader Database)
+
+withConnection :: Database -> OrvilleM a -> IO a
+openConnection :: String -> IO Database
+closeConnection :: Database -> IO ()
+```
+
+No connection pool. No `OrvilleState`. No transaction callbacks. For MVP, transactions are explicit:
+```haskell
+withTransaction :: OrvilleM a -> OrvilleM a
+```
+
+---
+
+## Example Usage (Target API)
+
+```haskell
+import Orville.SQLite
+
+data Person = Person
+  { personId :: Int64
+  , firstName :: Text
+  , lastName :: Text
+  , age :: Int64
+  }
+
+personIdField :: FieldDefinition 'NotNull Int64
+personIdField = integerField "id"
+
+firstNameField :: FieldDefinition 'NotNull Text
+firstNameField = textField "first_name"
+
+lastNameField :: FieldDefinition 'NotNull Text
+lastNameField = textField "last_name"
+
+ageField :: FieldDefinition 'NotNull Int64
+ageField = integerField "age"
+
+personMarshaller :: SqlMarshaller Person Person
+personMarshaller =
+  Person
+    <$> marshallReadOnlyField personIdField
+    <*> marshallField firstName firstNameField
+    <*> marshallField lastName  lastNameField
+    <*> marshallField age       ageField
+
+personTable :: TableDefinition Int64 Person Person
+personTable =
+  mkTableDefinition "person" (primaryKey personId personIdField) personMarshaller
+
+main :: IO ()
+main = do
+  db <- openConnection "people.db"
+  withConnection db $ do
+    autoMigrateSchema defaultOptions [SchemaTable personTable]
+    insertEntity personTable (Person 0 "Alice" "Smith" 30)   -- 0 is ignored (read-only PK)
+    mAlice <- findEntity personTable 1
+    -- mAlice = Just (Person 1 "Alice" "Smith" 30)
+    updateEntity personTable (Person 1 "Alice" "Smith" 31)  -- PK extracted for WHERE
+    pure ()
+  closeConnection db
+```
+
+---
+
+## Implementation Order
+
+1. **Monad + RawSql** — Foundation: `OrvilleM`, `withConnection`, `openConnection`, minimal raw SQL builder
+2. **SqlType** — Type encoding/decoding for SQLite storage classes
+3. **FieldDefinition** — GADT with `NotNull`/`Nullable`, construction functions
+4. **SqlMarshaller** — GADT with applicative combinators, encode/decode/column listing internals
+5. **TableDefinition** — `mkTableDefinition`, `PrimaryKey`
+6. **AutoMigration** — `PRAGMA table_info` introspection, schema comparison, DDL generation
+7. **Execution** — `insertEntity`, `findEntity`, `findAll`, `updateEntity`, `deleteEntity`
+8. **Top-level re-exports** — `Orville.SQLite` module