From bbbe1beb093bac53722e0a1074b555cba9afb3cf Mon Sep 17 00:00:00 2001 From: James Brechtel Date: Fri, 29 May 2026 22:51:44 -0400 Subject: [PATCH] Add orville-sqlite MVP design spec --- .gitmodules | 6 + .../2026-05-29-orville-sqlite-mvp-design.md | 347 ++++++++++++++++++ flipstone-orville | 1 + orville | 1 + 4 files changed, 355 insertions(+) create mode 100644 .gitmodules create mode 100644 docs/specs/2026-05-29-orville-sqlite-mvp-design.md create mode 160000 flipstone-orville create mode 160000 orville diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..647b61c --- /dev/null +++ b/.gitmodules @@ -0,0 +1,6 @@ +[submodule "orville"] + path = orville + url = https://github.com/flipstone/orville.git +[submodule "flipstone-orville"] + path = flipstone-orville + url = https://github.com/flipstone/orville.git diff --git a/docs/specs/2026-05-29-orville-sqlite-mvp-design.md b/docs/specs/2026-05-29-orville-sqlite-mvp-design.md new file mode 100644 index 0000000..d331191 --- /dev/null +++ b/docs/specs/2026-05-29-orville-sqlite-mvp-design.md @@ -0,0 +1,347 @@ +# orville-sqlite MVP Design + +**Date**: 2026-05-29 +**Status**: Draft +**Scope**: Minimal Viable Product — type-safe SQLite API modeled after [Flipstone Orville](https://github.com/flipstone/orville/) + +--- + +## Overview + +`orville-sqlite` is a Haskell library providing a type-safe SQLite API modeled after the Flipstone Orville PostgreSQL library. It maps Haskell data types to SQLite tables with compile-time schema guarantees. + +The MVP covers three layers: +1. **FieldDefinition & SqlType** — type-safe column mapping +2. **SqlMarshaller & TableDefinition** — record-to-table mapping with primary keys +3. **AutoMigration** — automatic schema creation and column migration +4. **Execution** — basic CRUD operations + +### What's NOT in MVP + +- Foreign keys +- Indexes (beyond implicit primary key indexes) +- Composite primary keys +- Table comments +- Plans (N+1 query prevention) +- Batch operations +- Upsert +- `findEntitiesBy` (filter on non-PK columns) +- `updateFields` (partial update) +- `marshallPartial`, `marshallQualifyFields`, `prefixMarshaller` +- `AnnotatedSqlMarshaller` / `marshallResultFromSql` +- `SyntheticField` +- Nested record marshalling +- Raw SQL execution (`executeAndDecode`) + +--- + +## Module Structure + +Single package `orville-sqlite`. Flat module hierarchy under `Orville.SQLite.*`. + +``` +src/ + Orville/SQLite.hs + Orville/SQLite/Monad.hs + Orville/SQLite/RawSql.hs + Orville/SQLite/SqlType.hs + Orville/SQLite/FieldDefinition.hs + Orville/SQLite/SqlMarshaller.hs + Orville/SQLite/TableDefinition.hs + Orville/SQLite/AutoMigration.hs + Orville/SQLite/Execution.hs +``` + +**Dependencies**: `base >=4.17 && <5`, `direct-sqlite`, `text`, `bytestring` + +**Comparison to Orville PostgreSQL**: Drops ~60+ submodules. No Expression sub-hierarchy, no Execution sub-hierarchy, no extensions/sequences/triggers/indexes/windows/locking. + +--- + +## Component Designs + +### 1. SqlType (`Orville.SQLite.SqlType`) + +SQLite has 5 storage classes: NULL, INTEGER, REAL, TEXT, BLOB. The `SqlType` encodes how Haskell values convert to/from SQLite values. + +```haskell +data SqlType a = SqlType + { sqlTypeName :: String -- "INTEGER", "TEXT", "BLOB", "REAL" + , sqlTypeToSql :: a -> SqlValue -- Haskell → SQLite value + , sqlTypeFromSql :: SqlValue -> Either String a -- SQLite value → Haskell + } +``` + +Pre-built instances: +- `integerType :: SqlType Int64` +- `textType :: SqlType Text` +- `realType :: SqlType Double` +- `blobType :: SqlType ByteString` + +Helper: `convertSqlType :: (a -> b) -> (b -> a) -> SqlType a -> SqlType b` for wrapping custom types. + +### 2. FieldDefinition (`Orville.SQLite.FieldDefinition`) + +A GADT parameterized by nullability. Maps a Haskell type to a named SQL column. + +```haskell +data Nullability = NotNull | Nullable + +data FieldDefinition (nullability :: Nullability) a where + FieldDefinition :: + { fieldName :: String + , fieldSqlType :: SqlType a + } -> FieldDefinition 'NotNull a + NullableFieldDefinition :: + { nullableFieldName :: String + , nullableFieldSqlType :: SqlType a + } -> FieldDefinition 'Nullable a +``` + +Construction functions: +- `integerField :: String -> FieldDefinition 'NotNull Int64` +- `textField :: String -> FieldDefinition 'NotNull Text` +- `realField :: String -> FieldDefinition 'NotNull Double` +- `blobField :: String -> FieldDefinition 'NotNull ByteString` +- `nullableField :: FieldDefinition 'NotNull a -> FieldDefinition 'Nullable a` +- `convertField :: (a -> b) -> (b -> a) -> FieldDefinition null a -> FieldDefinition null b` + +### 3. SqlMarshaller (`Orville.SQLite.SqlMarshaller`) + +A GADT combining field definitions into record-level marshalling using applicative syntax. This is the core type-safe mapping between Haskell records and SQLite tables. + +```haskell +data SqlMarshaller writeEntity readEntity where + MarshallPure :: readEntity -> SqlMarshaller writeEntity readEntity + MarshallApply :: SqlMarshaller writeEntity (a -> b) + -> SqlMarshaller writeEntity a + -> SqlMarshaller writeEntity b + MarshallNest :: (writeEntity -> a) + -> SqlMarshaller a readEntity + -> SqlMarshaller writeEntity readEntity + MarshallField :: FieldDefinition nullability a -> SqlMarshaller a a +``` + +Combinators: +- `marshallField :: (writeEntity -> a) -> FieldDefinition 'NotNull a -> SqlMarshaller writeEntity a` — read+write column +- `marshallReadOnlyField :: FieldDefinition nullability a -> SqlMarshaller writeEntity a` — for DB-populated columns (e.g., auto-increment PK). Excluded from INSERT/UPDATE, included in SELECT. +- `marshallMaybe :: FieldDefinition 'Nullable a -> SqlMarshaller writeEntity (Maybe a)` — wraps nullable column as `Maybe`, handles SQL NULL. + +Internal operations (not exposed as public API): +- Collect derived columns for SELECT +- Encode writeEntity to `[(String, SqlValue)]` for INSERT/UPDATE +- Decode a result row into readEntity +- Enumerate `FieldDefinition`s for AutoMigration schema comparison + +Usage pattern: +```haskell +data Person = Person { firstName :: Text, lastName :: Text, age :: Int64 } + +personMarshaller :: SqlMarshaller Person Person +personMarshaller = + Person + <$> marshallField firstName firstNameField + <*> marshallField lastName lastNameField + <*> marshallField age ageField +``` + +### 4. TableDefinition (`Orville.SQLite.TableDefinition`) + +Ties a table name, primary key, and marshaller together. + +```haskell +data PrimaryKey writeEntity key where + PrimaryKey :: + (writeEntity -> key) -> -- accessor to extract key from writeEntity + FieldDefinition 'NotNull key -> -- column definition + PrimaryKey writeEntity key + +data TableDefinition key writeEntity readEntity = TableDefinition + { tableName :: String + , tablePrimaryKey :: PrimaryKey writeEntity key + , tableMarshaller :: SqlMarshaller writeEntity readEntity + } +``` + +The `PrimaryKey` includes an accessor function so that `updateEntity` and `deleteEntity` can extract the key value from the `writeEntity` for the WHERE clause, even when the PK field is mapped as read-only in the marshaller. + +Construction: +- `primaryKey :: (writeEntity -> key) -> FieldDefinition 'NotNull key -> PrimaryKey writeEntity key` +- `mkTableDefinition :: String -> PrimaryKey writeEntity key -> SqlMarshaller writeEntity readEntity -> TableDefinition key writeEntity readEntity` +- `mkTableDefinitionWithoutKey :: String -> SqlMarshaller w r -> TableDefinition () w r` + +### 5. AutoMigration (`Orville.SQLite.AutoMigration`) + +Compares expected schema (from `TableDefinition`) against actual SQLite schema (via `PRAGMA table_info`) and generates DDL. + +#### API + +```haskell +data MigrationOptions = MigrationOptions + { runSchemaChanges :: Bool -- True = apply changes, False = plan only + } + +defaultOptions :: MigrationOptions +defaultOptions = MigrationOptions { runSchemaChanges = True } + +data SchemaItem where + SchemaTable :: TableDefinition key writeEntity readEntity -> SchemaItem + +autoMigrateSchema :: MigrationOptions -> [SchemaItem] -> OrvilleM () + +generateMigrationPlan :: [SchemaItem] -> OrvilleM [MigrationStep] + +executeMigrationPlan :: [MigrationStep] -> OrvilleM () + +data MigrationStep + = CreateTable String [(String, String, Bool)] (String, String) + -- table name, [(colName, colType, notNull)], (pkName, pkType) + | AddColumn String String String Bool + -- table name, colName, colType, notNull + | DropColumn String String + -- table name, colName +``` + +Explicit column drop opt-in (mirroring Orville): +```haskell +dropColumns :: [String] -> TableDefinition key w r -> SchemaItem +``` + +#### Schema Comparison Logic + +1. Query `PRAGMA table_info()` which returns `(cid, name, type, notnull, dflt_value, pk)` +2. **Table missing** → generate `CREATE TABLE IF NOT EXISTS` with all expected columns and primary key constraint +3. **Column missing in DB but expected** → generate `ALTER TABLE ... ADD COLUMN`. The column must be nullable. If the ColumnDefinition specifies NOT NULL, error with guidance suggesting multi-step migration. +4. **Column in DB but not expected** → ignored (kept). Columns are only dropped with explicit `dropColumns` opt-in. +5. **Column type or nullability mismatch** → error with guidance explaining the mismatch and suggesting multi-step migration approach. + +#### Error Handling for Incompatible Changes + +When a column type or nullability changes in a way SQLite can't handle via ALTER TABLE: +- Error message includes: table name, column name, expected vs actual type/nullability +- Suggestion: "Consider a multi-step migration: add a new nullable column, backfill data, then use dropColumns to remove the old column" + +#### SQLite DDL Constraints (Informing Design) + +- `ALTER TABLE` can: rename table, add column (to end), rename column, drop column +- `ALTER TABLE` cannot: change column type, change NOT NULL constraint, remove column without explicit drop +- Added columns must be nullable or have a default value + +### 6. Execution (`Orville.SQLite.Execution`) + +Basic CRUD operations running in `OrvilleM`. + +#### API + +```haskell +-- INSERT +insertEntity :: TableDefinition key w r -> w -> OrvilleM () + +-- SELECT +findEntity :: TableDefinition key w r -> key -> OrvilleM (Maybe r) +findAll :: TableDefinition key w r -> OrvilleM [r] + +-- UPDATE +updateEntity :: TableDefinition key w r -> w -> OrvilleM () + +-- DELETE +deleteEntity :: TableDefinition key w r -> key -> OrvilleM () +``` + +#### How Operations Work + +**insertEntity**: Extracts `(colName, sqlValue)` pairs from writeEntity via marshaller (excluding read-only fields), builds `INSERT INTO table (cols) VALUES (vals)`, binds values as parameters, executes via `direct-sqlite`. + +**findEntity**: Uses marshaller to generate `SELECT cols FROM table WHERE pk = ?`, binds the key value, decodes the single result row. Returns `Nothing` if no row found. + +**findAll**: Generates `SELECT cols FROM table`, decodes all rows. + +**updateEntity**: Generates `UPDATE table SET col=val, ... WHERE pk = val`. Only non-read-only fields from the marshaller are included in the SET clause. The PK value for the WHERE clause is extracted from writeEntity via `PrimaryKey`'s accessor function (which works even when the PK is mapped as read-only in the marshaller for auto-increment columns). + +**deleteEntity**: `DELETE FROM table WHERE pk = ?` — the PK value is extracted from the provided key argument via the `PrimaryKey`'s `FieldDefinition` for encoding. + +#### Error Handling + +Uses `MarshallError` for decoding failures — identifies which column failed and why (type mismatch, out of range, etc.). + +### 7. Monad (`Orville.SQLite.Monad`) + +A simple reader monad wrapping a `direct-sqlite` `Database` handle. + +```haskell +newtype OrvilleM a = OrvilleM (ReaderT Database IO a) + deriving (Functor, Applicative, Monad, MonadIO, MonadReader Database) + +withConnection :: Database -> OrvilleM a -> IO a +openConnection :: String -> IO Database +closeConnection :: Database -> IO () +``` + +No connection pool. No `OrvilleState`. No transaction callbacks. For MVP, transactions are explicit: +```haskell +withTransaction :: OrvilleM a -> OrvilleM a +``` + +--- + +## Example Usage (Target API) + +```haskell +import Orville.SQLite + +data Person = Person + { personId :: Int64 + , firstName :: Text + , lastName :: Text + , age :: Int64 + } + +personIdField :: FieldDefinition 'NotNull Int64 +personIdField = integerField "id" + +firstNameField :: FieldDefinition 'NotNull Text +firstNameField = textField "first_name" + +lastNameField :: FieldDefinition 'NotNull Text +lastNameField = textField "last_name" + +ageField :: FieldDefinition 'NotNull Int64 +ageField = integerField "age" + +personMarshaller :: SqlMarshaller Person Person +personMarshaller = + Person + <$> marshallReadOnlyField personIdField + <*> marshallField firstName firstNameField + <*> marshallField lastName lastNameField + <*> marshallField age ageField + +personTable :: TableDefinition Int64 Person Person +personTable = + mkTableDefinition "person" (primaryKey personId personIdField) personMarshaller + +main :: IO () +main = do + db <- openConnection "people.db" + withConnection db $ do + autoMigrateSchema defaultOptions [SchemaTable personTable] + insertEntity personTable (Person 0 "Alice" "Smith" 30) -- 0 is ignored (read-only PK) + mAlice <- findEntity personTable 1 + -- mAlice = Just (Person 1 "Alice" "Smith" 30) + updateEntity personTable (Person 1 "Alice" "Smith" 31) -- PK extracted for WHERE + pure () + closeConnection db +``` + +--- + +## Implementation Order + +1. **Monad + RawSql** — Foundation: `OrvilleM`, `withConnection`, `openConnection`, minimal raw SQL builder +2. **SqlType** — Type encoding/decoding for SQLite storage classes +3. **FieldDefinition** — GADT with `NotNull`/`Nullable`, construction functions +4. **SqlMarshaller** — GADT with applicative combinators, encode/decode/column listing internals +5. **TableDefinition** — `mkTableDefinition`, `PrimaryKey` +6. **AutoMigration** — `PRAGMA table_info` introspection, schema comparison, DDL generation +7. **Execution** — `insertEntity`, `findEntity`, `findAll`, `updateEntity`, `deleteEntity` +8. **Top-level re-exports** — `Orville.SQLite` module diff --git a/flipstone-orville b/flipstone-orville new file mode 160000 index 0000000..818546f --- /dev/null +++ b/flipstone-orville @@ -0,0 +1 @@ +Subproject commit 818546f2dabe112b14647827ec2b9e5d3d9dd7d0 diff --git a/orville b/orville new file mode 160000 index 0000000..818546f --- /dev/null +++ b/orville @@ -0,0 +1 @@ +Subproject commit 818546f2dabe112b14647827ec2b9e5d3d9dd7d0