Add orville-sqlite MVP design spec
This commit is contained in:
@@ -0,0 +1,347 @@
|
||||
# orville-sqlite MVP Design
|
||||
|
||||
**Date**: 2026-05-29
|
||||
**Status**: Draft
|
||||
**Scope**: Minimal Viable Product — type-safe SQLite API modeled after [Flipstone Orville](https://github.com/flipstone/orville/)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`orville-sqlite` is a Haskell library providing a type-safe SQLite API modeled after the Flipstone Orville PostgreSQL library. It maps Haskell data types to SQLite tables with compile-time schema guarantees.
|
||||
|
||||
The MVP covers three layers:
|
||||
1. **FieldDefinition & SqlType** — type-safe column mapping
|
||||
2. **SqlMarshaller & TableDefinition** — record-to-table mapping with primary keys
|
||||
3. **AutoMigration** — automatic schema creation and column migration
|
||||
4. **Execution** — basic CRUD operations
|
||||
|
||||
### What's NOT in MVP
|
||||
|
||||
- Foreign keys
|
||||
- Indexes (beyond implicit primary key indexes)
|
||||
- Composite primary keys
|
||||
- Table comments
|
||||
- Plans (N+1 query prevention)
|
||||
- Batch operations
|
||||
- Upsert
|
||||
- `findEntitiesBy` (filter on non-PK columns)
|
||||
- `updateFields` (partial update)
|
||||
- `marshallPartial`, `marshallQualifyFields`, `prefixMarshaller`
|
||||
- `AnnotatedSqlMarshaller` / `marshallResultFromSql`
|
||||
- `SyntheticField`
|
||||
- Nested record marshalling
|
||||
- Raw SQL execution (`executeAndDecode`)
|
||||
|
||||
---
|
||||
|
||||
## Module Structure
|
||||
|
||||
Single package `orville-sqlite`. Flat module hierarchy under `Orville.SQLite.*`.
|
||||
|
||||
```
|
||||
src/
|
||||
Orville/SQLite.hs
|
||||
Orville/SQLite/Monad.hs
|
||||
Orville/SQLite/RawSql.hs
|
||||
Orville/SQLite/SqlType.hs
|
||||
Orville/SQLite/FieldDefinition.hs
|
||||
Orville/SQLite/SqlMarshaller.hs
|
||||
Orville/SQLite/TableDefinition.hs
|
||||
Orville/SQLite/AutoMigration.hs
|
||||
Orville/SQLite/Execution.hs
|
||||
```
|
||||
|
||||
**Dependencies**: `base >=4.17 && <5`, `direct-sqlite`, `text`, `bytestring`
|
||||
|
||||
**Comparison to Orville PostgreSQL**: Drops ~60+ submodules. No Expression sub-hierarchy, no Execution sub-hierarchy, no extensions/sequences/triggers/indexes/windows/locking.
|
||||
|
||||
---
|
||||
|
||||
## Component Designs
|
||||
|
||||
### 1. SqlType (`Orville.SQLite.SqlType`)
|
||||
|
||||
SQLite has 5 storage classes: NULL, INTEGER, REAL, TEXT, BLOB. The `SqlType` encodes how Haskell values convert to/from SQLite values.
|
||||
|
||||
```haskell
|
||||
data SqlType a = SqlType
|
||||
{ sqlTypeName :: String -- "INTEGER", "TEXT", "BLOB", "REAL"
|
||||
, sqlTypeToSql :: a -> SqlValue -- Haskell → SQLite value
|
||||
, sqlTypeFromSql :: SqlValue -> Either String a -- SQLite value → Haskell
|
||||
}
|
||||
```
|
||||
|
||||
Pre-built instances:
|
||||
- `integerType :: SqlType Int64`
|
||||
- `textType :: SqlType Text`
|
||||
- `realType :: SqlType Double`
|
||||
- `blobType :: SqlType ByteString`
|
||||
|
||||
Helper: `convertSqlType :: (a -> b) -> (b -> a) -> SqlType a -> SqlType b` for wrapping custom types.
|
||||
|
||||
### 2. FieldDefinition (`Orville.SQLite.FieldDefinition`)
|
||||
|
||||
A GADT parameterized by nullability. Maps a Haskell type to a named SQL column.
|
||||
|
||||
```haskell
|
||||
data Nullability = NotNull | Nullable
|
||||
|
||||
data FieldDefinition (nullability :: Nullability) a where
|
||||
FieldDefinition ::
|
||||
{ fieldName :: String
|
||||
, fieldSqlType :: SqlType a
|
||||
} -> FieldDefinition 'NotNull a
|
||||
NullableFieldDefinition ::
|
||||
{ nullableFieldName :: String
|
||||
, nullableFieldSqlType :: SqlType a
|
||||
} -> FieldDefinition 'Nullable a
|
||||
```
|
||||
|
||||
Construction functions:
|
||||
- `integerField :: String -> FieldDefinition 'NotNull Int64`
|
||||
- `textField :: String -> FieldDefinition 'NotNull Text`
|
||||
- `realField :: String -> FieldDefinition 'NotNull Double`
|
||||
- `blobField :: String -> FieldDefinition 'NotNull ByteString`
|
||||
- `nullableField :: FieldDefinition 'NotNull a -> FieldDefinition 'Nullable a`
|
||||
- `convertField :: (a -> b) -> (b -> a) -> FieldDefinition null a -> FieldDefinition null b`
|
||||
|
||||
### 3. SqlMarshaller (`Orville.SQLite.SqlMarshaller`)
|
||||
|
||||
A GADT combining field definitions into record-level marshalling using applicative syntax. This is the core type-safe mapping between Haskell records and SQLite tables.
|
||||
|
||||
```haskell
|
||||
data SqlMarshaller writeEntity readEntity where
|
||||
MarshallPure :: readEntity -> SqlMarshaller writeEntity readEntity
|
||||
MarshallApply :: SqlMarshaller writeEntity (a -> b)
|
||||
-> SqlMarshaller writeEntity a
|
||||
-> SqlMarshaller writeEntity b
|
||||
MarshallNest :: (writeEntity -> a)
|
||||
-> SqlMarshaller a readEntity
|
||||
-> SqlMarshaller writeEntity readEntity
|
||||
MarshallField :: FieldDefinition nullability a -> SqlMarshaller a a
|
||||
```
|
||||
|
||||
Combinators:
|
||||
- `marshallField :: (writeEntity -> a) -> FieldDefinition 'NotNull a -> SqlMarshaller writeEntity a` — read+write column
|
||||
- `marshallReadOnlyField :: FieldDefinition nullability a -> SqlMarshaller writeEntity a` — for DB-populated columns (e.g., auto-increment PK). Excluded from INSERT/UPDATE, included in SELECT.
|
||||
- `marshallMaybe :: FieldDefinition 'Nullable a -> SqlMarshaller writeEntity (Maybe a)` — wraps nullable column as `Maybe`, handles SQL NULL.
|
||||
|
||||
Internal operations (not exposed as public API):
|
||||
- Collect derived columns for SELECT
|
||||
- Encode writeEntity to `[(String, SqlValue)]` for INSERT/UPDATE
|
||||
- Decode a result row into readEntity
|
||||
- Enumerate `FieldDefinition`s for AutoMigration schema comparison
|
||||
|
||||
Usage pattern:
|
||||
```haskell
|
||||
data Person = Person { firstName :: Text, lastName :: Text, age :: Int64 }
|
||||
|
||||
personMarshaller :: SqlMarshaller Person Person
|
||||
personMarshaller =
|
||||
Person
|
||||
<$> marshallField firstName firstNameField
|
||||
<*> marshallField lastName lastNameField
|
||||
<*> marshallField age ageField
|
||||
```
|
||||
|
||||
### 4. TableDefinition (`Orville.SQLite.TableDefinition`)
|
||||
|
||||
Ties a table name, primary key, and marshaller together.
|
||||
|
||||
```haskell
|
||||
data PrimaryKey writeEntity key where
|
||||
PrimaryKey ::
|
||||
(writeEntity -> key) -> -- accessor to extract key from writeEntity
|
||||
FieldDefinition 'NotNull key -> -- column definition
|
||||
PrimaryKey writeEntity key
|
||||
|
||||
data TableDefinition key writeEntity readEntity = TableDefinition
|
||||
{ tableName :: String
|
||||
, tablePrimaryKey :: PrimaryKey writeEntity key
|
||||
, tableMarshaller :: SqlMarshaller writeEntity readEntity
|
||||
}
|
||||
```
|
||||
|
||||
The `PrimaryKey` includes an accessor function so that `updateEntity` and `deleteEntity` can extract the key value from the `writeEntity` for the WHERE clause, even when the PK field is mapped as read-only in the marshaller.
|
||||
|
||||
Construction:
|
||||
- `primaryKey :: (writeEntity -> key) -> FieldDefinition 'NotNull key -> PrimaryKey writeEntity key`
|
||||
- `mkTableDefinition :: String -> PrimaryKey writeEntity key -> SqlMarshaller writeEntity readEntity -> TableDefinition key writeEntity readEntity`
|
||||
- `mkTableDefinitionWithoutKey :: String -> SqlMarshaller w r -> TableDefinition () w r`
|
||||
|
||||
### 5. AutoMigration (`Orville.SQLite.AutoMigration`)
|
||||
|
||||
Compares expected schema (from `TableDefinition`) against actual SQLite schema (via `PRAGMA table_info`) and generates DDL.
|
||||
|
||||
#### API
|
||||
|
||||
```haskell
|
||||
data MigrationOptions = MigrationOptions
|
||||
{ runSchemaChanges :: Bool -- True = apply changes, False = plan only
|
||||
}
|
||||
|
||||
defaultOptions :: MigrationOptions
|
||||
defaultOptions = MigrationOptions { runSchemaChanges = True }
|
||||
|
||||
data SchemaItem where
|
||||
SchemaTable :: TableDefinition key writeEntity readEntity -> SchemaItem
|
||||
|
||||
autoMigrateSchema :: MigrationOptions -> [SchemaItem] -> OrvilleM ()
|
||||
|
||||
generateMigrationPlan :: [SchemaItem] -> OrvilleM [MigrationStep]
|
||||
|
||||
executeMigrationPlan :: [MigrationStep] -> OrvilleM ()
|
||||
|
||||
data MigrationStep
|
||||
= CreateTable String [(String, String, Bool)] (String, String)
|
||||
-- table name, [(colName, colType, notNull)], (pkName, pkType)
|
||||
| AddColumn String String String Bool
|
||||
-- table name, colName, colType, notNull
|
||||
| DropColumn String String
|
||||
-- table name, colName
|
||||
```
|
||||
|
||||
Explicit column drop opt-in (mirroring Orville):
|
||||
```haskell
|
||||
dropColumns :: [String] -> TableDefinition key w r -> SchemaItem
|
||||
```
|
||||
|
||||
#### Schema Comparison Logic
|
||||
|
||||
1. Query `PRAGMA table_info(<tableName>)` which returns `(cid, name, type, notnull, dflt_value, pk)`
|
||||
2. **Table missing** → generate `CREATE TABLE IF NOT EXISTS` with all expected columns and primary key constraint
|
||||
3. **Column missing in DB but expected** → generate `ALTER TABLE ... ADD COLUMN`. The column must be nullable. If the ColumnDefinition specifies NOT NULL, error with guidance suggesting multi-step migration.
|
||||
4. **Column in DB but not expected** → ignored (kept). Columns are only dropped with explicit `dropColumns` opt-in.
|
||||
5. **Column type or nullability mismatch** → error with guidance explaining the mismatch and suggesting multi-step migration approach.
|
||||
|
||||
#### Error Handling for Incompatible Changes
|
||||
|
||||
When a column type or nullability changes in a way SQLite can't handle via ALTER TABLE:
|
||||
- Error message includes: table name, column name, expected vs actual type/nullability
|
||||
- Suggestion: "Consider a multi-step migration: add a new nullable column, backfill data, then use dropColumns to remove the old column"
|
||||
|
||||
#### SQLite DDL Constraints (Informing Design)
|
||||
|
||||
- `ALTER TABLE` can: rename table, add column (to end), rename column, drop column
|
||||
- `ALTER TABLE` cannot: change column type, change NOT NULL constraint, remove column without explicit drop
|
||||
- Added columns must be nullable or have a default value
|
||||
|
||||
### 6. Execution (`Orville.SQLite.Execution`)
|
||||
|
||||
Basic CRUD operations running in `OrvilleM`.
|
||||
|
||||
#### API
|
||||
|
||||
```haskell
|
||||
-- INSERT
|
||||
insertEntity :: TableDefinition key w r -> w -> OrvilleM ()
|
||||
|
||||
-- SELECT
|
||||
findEntity :: TableDefinition key w r -> key -> OrvilleM (Maybe r)
|
||||
findAll :: TableDefinition key w r -> OrvilleM [r]
|
||||
|
||||
-- UPDATE
|
||||
updateEntity :: TableDefinition key w r -> w -> OrvilleM ()
|
||||
|
||||
-- DELETE
|
||||
deleteEntity :: TableDefinition key w r -> key -> OrvilleM ()
|
||||
```
|
||||
|
||||
#### How Operations Work
|
||||
|
||||
**insertEntity**: Extracts `(colName, sqlValue)` pairs from writeEntity via marshaller (excluding read-only fields), builds `INSERT INTO table (cols) VALUES (vals)`, binds values as parameters, executes via `direct-sqlite`.
|
||||
|
||||
**findEntity**: Uses marshaller to generate `SELECT cols FROM table WHERE pk = ?`, binds the key value, decodes the single result row. Returns `Nothing` if no row found.
|
||||
|
||||
**findAll**: Generates `SELECT cols FROM table`, decodes all rows.
|
||||
|
||||
**updateEntity**: Generates `UPDATE table SET col=val, ... WHERE pk = val`. Only non-read-only fields from the marshaller are included in the SET clause. The PK value for the WHERE clause is extracted from writeEntity via `PrimaryKey`'s accessor function (which works even when the PK is mapped as read-only in the marshaller for auto-increment columns).
|
||||
|
||||
**deleteEntity**: `DELETE FROM table WHERE pk = ?` — the PK value is extracted from the provided key argument via the `PrimaryKey`'s `FieldDefinition` for encoding.
|
||||
|
||||
#### Error Handling
|
||||
|
||||
Uses `MarshallError` for decoding failures — identifies which column failed and why (type mismatch, out of range, etc.).
|
||||
|
||||
### 7. Monad (`Orville.SQLite.Monad`)
|
||||
|
||||
A simple reader monad wrapping a `direct-sqlite` `Database` handle.
|
||||
|
||||
```haskell
|
||||
newtype OrvilleM a = OrvilleM (ReaderT Database IO a)
|
||||
deriving (Functor, Applicative, Monad, MonadIO, MonadReader Database)
|
||||
|
||||
withConnection :: Database -> OrvilleM a -> IO a
|
||||
openConnection :: String -> IO Database
|
||||
closeConnection :: Database -> IO ()
|
||||
```
|
||||
|
||||
No connection pool. No `OrvilleState`. No transaction callbacks. For MVP, transactions are explicit:
|
||||
```haskell
|
||||
withTransaction :: OrvilleM a -> OrvilleM a
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Usage (Target API)
|
||||
|
||||
```haskell
|
||||
import Orville.SQLite
|
||||
|
||||
data Person = Person
|
||||
{ personId :: Int64
|
||||
, firstName :: Text
|
||||
, lastName :: Text
|
||||
, age :: Int64
|
||||
}
|
||||
|
||||
personIdField :: FieldDefinition 'NotNull Int64
|
||||
personIdField = integerField "id"
|
||||
|
||||
firstNameField :: FieldDefinition 'NotNull Text
|
||||
firstNameField = textField "first_name"
|
||||
|
||||
lastNameField :: FieldDefinition 'NotNull Text
|
||||
lastNameField = textField "last_name"
|
||||
|
||||
ageField :: FieldDefinition 'NotNull Int64
|
||||
ageField = integerField "age"
|
||||
|
||||
personMarshaller :: SqlMarshaller Person Person
|
||||
personMarshaller =
|
||||
Person
|
||||
<$> marshallReadOnlyField personIdField
|
||||
<*> marshallField firstName firstNameField
|
||||
<*> marshallField lastName lastNameField
|
||||
<*> marshallField age ageField
|
||||
|
||||
personTable :: TableDefinition Int64 Person Person
|
||||
personTable =
|
||||
mkTableDefinition "person" (primaryKey personId personIdField) personMarshaller
|
||||
|
||||
main :: IO ()
|
||||
main = do
|
||||
db <- openConnection "people.db"
|
||||
withConnection db $ do
|
||||
autoMigrateSchema defaultOptions [SchemaTable personTable]
|
||||
insertEntity personTable (Person 0 "Alice" "Smith" 30) -- 0 is ignored (read-only PK)
|
||||
mAlice <- findEntity personTable 1
|
||||
-- mAlice = Just (Person 1 "Alice" "Smith" 30)
|
||||
updateEntity personTable (Person 1 "Alice" "Smith" 31) -- PK extracted for WHERE
|
||||
pure ()
|
||||
closeConnection db
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Monad + RawSql** — Foundation: `OrvilleM`, `withConnection`, `openConnection`, minimal raw SQL builder
|
||||
2. **SqlType** — Type encoding/decoding for SQLite storage classes
|
||||
3. **FieldDefinition** — GADT with `NotNull`/`Nullable`, construction functions
|
||||
4. **SqlMarshaller** — GADT with applicative combinators, encode/decode/column listing internals
|
||||
5. **TableDefinition** — `mkTableDefinition`, `PrimaryKey`
|
||||
6. **AutoMigration** — `PRAGMA table_info` introspection, schema comparison, DDL generation
|
||||
7. **Execution** — `insertEntity`, `findEntity`, `findAll`, `updateEntity`, `deleteEntity`
|
||||
8. **Top-level re-exports** — `Orville.SQLite` module
|
||||
Reference in New Issue
Block a user