Compare commits

...

4 Commits

Author SHA1 Message Date
jbrechtel bdd9e663b9 chore: apply fourmolu formatting
Build / build (push) Failing after 7s
2026-05-14 14:50:23 -04:00
jbrechtel 662854a977 tests: SSE parsing + relay event handling, update AGENTS.md
- New test module test/RelaySpec.hs (20 tests):
  - extractLines: line splitting, CRLF, partial lines, empty input
  - foldLines: single/two push events, non-push ignored, comments, no-space colon,
    partial state carry-over across batches
  - parseEvent: valid push, non-push, malformed JSON, multi-line data
  - handleSSEEvent: pull on matching repo, ignore wrong gitea_repo/branch,
    skip pull when merge/rebase/operation in progress
- Exported PushEvent(..), extractLines, foldLines, parseEvent, handleSSEEvent
- Fixed data line order (append instead of prepend+reverse) and event ordering
- Updated AGENTS.md: relay architecture, Go component, new deps, file layout
2026-05-14 14:40:47 -04:00
jbrechtel 7feff5abfa converge: phase 2 — SSE relay client for realtime pull on push
- RepoConfig gains rcGiteaRepo :: Maybe Text for relay matching
- Options gains optRelayUrl :: Maybe Text (--relay-url CLI, relay_url config)
- connectRelay connects to converge-relay SSE endpoint, parses push events,
  triggers gitPull on matching repos (by gitea_repo + branch)
- Exponential backoff reconnection (1s .. 60s) on disconnect/error
- Skips pull if repo is mid-operation (merge/rebase/etc.)
- Dependencies: http-client, http-client-tls, http-types, aeson, bytestring
- SSE parser handles event:/data: fields, blank-line delimiters, CRLF endings
- Updated SPEC.md, example config, and help text
2026-05-14 14:13:43 -04:00
jbrechtel 25881f3ff8 converge-relay: phase 1 — webhook-to-SSE relay server in Go
- Pub/sub broker with fan-out to SSE clients
- POST /webhook — receives Gitea push events, optional HMAC verification
- GET /events — SSE stream of push events
- GET /health — health check endpoint
- Zero external dependencies (stdlib only)
- Docker-based build via goctl wrapper (golang:1.26-alpine)
- systemd unit, README with deployment docs
- Tests: broker pub/sub, slow subscriber drop, ref parsing, signature verification
- 6.0M static binary
2026-05-14 13:54:26 -04:00
21 changed files with 1154 additions and 7 deletions
+2
View File
@@ -10,3 +10,5 @@ dist/
dist-newstyle/
.claude/
.pi/
converge-relay/.go-cache/
converge-relay/build/
+44 -1
View File
@@ -34,6 +34,25 @@ All core logic lives in the single `Converge` module (`src/Converge.hs`). Key se
- **Sync cycle** (`syncRepo`): guarded by `isInMiddleOfOperation`, then stage + commit + pull + conflict check
- **Git operations**: shell out to `git` via `System.Process.readProcessWithExitCode` — no libgit2 dependency
- **Notifications**: shell out to `notify-send` via `System.Process.spawnProcess`
- **Relay listener**: `connectRelay` connects to a converge-relay SSE endpoint, parses push events, and triggers pulls on matching repos (by `gitea_repo` + branch). Runs as a background thread with exponential backoff reconnection.
### Relay architecture
```
┌─────────────┐ POST /webhook ┌──────────────────┐ GET /events (SSE) ┌──────────────────┐
│ Gitea │ ────────────────> │ converge-relay │ ───────────────────> │ converge (local) │
│ Server │ (push event) │ (Go, hosted) │ event: push │ (Haskell) │
└─────────────┘ └──────────────────┘ └─────────┬─────────┘
matches gitea_repo
→ git pull
```
The Go relay is a zero-dependency pub/sub server. Key design:
- `POST /webhook` — receives Gitea push events, optional HMAC-SHA256 verification
- `GET /events` — SSE stream, clients connect and hold the connection open
- Channel-based broker: `Subscribe()` → channel, `Unsubscribe(ch)`, `Publish(event)` — non-blocking send to all subscribers
- Slow subscribers are silently dropped (channel buffer = 16)
### Git operations
@@ -62,6 +81,8 @@ All core logic lives in the single `Converge` module (`src/Converge.hs`). Key se
## Build / test / run
### Haskell (converge)
```bash
stack build # Build library + executable + tests
stack test # Build and run the test suite
@@ -71,6 +92,15 @@ stack exec converge -- [options] # Run the tool directly
./scripts/run # Run the tool (uses Docker via ./hs)
```
### Go (converge-relay)
```bash
cd converge-relay
./scripts/build # Format + test + build (uses Docker via ./goctl)
./scripts/test # Run tests (uses Docker via ./goctl)
./scripts/run --listen :8080 # Run the relay (uses Docker via ./goctl)
```
## Test conventions
Tests use **hspec** and create real temporary git repositories — no mocking of git commands. This avoids complexity and tests actual git semantics.
@@ -105,6 +135,7 @@ Anytime new behavior is added the `SPEC.md` should be updated and a correspondin
| Pulls from remote origin | `GitPullSpec` | Fast-forward pull scenario |
| Rebases on pull | `GitPullSpec` | Divergent history → linear history, no merge commits |
| No-op during operations | `GitSafetySpec` | All 7 sentinel types + clean repo |
| `RelaySpec` | SSE line parsing, event folding, JSON parsing, repo matching, operation-in-progress skip |
## Key conventions
@@ -117,17 +148,29 @@ Anytime new behavior is added the `SPEC.md` should be updated and a correspondin
## Dependencies
### Haskell
| Package | Purpose |
|---|---|
| `async` | Concurrent repo watching |
| `fsnotify` | File system event monitoring |
| `optparse-applicative` | CLI argument parsing |
| `yaml` | Config file parsing |
| `http-client` | HTTP client for SSE relay connection |
| `http-client-tls` | TLS support for HTTPS relay endpoints |
| `http-types` | HTTP status codes |
| `aeson` | JSON parsing (push events from relay) |
| `bytestring` | Efficient binary data for SSE stream parsing |
| `temporary` (test only) | Temp directories for test repos |
| `hspec` (test only) | Test framework |
### Go (converge-relay)
Zero external dependencies — stdlib only (`net/http`, `log/slog`, `crypto/hmac`, `encoding/json`, `sync`).
## External requirements
- `git` must be on PATH (no libgit2 dependency)
- `notify-send` for desktop notifications (optional — only called on conflict)
- `notify-send` for desktop notifications (optional — only called on conflict/pull)
- `hostname` command for commit message generation
- For realtime pull: a running `converge-relay` instance reachable via HTTP(S)
+5
View File
@@ -18,3 +18,8 @@
## Can notify on pull: per-repo notify_on_pull setting sends a desktop notification when new commits are pulled from the remote
## --sync / -s flag runs a single sync cycle (commit + pull + push) and exits, useful for wake-from-sleep hooks
## Can connect to a converge-relay SSE endpoint and automatically pull when push events arrive
### Configured via --relay-url CLI flag or relay_url config file entry
### Matches repos by gitea_repo field (Gitea repository full_name) and branch
### Reconnects automatically with exponential backoff on disconnect or error
+18 -5
View File
@@ -24,6 +24,8 @@ data CliArgs = CliArgs
-- ^ Log file path.
, cliSync :: !Bool
-- ^ Run a single sync cycle and exit (useful for wake-from-sleep hooks).
, cliRelayUrl :: !(Maybe Text)
-- ^ URL of a converge-relay SSE endpoint (overrides config).
}
cliParser :: Parser CliArgs
@@ -97,6 +99,13 @@ cliParser =
<> short 's'
<> help "Run a single sync cycle (commit + pull + push) and exit"
)
<*> optional
( strOption
( long "relay-url"
<> metavar "URL"
<> help "URL of a converge-relay SSE endpoint for realtime push notifications"
)
)
parserInfo :: ParserInfo CliArgs
parserInfo =
@@ -114,7 +123,7 @@ main = do
(options, configSource) <- case cliConfig cli of
Just cfgPath -> do
cfg <- loadConfig cfgPath
let opts = applyOverrides mDebounceOverride (cliLogLevel cli) (cliLogFile cli) (configToOptions cfg)
let opts = applyOverrides mDebounceOverride (cliLogLevel cli) (cliLogFile cli) (cliRelayUrl cli) (configToOptions cfg)
pure (opts, Just cfgPath)
Nothing -> do
exists <- configFileExists
@@ -122,7 +131,7 @@ main = do
then do
cfgPath <- defaultConfigPath
cfg <- loadConfig cfgPath
let opts = applyOverrides mDebounceOverride (cliLogLevel cli) (cliLogFile cli) (configToOptions cfg)
let opts = applyOverrides mDebounceOverride (cliLogLevel cli) (cliLogFile cli) (cliRelayUrl cli) (configToOptions cfg)
pure (opts, Just cfgPath)
else
pure
@@ -130,6 +139,7 @@ main = do
mDebounceOverride
(cliLogLevel cli)
(cliLogFile cli)
(cliRelayUrl cli)
defaultOptions
{ optRepos =
[ defaultRepoConfig
@@ -168,13 +178,16 @@ main = do
mapM_ (\r -> hPutStrLn stderr (" - " <> rcPath r)) repos
runConverge options
-- | Apply CLI overrides (debounce, log level, log file) to Options.
applyOverrides :: Maybe Int -> Maybe LogLevel -> Maybe FilePath -> Options -> Options
applyOverrides mDebounce mLogLevel mLogFile opts =
-- | Apply CLI overrides (debounce, log level, log file, relay url) to Options.
applyOverrides :: Maybe Int -> Maybe LogLevel -> Maybe FilePath -> Maybe Text -> Options -> Options
applyOverrides mDebounce mLogLevel mLogFile mRelayUrl opts =
opts
{ optDebounceMs = maybe (optDebounceMs opts) id mDebounce
, optLogLevel = fromMaybe (optLogLevel opts) mLogLevel
, optLogFile = case mLogFile of
Just _ -> mLogFile -- CLI --log-file overrides config
Nothing -> optLogFile opts
, optRelayUrl = case mRelayUrl of
Just _ -> mRelayUrl -- CLI --relay-url overrides config
Nothing -> optRelayUrl opts
}
+89
View File
@@ -0,0 +1,89 @@
# converge-relay
Webhook-to-SSE relay for [converge](https://github.com/jbrechtel/converge). Receives Gitea push webhooks and fans them out to connected converge clients via Server-Sent Events.
## How it works
```
┌─────────────┐ POST /webhook ┌──────────────────┐ GET /events (SSE) ┌──────────────────┐
│ Gitea │ ────────────────> │ converge-relay │ ──────────────────> │ converge (local) │
│ Server │ (push event) │ (this server) │ event: push │ │
└─────────────┘ └──────────────────┘ └──────────────────┘
```
1. Configure Gitea to send push webhooks to `https://your-server:8080/webhook`
2. Run `converge-relay` on a publicly reachable server (or behind nginx)
3. Point your local `converge` at the relay's SSE endpoint with `--relay-url http://your-server:8080/events`
## Build & run (Docker-only, no host Go tools required)
```bash
# Build (formats, tests, compiles) → ./build/converge-relay
./scripts/build
# Run tests only
./scripts/test
# Run in development
./scripts/run --listen :8080
# Run go commands directly
./goctl go fmt ./...
./goctl go mod tidy
```
## Configuration
| Flag | Env var | Default | Description |
|------|---------|---------|-------------|
| `--listen` | `CONVERGE_RELAY_LISTEN` | `:8080` | Address to listen on |
| `--secret` | `CONVERGE_RELAY_SECRET` | `""` | Shared secret for HMAC webhook verification |
## Endpoints
### `POST /webhook`
Receives Gitea push webhooks. Validates `X-Gitea-Signature` or `X-Hub-Signature-256` against the configured secret. Extracts `repository.full_name` and branch name from `ref`.
### `GET /events`
SSE stream. Clients connect and receive push events as they arrive:
```
event: push
data: {"full_name":"myorg/myrepo","branch":"main","ref":"refs/heads/main"}
```
Automatic reconnection with `Last-Event-ID` is supported. When the connection drops, clients reconnect with exponential backoff.
### `GET /health`
Returns `ok\n` — useful for load balancer health checks.
## Deployment
```bash
# Build
./scripts/build
# Install
sudo cp build/converge-relay /usr/local/bin/
# Install and enable the systemd service
sudo cp converge-relay.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now converge-relay
```
For production, set the `CONVERGE_RELAY_SECRET` environment variable in the service file or drop-in.
## Gitea webhook setup
1. Go to your repository → Settings → Webhooks → Add Webhook → Gitea
2. Target URL: `https://your-server:8080/webhook`
3. Secret: same value as `CONVERGE_RELAY_SECRET`
4. Events: Push events only
## Docker image
Built via `golang:1.26-alpine`. No external Go dependencies — stdlib only.
+94
View File
@@ -0,0 +1,94 @@
package main
import "log/slog"
// PushEvent is broadcast to all SSE subscribers when a webhook is received.
// It contains just enough information for converge clients to decide whether
// to pull.
type PushEvent struct {
FullName string `json:"full_name"` // e.g. "myorg/myrepo"
Branch string `json:"branch"` // e.g. "main"
Ref string `json:"ref"` // e.g. "refs/heads/main"
}
// Broker implements a pub/sub fan-out for PushEvents.
// Subscribe to get a channel that receives events; unsubscribe when done.
// Slow subscribers (channel full) are silently dropped for that event.
type Broker struct {
subscribe chan chan PushEvent
unsubscribe chan chan PushEvent
publish chan PushEvent
stop chan struct{}
subs map[chan PushEvent]struct{}
}
// NewBroker creates a ready-to-run Broker.
func NewBroker() *Broker {
return &Broker{
subscribe: make(chan chan PushEvent),
unsubscribe: make(chan chan PushEvent),
publish: make(chan PushEvent, 64),
stop: make(chan struct{}),
subs: make(map[chan PushEvent]struct{}),
}
}
// Run starts the broker event loop. Must be called in its own goroutine.
func (b *Broker) Run() {
for {
select {
case ch := <-b.subscribe:
b.subs[ch] = struct{}{}
slog.Debug("subscriber connected", "total", len(b.subs))
case ch := <-b.unsubscribe:
delete(b.subs, ch)
// Drain and close the channel so readers exit cleanly
close(ch)
for range ch {
}
slog.Debug("subscriber disconnected", "total", len(b.subs))
case evt := <-b.publish:
slog.Info("publishing event", "full_name", evt.FullName, "branch", evt.Branch)
for ch := range b.subs {
select {
case ch <- evt:
default:
// Subscriber too slow; drop the event for this subscriber
slog.Warn("dropping event for slow subscriber")
}
}
case <-b.stop:
for ch := range b.subs {
close(ch)
}
return
}
}
}
// Subscribe returns a buffered channel that receives PushEvents.
func (b *Broker) Subscribe() chan PushEvent {
// Small buffer to smooth out delivery; slow consumers are dropped by
// the broker's non-blocking publish anyway.
ch := make(chan PushEvent, 16)
b.subscribe <- ch
return ch
}
// Unsubscribe removes a subscriber channel.
func (b *Broker) Unsubscribe(ch chan PushEvent) {
b.unsubscribe <- ch
}
// Publish sends an event to all current subscribers. Never blocks.
func (b *Broker) Publish(evt PushEvent) {
b.publish <- evt
}
// Stop shuts down the broker.
func (b *Broker) Stop() {
close(b.stop)
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"net/http"
"testing"
"time"
)
func TestBrokerSubscribePublishUnsubscribe(t *testing.T) {
broker := NewBroker()
go broker.Run()
defer broker.Stop()
ch1 := broker.Subscribe()
ch2 := broker.Subscribe()
evt := PushEvent{FullName: "org/repo", Branch: "main", Ref: "refs/heads/main"}
// Publish
broker.Publish(evt)
// Both subscribers should receive
select {
case got := <-ch1:
if got != evt {
t.Errorf("ch1: got %+v, want %+v", got, evt)
}
case <-time.After(time.Second):
t.Fatal("ch1: timed out waiting for event")
}
select {
case got := <-ch2:
if got != evt {
t.Errorf("ch2: got %+v, want %+v", got, evt)
}
case <-time.After(time.Second):
t.Fatal("ch2: timed out waiting for event")
}
// Unsubscribe ch2, publish again
broker.Unsubscribe(ch2)
// Drain ch2 (Unsubscribe closes it)
for range ch2 {
}
evt2 := PushEvent{FullName: "org/repo2", Branch: "dev", Ref: "refs/heads/dev"}
broker.Publish(evt2)
// ch1 gets it
select {
case got := <-ch1:
if got != evt2 {
t.Errorf("ch1 second: got %+v, want %+v", got, evt2)
}
case <-time.After(time.Second):
t.Fatal("ch1: timed out waiting for second event")
}
// ch2 is closed, should not receive
_, open := <-ch2
if open {
t.Error("ch2 should be closed after unsubscribe")
}
}
func TestBrokerSlowSubscriberDropped(t *testing.T) {
broker := NewBroker()
go broker.Run()
defer broker.Stop()
// Channel with buffer 0 — publish will always fail to send
ch := make(chan PushEvent)
broker.subs[ch] = struct{}{} // bypass Subscribe for manual control
evt := PushEvent{FullName: "org/repo", Branch: "main"}
broker.Publish(evt)
// Give it time to process
time.Sleep(50 * time.Millisecond)
// Should not have blocked the broker; Publish returned fine
// and we didn't deadlock.
select {
case <-ch:
t.Error("slow subscriber should have been dropped")
default:
// expected: no event on unbuffered channel
}
// ch is closed by broker.Stop() via defer
}
func TestRefToBranch(t *testing.T) {
tests := []struct {
ref string
want string
}{
{"refs/heads/main", "main"},
{"refs/heads/feature/x", "feature/x"},
{"refs/tags/v1.0", ""},
{"refs/heads/", ""},
{"main", ""},
}
for _, tt := range tests {
got := refToBranch(tt.ref)
if got != tt.want {
t.Errorf("refToBranch(%q) = %q, want %q", tt.ref, got, tt.want)
}
}
}
func TestVerifySignature(t *testing.T) {
secret := "test-secret"
body := []byte(`{"ref":"refs/heads/main"}`)
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(body)
validSig := "sha256=" + hex.EncodeToString(mac.Sum(nil))
headers := http.Header{}
headers.Set("X-Gitea-Signature", validSig)
if !verifySignature(body, secret, headers) {
t.Error("valid signature should pass")
}
headers.Set("X-Gitea-Signature", "sha256=bad")
if verifySignature(body, secret, headers) {
t.Error("bad signature should fail")
}
headers.Del("X-Gitea-Signature")
headers.Set("X-Hub-Signature-256", validSig)
if !verifySignature(body, secret, headers) {
t.Error("X-Hub-Signature-256 with valid signature should pass")
}
}
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Converge relay - webhook-to-SSE relay server
Documentation=https://github.com/jbrechtel/converge
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=%h/.local/bin/converge-relay --listen :8080
Restart=on-failure
RestartSec=5
# Set a shared secret for webhook HMAC verification (optional but recommended).
# Generate with: openssl rand -hex 32
# Environment=CONVERGE_RELAY_SECRET=your-secret-here
[Install]
WantedBy=default.target
+3
View File
@@ -0,0 +1,3 @@
module github.com/jbrechtel/converge-relay
go 1.26
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Thin wrapper to run Go tooling inside a golang:1.26-alpine Docker container.
# Usage: ./goctl go build ./..., ./goctl go test ./..., ./goctl go fmt ./...
set -euo pipefail
IMAGE="${CONVERGE_GO_IMAGE:-golang:1.26-alpine}"
PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
GO_CACHE="${PROJECT_DIR}/.go-cache"
mkdir -p "${GO_CACHE}/mod" "${GO_CACHE}/build"
exec docker run --rm -i $([ -t 0 ] && printf -- -t) \
-v "${PROJECT_DIR}:/work" \
-v "${GO_CACHE}/mod:/go/pkg/mod" \
-v "${GO_CACHE}/build:/root/.cache/go-build" \
-e GOMODCACHE=/go/pkg/mod \
-e GOCACHE=/root/.cache/go-build \
-w /work \
"${IMAGE}" \
"$@"
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"context"
"errors"
"flag"
"log/slog"
"net/http"
"os"
"os/signal"
)
func main() {
listen := flag.String("listen", envDefault("CONVERGE_RELAY_LISTEN", ":8080"), "address to listen on")
secret := flag.String("secret", envDefault("CONVERGE_RELAY_SECRET", ""), "shared secret for Gitea webhook HMAC verification")
flag.Parse()
logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
slog.SetDefault(logger)
broker := NewBroker()
go broker.Run()
mux := http.NewServeMux()
mux.HandleFunc("POST /webhook", webhookHandler(broker, *secret))
mux.HandleFunc("GET /events", sseHandler(broker))
mux.HandleFunc("GET /health", healthHandler)
srv := &http.Server{Addr: *listen, Handler: mux}
// Graceful shutdown
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, os.Interrupt)
<-sig
slog.Info("shutting down")
broker.Stop()
srv.Shutdown(context.Background())
}()
slog.Info("converge-relay starting", "listen", *listen)
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
slog.Error("server error", "error", err)
os.Exit(1)
}
}
func envDefault(key, fallback string) string {
if v, ok := os.LookupEnv(key); ok {
return v
}
return fallback
}
func healthHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.Write([]byte("ok\n"))
}
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
echo "Formatting..."
./goctl go fmt ./...
echo "Running tests..."
./goctl go test ./...
echo "Building..."
./goctl go build -ldflags="-s -w" -o /work/build/converge-relay .
echo "Done: $(ls -lh build/converge-relay | awk '{print $5}')"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
./goctl go run . "$@"
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
./goctl go test -v ./...
+59
View File
@@ -0,0 +1,59 @@
package main
import (
"encoding/json"
"fmt"
"log/slog"
"net/http"
)
func sseHandler(broker *Broker) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "streaming not supported", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("X-Accel-Buffering", "no") // disable nginx buffering
ch := broker.Subscribe()
defer broker.Unsubscribe(ch)
slog.Debug("SSE client connected")
ctx := r.Context()
for {
select {
case <-ctx.Done():
slog.Debug("SSE client disconnected")
return
case evt, ok := <-ch:
if !ok {
// Broker shut down
return
}
writeSSEEvent(w, flusher, "push", evt)
}
}
}
}
func writeSSEEvent(w http.ResponseWriter, flusher http.Flusher, eventType string, data any) {
jsonData, err := json.Marshal(data)
if err != nil {
slog.Warn("failed to marshal SSE event", "error", err)
return
}
_, err = fmt.Fprintf(w, "event: %s\ndata: %s\n\n", eventType, jsonData)
if err != nil {
slog.Debug("failed to write SSE event, client likely disconnected", "error", err)
return
}
flusher.Flush()
}
+96
View File
@@ -0,0 +1,96 @@
package main
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"log/slog"
"net/http"
"strings"
)
// giteaPushPayload mirrors the relevant fields from Gitea's push webhook JSON.
type giteaPushPayload struct {
Ref string `json:"ref"`
Repository struct {
FullName string `json:"full_name"`
} `json:"repository"`
}
func webhookHandler(broker *Broker, secret string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20)) // 1 MB limit
if err != nil {
slog.Warn("failed to read webhook body", "error", err)
http.Error(w, "bad request", http.StatusBadRequest)
return
}
// Verify signature if a secret is configured
if secret != "" && !verifySignature(body, secret, r.Header) {
slog.Warn("webhook signature verification failed")
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
// Only handle push events
eventType := r.Header.Get("X-Gitea-Event")
if eventType != "" && eventType != "push" {
slog.Debug("ignoring non-push event", "type", eventType)
w.WriteHeader(http.StatusOK)
return
}
var payload giteaPushPayload
if err := json.Unmarshal(body, &payload); err != nil {
slog.Warn("failed to parse webhook payload", "error", err)
http.Error(w, "bad request", http.StatusBadRequest)
return
}
branch := refToBranch(payload.Ref)
if branch == "" {
slog.Debug("ignoring non-branch ref", "ref", payload.Ref)
w.WriteHeader(http.StatusOK)
return
}
broker.Publish(PushEvent{
FullName: payload.Repository.FullName,
Branch: branch,
Ref: payload.Ref,
})
w.WriteHeader(http.StatusOK)
}
}
// verifySignature checks the request body against the HMAC-SHA256 signature
// in X-Gitea-Signature or X-Hub-Signature-256 headers.
func verifySignature(body []byte, secret string, headers http.Header) bool {
sig := headers.Get("X-Gitea-Signature")
if sig == "" {
sig = headers.Get("X-Hub-Signature-256")
}
if sig == "" {
return false
}
// Strip "sha256=" prefix if present
sig = strings.TrimPrefix(sig, "sha256=")
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(body)
expected := hex.EncodeToString(mac.Sum(nil))
return hmac.Equal([]byte(sig), []byte(expected))
}
// refToBranch extracts the branch name from a git ref like "refs/heads/main".
func refToBranch(ref string) string {
const prefix = "refs/heads/"
if strings.HasPrefix(ref, prefix) {
return ref[len(prefix):]
}
return ""
}
+4
View File
@@ -2,14 +2,18 @@
# Place at ~/.config/converge/config.yaml to use by default,
# or pass explicitly with: converge --config converge.yaml
# relay_url: "http://localhost:8080/events" # converge-relay SSE endpoint for realtime push notifications
repos:
- path: /home/jbrechtel/org
remote: origin
branch: main
# gitea_repo: "myorg/org-repo" # Gitea repository full_name for relay matching
# notify_on_pull: true # send a notification when new commits are pulled
- path: /work/personal/hawat
# remote defaults to "origin"
# branch defaults to "main"
# gitea_repo: "myorg/hawat"
# debounce_ms: 5000 # optional, defaults to 5000
+5
View File
@@ -29,10 +29,15 @@ ghc-options:
dependencies:
- base >= 4.7 && < 5
- aeson
- async
- bytestring
- directory
- filepath
- fsnotify
- http-client
- http-client-tls
- http-types
- optparse-applicative
- process
- text
+168 -1
View File
@@ -35,20 +35,36 @@ module Converge (
-- * File filtering
isGitIgnored,
-- * Relay listener (SSE client)
PushEvent (..),
connectRelay,
extractLines,
foldLines,
parseEvent,
handleSSEEvent,
)
where
import Control.Applicative ((<|>))
import Control.Concurrent (forkIO, threadDelay)
import Control.Concurrent.Async (mapConcurrently_)
import Control.Concurrent.MVar (MVar, newEmptyMVar, newMVar, takeMVar, tryPutMVar, tryTakeMVar, withMVar)
import Control.Exception (bracket)
import Control.Exception (SomeException, bracket, try)
import Control.Monad (forever, unless, void, when)
import Data.Aeson (decode)
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy.Char8 as BL
import Data.Maybe (fromMaybe)
import Data.Text (Text)
import qualified Data.Text as T
import Data.Text.Encoding (decodeUtf8)
import Data.Time.Clock (getCurrentTime)
import Data.Time.Format (defaultTimeLocale, formatTime)
import Data.Yaml (FromJSON (..), decodeFileThrow, withObject, withText, (.:), (.:?))
import Network.HTTP.Client
import Network.HTTP.Client.TLS (tlsManagerSettings)
import Network.HTTP.Types.Status (status200)
import System.Directory (XdgDirectory (XdgConfig), doesDirectoryExist, doesFileExist, getCurrentDirectory, getXdgDirectory)
import System.Exit (ExitCode (..))
import System.FSNotify
@@ -66,6 +82,8 @@ data RepoConfig = RepoConfig
-- ^ Branch to pull.
, rcNotifyOnPull :: !Bool
-- ^ Send a notification when new commits are pulled from the remote.
, rcGiteaRepo :: !(Maybe Text)
-- ^ Gitea repository full_name (e.g. "myorg/myrepo") for relay matching.
}
deriving (Show, Eq)
@@ -97,6 +115,8 @@ data Options = Options
-- ^ Minimum log level to emit.
, optLogFile :: !(Maybe FilePath)
-- ^ Optional log file path (logs to stderr when Nothing).
, optRelayUrl :: !(Maybe Text)
-- ^ URL of a converge-relay SSE endpoint for realtime push notifications.
}
deriving (Show, Eq)
@@ -108,6 +128,7 @@ defaultRepoConfig =
, rcRemote = "origin"
, rcBranch = "main"
, rcNotifyOnPull = False
, rcGiteaRepo = Nothing
}
-- | Sensible defaults for options.
@@ -119,6 +140,7 @@ defaultOptions =
, optNotifyCmd = "notify-send"
, optLogLevel = Info
, optLogFile = Nothing
, optRelayUrl = Nothing
}
{- | Logger that serializes output so concurrent repo watchers
@@ -153,6 +175,7 @@ data Config = Config
, cfgDebounceMs :: !(Maybe Int)
, cfgLogLevel :: !(Maybe LogLevel)
, cfgLogFile :: !(Maybe FilePath)
, cfgRelayUrl :: !(Maybe Text)
}
deriving (Show, Eq)
@@ -163,6 +186,7 @@ instance FromJSON Config where
<*> o .:? "debounce_ms"
<*> o .:? "log_level"
<*> o .:? "log_file"
<*> o .:? "relay_url"
instance FromJSON RepoConfig where
parseJSON = withObject "RepoConfig" $ \o ->
@@ -171,6 +195,7 @@ instance FromJSON RepoConfig where
<*> (fromMaybe "origin" <$> o .:? "remote")
<*> (fromMaybe "main" <$> o .:? "branch")
<*> (fromMaybe False <$> o .:? "notify_on_pull")
<*> o .:? "gitea_repo"
-- | Load a YAML config file.
loadConfig :: FilePath -> IO Config
@@ -184,6 +209,7 @@ configToOptions cfg =
, optDebounceMs = fromMaybe 5000 (cfgDebounceMs cfg)
, optLogLevel = fromMaybe Info (cfgLogLevel cfg)
, optLogFile = cfgLogFile cfg
, optRelayUrl = cfgRelayUrl cfg
}
-- | The default XDG config file path: @$XDG_CONFIG_HOME/converge/config.yaml@.
@@ -239,6 +265,13 @@ logRepo logger repo msg = logRepoAt logger Info repo msg
runConverge :: Options -> IO ()
runConverge opts = do
logger <- newLogger (optLogLevel opts) (optLogFile opts)
-- Start relay listener if configured (runs in background forever)
case optRelayUrl opts of
Just url
| url /= "" -> do
_ <- forkIO $ connectRelay url logger (optRepos opts)
pure ()
_ -> pure ()
mapConcurrently_ (watchRepo opts logger) (optRepos opts)
{- | Watch a single repository for changes.
@@ -518,3 +551,137 @@ runGitIn :: FilePath -> [String] -> IO (ExitCode, Text, Text)
runGitIn repoPath args = do
(code, out, err) <- readProcessWithExitCode "git" ("-C" : repoPath : args) ""
pure (code, T.pack out, T.pack err)
----------------------------------------------------------------------
-- Relay listener (SSE client)
----------------------------------------------------------------------
-- | Push event received from a converge-relay SSE endpoint.
data PushEvent = PushEvent
{ peFullName :: !Text
-- ^ Gitea repository full_name (e.g. "myorg/myrepo").
, peBranch :: !Text
-- ^ Branch name.
, peRef :: !Text
-- ^ Full ref (e.g. "refs/heads/main").
}
deriving (Show, Eq)
instance FromJSON PushEvent where
parseJSON = withObject "PushEvent" $ \o ->
PushEvent
<$> o .: "full_name"
<*> o .: "branch"
<*> o .: "ref"
{- | Connect to a converge-relay SSE endpoint and trigger pulls on matching
repositories when push events arrive. Runs forever, reconnecting with
exponential backoff (1s .. 60s) on disconnection or error.
This is spawned as a background thread by 'runConverge' when an
@optRelayUrl@ is configured.
-}
connectRelay :: Text -> Logger -> [RepoConfig] -> IO ()
connectRelay url logger repos = do
logMsg logger Info ("relay: connecting to " <> url)
req <- parseRequest (T.unpack url)
manager <- newManager tlsManagerSettings
let loop delay = do
let delaySec = min delay (60 :: Int)
result <- try $
withResponse req manager $ \resp ->
if responseStatus resp /= status200
then logMsg logger Error ("relay returned " <> T.pack (show (responseStatus resp)))
else do
logMsg logger Info "relay: connected, waiting for push events"
processSSE logger repos (responseBody resp)
case result of
Left (e :: SomeException) ->
logMsg logger Warn ("relay: connection error: " <> T.pack (show e) <> ", reconnecting in " <> T.pack (show delaySec) <> "s")
Right () ->
logMsg logger Info ("relay: connection closed, reconnecting in " <> T.pack (show delaySec) <> "s")
threadDelay (delaySec * 1000000)
loop (delaySec * 2)
loop 1
-- | Process an SSE stream from a 'BodyReader', dispatching push events to matching repos.
processSSE :: Logger -> [RepoConfig] -> BodyReader -> IO ()
processSSE logger repos reader = go BS.empty Nothing []
where
go buf mEvtType dataLines = do
chunk <- brRead reader
if BS.null chunk
then pure ()
else do
let (completeLines, leftover) = extractLines (buf <> chunk)
let (mEvtType', dataLines', events) = foldLines mEvtType dataLines completeLines
mapM_ (handleSSEEvent logger repos) events
go leftover mEvtType' dataLines'
-- | Split a 'BS.ByteString' into complete lines (stripping trailing \r) and leftover partial data.
extractLines :: BS.ByteString -> ([BS.ByteString], BS.ByteString)
extractLines bs = case BS.breakSubstring "\n" bs of
(line, rest)
| not (BS.null rest) ->
let line' = stripTrailingCr line
(lines', leftover) = extractLines (BS.drop 1 rest)
in (line' : lines', leftover)
_ -> ([], bs)
-- | Strip trailing \r from a line (for CRLF line endings).
stripTrailingCr :: BS.ByteString -> BS.ByteString
stripTrailingCr bs
| not (BS.null bs) && BS.last bs == cr = BS.init bs
| otherwise = bs
where
cr = 13 -- '\r'
-- | Fold a list of SSE lines into accumulated state and parsed events.
foldLines :: Maybe Text -> [BS.ByteString] -> [BS.ByteString] -> (Maybe Text, [BS.ByteString], [PushEvent])
foldLines evtType0 dLines0 sseLines =
let (evt, dl, evts) = foldl' step (evtType0, dLines0, []) sseLines
in (evt, dl, reverse evts)
where
step (evt, dl, events) line
| BS.null line =
case dl of
[] -> (Nothing, [], events)
_ -> case parseEvent evt dl of
Just evt' -> (Nothing, [], evt' : events)
Nothing -> (Nothing, [], events)
| Just rest <- pick "event: " line <|> pick "event:" line =
(Just (decodeUtf8 (BS.dropWhile (== space) rest)), dl, events)
| Just rest <- pick "data: " line <|> pick "data:" line =
(evt, dl ++ [BS.dropWhile (== space) rest], events)
| otherwise = (evt, dl, events)
pick = BS.stripPrefix
space = 32 -- ' '
-- | Parse accumulated SSE data lines into a 'PushEvent' (push events only).
parseEvent :: Maybe Text -> [BS.ByteString] -> Maybe PushEvent
parseEvent (Just "push") dataLines =
decode (BL.fromStrict (BS.intercalate "\n" dataLines))
parseEvent _ _ = Nothing
-- | Dispatch a parsed push event to matching repos (by 'rcGiteaRepo' and 'rcBranch').
handleSSEEvent :: Logger -> [RepoConfig] -> PushEvent -> IO ()
handleSSEEvent logger repos evt = do
let matching =
filter
( \r ->
rcGiteaRepo r == Just (peFullName evt)
&& rcBranch r == peBranch evt
)
repos
if null matching
then logMsg logger Debug ("relay: ignoring event for unconfigured repo: " <> peFullName evt <> " (branch: " <> peBranch evt <> ")")
else
mapM_
( \repo -> do
logRepo logger repo "relay: push event received, pulling"
inProgress <- isInMiddleOfOperation (rcPath repo)
if inProgress
then logRepoAt logger Warn repo "relay: skipping pull -- operation in progress"
else void $ gitPull logger repo
)
matching
+307
View File
@@ -0,0 +1,307 @@
module RelaySpec (spec) where
import Converge (
PushEvent (..),
RepoConfig (..),
extractLines,
foldLines,
handleSSEEvent,
parseEvent,
)
import qualified Data.ByteString.Char8 as BSC
import qualified Data.Text as T
import System.FilePath ((</>))
import Test.Hspec
import TestHelper (commitCount, runGitIn, testLogger, withTempGitRepoWithRemote)
spec :: Spec
spec = do
extractLinesSpec
foldLinesSpec
parseEventSpec
handleSSEEventSpec
----------------------------------------------------------------------
-- extractLines
----------------------------------------------------------------------
extractLinesSpec :: Spec
extractLinesSpec = describe "extractLines" $ do
it "splits a single line" $ do
let (lines', leftover) = extractLines (bsc "\n")
lines' `shouldBe` [bsc ""]
leftover `shouldBe` bsc ""
it "splits multiple lines" $ do
let (lines', leftover) = extractLines (bsc "hello\nworld\n")
lines' `shouldBe` [bsc "hello", bsc "world"]
leftover `shouldBe` bsc ""
it "handles CRLF line endings" $ do
let (lines', leftover) = extractLines (bsc "hello\r\nworld\r\n")
lines' `shouldBe` [bsc "hello", bsc "world"]
leftover `shouldBe` bsc ""
it "returns partial line as leftover" $ do
let (lines', leftover) = extractLines (bsc "hello\nwor")
lines' `shouldBe` [bsc "hello"]
leftover `shouldBe` bsc "wor"
it "returns empty for empty input" $ do
let (lines', leftover) = extractLines (bsc "")
lines' `shouldBe` []
leftover `shouldBe` bsc ""
it "handles line with only CR" $ do
let (lines', leftover) = extractLines (bsc "\r\n")
lines' `shouldBe` [bsc ""]
leftover `shouldBe` bsc ""
----------------------------------------------------------------------
-- foldLines
----------------------------------------------------------------------
foldLinesSpec :: Spec
foldLinesSpec = describe "foldLines" $ do
it "parses a single push event" $ do
let lines' =
[ bsc "event: push"
, bsc "data: {\"full_name\":\"myorg/myrepo\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"
, bsc ""
]
(finalEvtType, finalDLines, events) = foldLines Nothing [] lines'
events
`shouldBe` [ PushEvent
{ peFullName = "myorg/myrepo"
, peBranch = "main"
, peRef = "refs/heads/main"
}
]
finalEvtType `shouldBe` Nothing
finalDLines `shouldBe` []
it "parses two push events" $ do
let lines' =
[ bsc "event: push"
, bsc "data: {\"full_name\":\"a/b\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"
, bsc ""
, bsc "event: push"
, bsc "data: {\"full_name\":\"c/d\",\"branch\":\"dev\",\"ref\":\"refs/heads/dev\"}"
, bsc ""
]
(_, _, events) = foldLines Nothing [] lines'
length events `shouldBe` 2
peFullName (events !! 0) `shouldBe` "a/b"
peFullName (events !! 1) `shouldBe` "c/d"
it "ignores non-push events" $ do
let lines' =
[ bsc "event: ping"
, bsc "data: hello"
, bsc ""
]
(_, _, events) = foldLines Nothing [] lines'
events `shouldBe` []
it "ignores comment lines (starting with colon)" $ do
let lines' =
[ bsc ": this is a comment"
, bsc "event: push"
, bsc "data: {\"full_name\":\"a/b\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"
, bsc ""
]
(_, _, events) = foldLines Nothing [] lines'
length events `shouldBe` 1
it "handles event: without space after colon" $ do
let lines' =
[ bsc "event:push"
, bsc "data:{\"full_name\":\"a/b\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"
, bsc ""
]
(_, _, events) = foldLines Nothing [] lines'
length events `shouldBe` 1
it "carries partial state across calls" $ do
-- First batch: incomplete event (missing blank line)
let batch1 = [bsc "event: push", bsc "data: {\"full_name\":\"a/b\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"]
(evtType1, dLines1, events1) = foldLines Nothing [] batch1
events1 `shouldBe` []
evtType1 `shouldBe` Just "push"
dLines1 `shouldBe` [bsc "{\"full_name\":\"a/b\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"]
-- Second batch: completes the event
let batch2 = [bsc ""]
(_, _, events2) = foldLines evtType1 dLines1 batch2
length events2 `shouldBe` 1
----------------------------------------------------------------------
-- parseEvent
----------------------------------------------------------------------
parseEventSpec :: Spec
parseEventSpec = describe "parseEvent" $ do
it "parses a valid push event" $ do
let evt = parseEvent (Just "push") [bsc "{\"full_name\":\"org/repo\",\"branch\":\"main\",\"ref\":\"refs/heads/main\"}"]
evt
`shouldBe` Just
PushEvent
{ peFullName = "org/repo"
, peBranch = "main"
, peRef = "refs/heads/main"
}
it "returns Nothing for non-push event type" $ do
let evt = parseEvent (Just "ping") [bsc "{}"]
evt `shouldBe` Nothing
it "returns Nothing for malformed JSON" $ do
let evt = parseEvent (Just "push") [bsc "not json"]
evt `shouldBe` Nothing
it "handles multiple data lines joined" $ do
let evt =
parseEvent
(Just "push")
[ bsc "{\"full_name\":\"x\""
, bsc ",\"branch\":\"y\""
, bsc ",\"ref\":\"z\"}"
]
evt
`shouldBe` Just
PushEvent
{ peFullName = "x"
, peBranch = "y"
, peRef = "z"
}
----------------------------------------------------------------------
-- handleSSEEvent (integration — uses real git repos)
----------------------------------------------------------------------
handleSSEEventSpec :: Spec
handleSSEEventSpec = describe "handleSSEEvent" $ do
it "pulls on a matching repo when a push event arrives" $
withTempGitRepoWithRemote $ \localRepo -> do
logger <- testLogger
-- Simulate someone else pushing: commit locally, push, then reset local behind
writeFile (localRepo </> "remote-change.txt") "pushed from elsewhere"
_ <- runGitIn localRepo ["add", "remote-change.txt"]
_ <- runGitIn localRepo ["commit", "-m", "simulated remote push"]
_ <- runGitIn localRepo ["push", "origin", "main"]
-- Verify remote has the commit
(_, remoteLog, _) <- runGitIn localRepo ["log", "--oneline", "origin/main"]
let remoteCount = length (filter (not . T.null) (T.lines remoteLog))
-- Reset local repo to one commit behind
_ <- runGitIn localRepo ["reset", "--hard", "HEAD~1"]
-- Verify local is behind
localBefore <- commitCount localRepo
localBefore `shouldBe` (remoteCount - 1)
-- Create a repo config with matching gitea_repo
let repoCfg =
(defaultRepoCfg localRepo)
{ rcGiteaRepo = Just "test-org/test-repo"
}
-- Send a matching push event
let evt =
PushEvent
{ peFullName = "test-org/test-repo"
, peBranch = "main"
, peRef = "refs/heads/main"
}
handleSSEEvent logger [repoCfg] evt
-- Local should now have the remote's commit
localAfter <- commitCount localRepo
localAfter `shouldBe` remoteCount
it "ignores events for repos with non-matching gitea_repo" $
withTempGitRepoWithRemote $ \localRepo -> do
logger <- testLogger
let repoCfg =
(defaultRepoCfg localRepo)
{ rcGiteaRepo = Just "test-org/test-repo"
}
let evt =
PushEvent
{ peFullName = "other-org/other-repo"
, peBranch = "main"
, peRef = "refs/heads/main"
}
-- This should not crash or pull — it just ignores
handleSSEEvent logger [repoCfg] evt
-- Test passes if no exception is thrown
it "ignores events for repos with non-matching branch" $
withTempGitRepoWithRemote $ \localRepo -> do
logger <- testLogger
let repoCfg =
(defaultRepoCfg localRepo)
{ rcGiteaRepo = Just "test-org/test-repo"
, rcBranch = "main"
}
let evt =
PushEvent
{ peFullName = "test-org/test-repo"
, peBranch = "dev"
, peRef = "refs/heads/dev"
}
commitsBefore <- commitCount localRepo
handleSSEEvent logger [repoCfg] evt
commitsAfter <- commitCount localRepo
commitsAfter `shouldBe` commitsBefore -- no pull should have happened
it "skips pull when an operation is in progress" $
withTempGitRepoWithRemote $ \localRepo -> do
logger <- testLogger
-- Create a MERGE_HEAD sentinel to simulate a merge in progress
(_, gitDirOut, _) <- runGitIn localRepo ["rev-parse", "--git-dir"]
let sentinelFile = localRepo </> T.unpack (T.strip gitDirOut) </> "MERGE_HEAD"
writeFile sentinelFile "fake merge head"
-- Push a new commit to remote
writeFile (localRepo </> "during-merge.txt") "change during merge"
_ <- runGitIn localRepo ["add", "during-merge.txt"]
_ <- runGitIn localRepo ["commit", "-m", "commit during merge"]
_ <- runGitIn localRepo ["push", "origin", "main"]
_ <- runGitIn localRepo ["reset", "--hard", "HEAD~1"]
commitsBefore <- commitCount localRepo
let repoCfg =
(defaultRepoCfg localRepo)
{ rcGiteaRepo = Just "test-org/test-repo"
}
let evt =
PushEvent
{ peFullName = "test-org/test-repo"
, peBranch = "main"
, peRef = "refs/heads/main"
}
handleSSEEvent logger [repoCfg] evt
commitsAfter <- commitCount localRepo
commitsAfter `shouldBe` commitsBefore -- pull should be skipped
----------------------------------------------------------------------
-- Helpers
----------------------------------------------------------------------
-- | Shortcut for ByteString from ASCII.
bsc :: String -> BSC.ByteString
bsc = BSC.pack
-- | Create a default RepoConfig for a given path (for testing).
defaultRepoCfg :: FilePath -> RepoConfig
defaultRepoCfg path =
RepoConfig
{ rcPath = path
, rcRemote = "origin"
, rcBranch = "main"
, rcNotifyOnPull = False
, rcGiteaRepo = Nothing
}
+2
View File
@@ -3,6 +3,7 @@ module Main (main) where
import qualified GitCommitSpec
import qualified GitPullSpec
import qualified GitSafetySpec
import qualified RelaySpec
import Test.Hspec
main :: IO ()
@@ -10,3 +11,4 @@ main = hspec $ do
GitCommitSpec.spec
GitPullSpec.spec
GitSafetySpec.spec
RelaySpec.spec