JsonlStorage

Storage over a directory of JSON-lines files, ported from pi-durable’s JSONL storage.

main.jsonl is the append-only commit log: one commit marker per line, carrying every small write inline. Live task records and document content go to sidecars (task-<id>.jsonl, doc-<id>.jsonl), appended before the marker that confirms them. A commit is durable once its marker line is complete. Opening the directory replays the log into an in-memory index (a MemoryStore), drops a torn last line of every file, truncates unconfirmed sidecar tails, and finishes interrupted reclamation. Sidecars of terminal tasks and of retired or rebased current-only documents are reclaimed after the marker that makes them obsolete.

One process owns a directory at a time, as in pi: nothing locks it, so two concurrent owners corrupt it. After a failed append the storage is poisoned and must be reopened, which recovers exactly the confirmed commits.

import { Harness, JsonlStorage, Registry } from "@effective-harness/core"
import * as Layer from "effect/Layer"

const layer = Harness.layer({}).pipe(
  Layer.provideMerge(Registry.layer([])),
  Layer.provideMerge(JsonlStorage.layer({ directory: "./.harness", fsync: true }))
)

Since v0.0.0

---

utils

FileSystem (interface)

The file operations the storage needs. nodeFileSystem is the default; tests substitute one that injects failures.

Signature

export interface FileSystem {
  /** Create a directory and its parents; succeed when it exists. */
  readonly makeDirectory: (path: string) => Effect.Effect<void, StorageFailure>
  /** The file's bytes; `None` when it does not exist. */
  readonly readFile: (path: string) => Effect.Effect<Option.Option<Uint8Array>, StorageFailure>
  /** Names of the regular files in a directory. */
  readonly listFiles: (path: string) => Effect.Effect<ReadonlyArray<string>, StorageFailure>
  /** Append to a file, creating it when absent. */
  readonly appendFile: (path: string, content: string) => Effect.Effect<void, StorageFailure>
  /** Replace a file's content, creating it when absent. */
  readonly writeFile: (path: string, content: string) => Effect.Effect<void, StorageFailure>
  /** Flush a file's data to stable storage. */
  readonly sync: (path: string) => Effect.Effect<void, StorageFailure>
  readonly truncate: (path: string, size: number) => Effect.Effect<void, StorageFailure>
  readonly rename: (from: string, to: string) => Effect.Effect<void, StorageFailure>
  /** Remove a file; succeed when it is absent. */
  readonly remove: (path: string) => Effect.Effect<void, StorageFailure>
}

Source

Since v0.0.0

Options (interface)

Options of a JSONL storage.

Signature

export interface Options {
  /** The storage directory; created when absent. Relative paths resolve against the process's working directory. */
  readonly directory: string
  /**
   * Flush every affected sidecar before appending a commit marker, and the marker before reclaiming sidecars.
   * Default: `false` (the operating system decides when appended lines reach the disk).
   */
  readonly fsync?: boolean
  /** Default: `nodeFileSystem`. */
  readonly fileSystem?: FileSystem
}

Source

Since v0.0.0

layer

A Storage over a JSONL directory. Building the layer opens and recovers the directory, failing with StorageFailure when it cannot be read or holds corrupt confirmed data; closing the layer’s scope closes it.

Signature

declare const layer: (options: Options) => Layer.Layer<Storage, StorageFailure>

Source

Since v0.0.0

make

Open (or create) the storage directory, recover it, and return a Storage that closes with the scope.

Signature

declare const make: any

Source

Since v0.0.0

nodeFileSystem

The local file system through node:fs/promises.

Signature

declare const nodeFileSystem: FileSystem

Source

Since v0.0.0