MishimaDb

MishimaDb is a process-embedded, single-file storage engine in C# (embedded engine 1.1.0; shares MishimaWire for CRC). It targets a self-describing on-disk layout (header + schema registry + slab data region), memory-mapped read access, optional WAL durability, snapshot-isolated reads, and single-writer transactions. The product roadmap spans P0 → P5; the GA kernel ships durability, MVCC, indexes, engine ports, replication log, optional encryption, and operability—not a full SQL surface (that is Nexus).

This repository is the engine: binary format specs live under docs/, and normative byte layout is described in docs/format-mvp-v1.adoc. For the broader suite story, see the Mishima workspace architecture docs.

Why MishimaDb

Repository layout

Architecture overview

High-level placement of the primary file, WAL sidecar, and host process:

flowchart TB
  subgraph host["Host application (.NET)"]
    API["MishimaDb API\nTryOpenReadOnly / TryOpenLive"]
  end
  subgraph engine["MishimaDb engine"]
    MAP["Memory-mapped read\nMemoryMappedFile"]
    LIVE["Live state + WAL appender\nMVCC row bag"]
    CP["Checkpoint / vacuum hooks"]
  end
  subgraph disk["Disk"]
    PF["Primary file\n*.mishima\nheader · schema · slab"]
    WAL["Sidecar WAL\n*.mishima.wal\nappend + CRC frames"]
  end
  API --> MAP
  API --> LIVE
  LIVE --> WAL
  MAP --> PF
  LIVE --> PF
  CP --> PF

Read path (P0-style)

Opening read-only maps the primary file, parses the header and schema, and exposes rows as enumerables or a scoped slab cursor (slab v1 fixed Int32 packing and/or SlabV2 length-prefixed bodies depending on format_revision and layout kind).

flowchart LR
  A[TryOpenReadOnly] --> B[Parse 512-byte header]
  B --> C[Load schema registry blob]
  C --> D[Resolve slab layout per logical type]
  D --> E[Enumerate rows or\nIMishimaSlabRowCursor]
  E --> F[LINQ / application code]

Live write path (P1/P2)

A live open attaches a WAL, replays frames into in-memory state, and uses a single-writer lock for transactions: begin → staged inserts/deletes → commit appends WAL and advances visibility.

sequenceDiagram
  participant App as Host
  participant Live as MishimaLiveDatabase
  participant WAL as WAL file
  participant Mem as LiveEngineState
  App->>Live: TryBeginWriteTransaction
  Live->>Mem: stage inserts / deletes
  App->>Live: Commit
  Live->>WAL: append frames (CRC payloads)
  Live->>Mem: apply committed txn
  App->>Live: TryBeginReadSnapshot
  Live->>Mem: MVCC-visible rows for snapshot id

Roadmap phases (engine)

Milestone phases from the architecture roadmap; product MVP is P0 + P1 + P2 (embedded durability + transactions/MVCC). SQL and parallel execution land later.

flowchart LR
  P0["P0 — mmap + materialized reads"] --> P1["P1 — storage + WAL"]
  P1 --> P2["P2 — transactions + MVCC"]
  P2 --> P3["P3 — SQL + indexes"]
  P3 --> P4["P4 — parallel executor"]
  P4 --> P5["P5 — differentiators"]

For full definitions, see docs/mishima-db-engine-architecture-roadmap.adoc.

Build & test

From this directory (mishima-db):

dotnet build src/MishimaDb/MishimaDb.csproj
dotnet test tests/MishimaDb.Tests/MishimaDb.Tests.csproj

The test project enforces line coverage on the MishimaDb assembly via Coverlet (see Threshold in the test .csproj).

Pack a NuGet (MishimaDb assembly)

dotnet pack src/MishimaDb/MishimaDb.csproj -c Release -o artifacts/nuget
# or release list + version check:
# bash scripts/pack-release.sh   # → nupkgs/MishimaDb.{Version}.nupkg

Produces artifacts/nuget/MishimaDb.1.0.0.nupkg (version from Directory.Build.props; output folder is gitignored).

Pre-tag gate (local RC, no tag/push):

bash scripts/pre-tag-release-gate.sh
# Full Coverlet ≥90% before merging to main:
# WOODPECKER_COLLECT_COVERAGE=true bash scripts/pre-tag-release-gate.sh

CI release on main (Woodpecker): scripts/woodpecker-verify.sh then scripts/woodpecker-release.sh (tag v{Version}, pack, push to download.robotico.dev).

Normal mode (no RAM) + observables

Normal mode means a live database without MishimaRamPersistencePlan: omit RamPersistence on MishimaCreateLiveOptions / MishimaLiveOpenOptions (it defaults to null). WAL behavior is controlled by WalDurability (Strict = flush per frame where applicable; Throughput = deferred/async WAL where supported — typical for desktop throughput).

MishimaLiveDatabase exposes BCL-only IObservable<T> streams (no System.Reactive dependency on the engine package; see docs/adr/0002-mvp-observables.adoc). Emissions for commits and checkpoints occur after the engine write lock is released, so observers can safely marshal to a UI thread.

Minimal pattern: subscribe with an IObserver<T> (or add System.Reactive only in your host app for ergonomic Subscribe overloads).

using System;
using System.Collections.Immutable;
using MishimaDb;
using Robotico.Result;

sealed class CommitObserver : IObserver<MishimaCommitNotice>
{
    private readonly Action<MishimaCommitNotice> _onNext;
    public CommitObserver(Action<MishimaCommitNotice> onNext) => _onNext = onNext;
    public void OnNext(MishimaCommitNotice value) => _onNext(value);
    public void OnError(Exception error) { }
    public void OnCompleted() { }
}

// Open existing DB (no RamPersistence → normal mode)
var open = new MishimaLiveOpenOptions(
    DatabaseFilePath: "/path/to/db.mishima",
    WalFilePath: "/path/to/db.mishima.wal",
    WalDurability: MishimaWalDurabilityMode.Throughput);

Result<MishimaLiveDatabase> opened = MishimaLiveDatabaseFactory.TryOpenLive(open);
if (!opened.IsSuccess(out MishimaLiveDatabase? live, out IError? err))
{
    Console.WriteLine(err!.Code);
    return;
}

using (live!.CommittedWrites.Subscribe(new CommitObserver(_ =>
{
    // e.g. MainThread.BeginInvokeOnMainThread(RefreshList);
})))
{
    Result<MishimaWriteTransaction> txr = live.TryBeginWriteTransaction();
    if (!txr.IsSuccess(out MishimaWriteTransaction? tx, out _))
        return;
    using (tx!)
    {
        _ = tx.TryInsertInt32Row("User", ImmutableArray.Create(1, 42));
        _ = tx.Commit();
    } // CommittedWrites fires after Dispose releases the write transaction
}

For polling instead of reactive updates, call live.TryGetStorageHealth() on a timer (same metrics as StorageHealthSampled samples). Deeper host guidance: docs/hosting-embedded.adoc.

Related

• MishimaDocs / mishima-docs — collection-oriented document API (IMishimaDatabase); different file format from this engine. See the roadmap section Relationship to MishimaDocs in the architecture doc.
• Robotico.Result — shared Result types for command boundaries.

Documentation index