A high-performance, persistent key-value storage library written in Go. Supports three storage engines (B+tree, LSM-tree, and KV-Separation) with WAL (Write-Ahead Logging) and snapshot-based persistence.

• Three Storage Engines: Choose between B+tree, LSM-tree, or KV-Separation based on your workload
• Persistence: WAL + Snapshot for durability and fast recovery
• Concurrent Access: Thread-safe with fine-grained locking
• Range Queries: Efficient range scans and prefix searches
• Batch Operations: Atomic batch writes
• Configurable: Tunable parameters for different workloads
• Compression: Optional gzip compression for snapshots

go get cnb.cool/asphodelus_dev/gokv2

package main

import (
    "fmt"
    "log"

    "cnb.cool/asphodelus_dev/gokv2"
)

func main() {
    // Open database with default options (B+tree engine)
    db, err := gokv2.Open("./data")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // Put
    err = db.Put([]byte("hello"), []byte("world"))
    if err != nil {
        log.Fatal(err)
    }

    // Get
    value, err := db.Get([]byte("hello"))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Value: %s\n", value)

    // Delete
    err = db.Delete([]byte("hello"))
    if err != nil {
        log.Fatal(err)
    }
}

// Use B+tree engine (optimized for read-heavy workloads)
db, err := gokv2.Open("./data", gokv2.WithStorageEngine(gokv2.BTreeEngine))

// Use LSM-tree engine (optimized for write-heavy workloads)
db, err := gokv2.Open("./data", gokv2.WithStorageEngine(gokv2.LSMEngine))

// Use KV-Separation engine (default, optimized for large values)
db, err := gokv2.Open("./data", gokv2.WithStorageEngine(gokv2.KVSepEngine))

Feature	B+tree	LSM-tree	KV-Separation
Read Performance	Excellent	Good	Good
Write Performance	Good	Excellent	Excellent
Space Amplification	Low	Medium	Low
Write Amplification	Medium	Low	Very Low
Range Scans	Excellent	Good	Good
Large Values	Good	Medium	Excellent
Best For	Read-heavy, OLTP	Write-heavy, Logs	Large values, Mixed workload

• B+tree Engine: Traditional B+tree with in-place updates. Best for read-heavy workloads with balanced read/write patterns.

• LSM-tree Engine: Log-Structured Merge-tree with compaction. Optimized for write-heavy workloads with sequential writes.

• KV-Separation Engine (Default): Separates keys and values into different storage structures. Keys are stored in a sorted index (LSM-like), while values are stored in an append-only value log. Ideal for:

  • Large value sizes (>1KB)
  • Mixed read/write workloads
  • Scenarios requiring low write amplification
  • Applications with value log garbage collection needs

// Open database
func Open(path string, opts ...Option) (*DB, error)

// Close database
func (db *DB) Close() error

// Get value by key
func (db *DB) Get(key []byte) ([]byte, error)

// Put key-value pair
func (db *DB) Put(key, value []byte) error

// Delete key
func (db *DB) Delete(key []byte) error

// Check if key exists
func (db *DB) Has(key []byte) (bool, error)

// Get database size (number of keys)
func (db *DB) Size() int

// Force sync to disk
func (db *DB) Sync() error

// Create snapshot manually
func (db *DB) Snapshot() error

// Create iterator over all keys
func (db *DB) NewIterator() (*Iterator, error)

// Range iterator [start, end)
func (db *DB) Range(start, end []byte) (*Iterator, error)

// Prefix scan
func (db *DB) Scan(prefix []byte) (*Iterator, error)

// ForEach iteration
func (db *DB) ForEach(fn func(key, value []byte) bool) error

// Check if iterator is valid
func (it *Iterator) Valid() bool

// Move to next entry
func (it *Iterator) Next() bool

// Move to previous entry
func (it *Iterator) Prev() bool

// Get current key
func (it *Iterator) Key() []byte

// Get current value
func (it *Iterator) Value() []byte

// Seek to key >= target
func (it *Iterator) Seek(target []byte) bool

// Seek to first key
func (it *Iterator) SeekToFirst() bool

// Seek to last key
func (it *Iterator) SeekToLast() bool

// Close iterator
func (it *Iterator) Close() error

// Create batch
batch := db.Batch()

// Add operations
batch.Put([]byte("key1"), []byte("value1"))
batch.Put([]byte("key2"), []byte("value2"))
batch.Delete([]byte("key3"))

// Commit atomically
err := batch.Commit()

// Storage engine selection
gokv2.WithStorageEngine(gokv2.BTreeEngine)  // or gokv2.LSMEngine, gokv2.KVSepEngine

// B+tree order (default: 128)
gokv2.WithBTreeOrder(256)

// WAL sync mode
gokv2.WithSyncMode(gokv2.SyncAlways)  // SyncNone, SyncBatch, SyncAlways

// Snapshot settings
gokv2.WithSnapshotInterval(time.Hour)
gokv2.WithSnapshotWALThreshold(256 * 1024 * 1024)
gokv2.WithSnapshotOpsThreshold(100000)

// Compression
gokv2.WithCompression(true)

// Read-only mode
gokv2.WithReadOnly(true)

// LSM-specific options
gokv2.WithMemTableSize(64 * 1024 * 1024)  // 64MB
gokv2.WithLevel0FileNum(4)
gokv2.WithLevelSizeMultiplier(10)

// KVSep-specific options
gokv2.WithVLogSegmentSize(256 * 1024 * 1024)  // 256MB value log segment size

gokv2/
├── go.mod
├── gokv.go                 # Main entry, public API
├── options.go              # Configuration options
├── errors.go               # Error definitions
├── iterator.go             # Iterator interface
├── batch.go                # Batch operations
├── transaction.go          # MVCC transactions
│
├── internal/
│   ├── btree/              # B+tree storage engine
│   │   ├── node.go         # Node definition
│   │   ├── tree.go         # B+tree operations
│   │   └── iterator.go     # Tree iterator
│   │
│   ├── lsm/                # LSM-tree storage engine
│   │   ├── memtable.go     # In-memory table (skiplist)
│   │   ├── sstable.go      # Sorted String Table
│   │   ├── level.go        # Level management
│   │   ├── compaction.go   # Compaction strategy
│   │   └── lsm.go          # LSM-tree manager
│   │
│   ├── kvsep/              # KV-Separation storage engine
│   │   ├── kvsep.go        # KVSep manager
│   │   ├── index_file.go   # Sorted index for keys
│   │   ├── vlog.go         # Value log (append-only)
│   │   ├── vlog_gc.go      # Value log garbage collection
│   │   ├── memtable.go     # In-memory index
│   │   └── compaction.go   # Index compaction
│   │
│   ├── wal/                # Write-Ahead Log
│   │   ├── wal.go          # WAL manager
│   │   ├── entry.go        # Log entry format
│   │   ├── segment.go      # Segment management
│   │   └── reader.go       # Log reader
│   │
│   ├── snapshot/           # Snapshot management
│   │   ├── snapshot.go     # Snapshot manager
│   │   └── format.go       # Snapshot format
│   │
│   ├── mvcc/               # Multi-Version Concurrency Control
│   │   ├── txn_manager.go  # Transaction manager
│   │   ├── version.go      # Version store
│   │   └── gc.go           # Garbage collector
│   │
│   └── engine/             # Storage engine abstraction
│       ├── interface.go    # Engine interface
│       ├── engine.go       # B+tree engine implementation
│       ├── lsm_adapter.go  # LSM engine adapter
│       ├── kvsep_adapter.go# KVSep engine adapter
│       ├── recovery.go     # Data recovery
│       └── compaction.go   # Compaction triggers
│
└── cmd/
    └── test/
        └── main.go         # Integration test

Write Path:
  Client -> DB.Put() -> WAL.Write() -> Engine.Put() -> [B+tree/LSM/KVSep] -> Memory/Disk

Read Path:
  Client -> DB.Get() -> Engine.Get() -> [B+tree/LSM/KVSep] -> Memory/Disk

Recovery Path:
  Startup -> Load Snapshot -> Replay WAL -> Ready

Write Path:
  Key -> Index (LSM-like sorted structure)
  Value -> Value Log (append-only file)

Read Path:
  Key -> Index lookup -> Value pointer -> Value Log read

Garbage Collection:
  Scan Value Log -> Identify dead values -> Rewrite live values -> Reclaim space

| CRC32 (4B) | Type (1B) | KeyLen (4B) | ValueLen (4B) | Timestamp (8B) | SeqNum (8B) | Key | Value |

| Header (32B) | [Compressed Data] | Checksum (4B) |

Header:
| Magic (4B) | Version (4B) | SeqNum (8B) | EntryCount (8B) | Flags (4B) | Reserved (4B) |

Benchmark results on typical hardware (Intel i7-10750H, NVMe SSD):

Operation	B+tree	LSM-tree	KV-Separation
Sequential Write	~348K ops/s	~111K ops/s	~16K ops/s
Random Read	~500K ops/s	~200K ops/s	~200K ops/s
Range Scan (100 keys)	~2M keys/s	~1M keys/s	~1M keys/s
Mixed Workload	~300K ops/s	~400K ops/s	~350K ops/s

Note: KV-Separation performance improves significantly with larger value sizes (>4KB).

var (
    ErrKeyNotFound     // Key does not exist
    ErrKeyEmpty        // Key cannot be empty
    ErrKeyTooLarge     // Key exceeds max size
    ErrValueTooLarge   // Value exceeds max size
    ErrDatabaseClosed  // Database is closed
    ErrCorruptedData   // Data corruption detected
    ErrBatchClosed     // Batch already committed/closed
    ErrIteratorClosed  // Iterator is closed
)

All public methods are thread-safe. The library uses:

• RWMutex for read/write separation
• Fine-grained locking in B+tree nodes
• Lock-free structures where possible in LSM-tree
• MVCC (Multi-Version Concurrency Control) for transactions

GoKV2 supports ACID transactions with snapshot isolation:

// Begin a read-write transaction
txn, err := db.Begin()
if err != nil {
    log.Fatal(err)
}

// Perform operations
txn.Put([]byte("key1"), []byte("value1"))
txn.Put([]byte("key2"), []byte("value2"))
value, err := txn.Get([]byte("key1"))

// Commit or rollback
if err := txn.Commit(); err != nil {
    txn.Rollback()
    log.Fatal(err)
}

MIT License