A new experimental Rust-based SQLite Virtual File System (VFS) implementation demonstrates the viability of executing cold database queries directly from S3 object storage with sub-second latency—and frequently achieving even faster response times.
Dubbed turbolite, this proof-of-concept implementation remains in early experimental stages with known stability issues and potential data integrity risks, making it unsuitable for production workloads at this juncture.
The project investigates a fundamental question: has object storage infrastructure evolved sufficiently to support embedded database operations over cloud storage? Traditional filesystems excel at handling small random read operations and in-place data mutations—characteristics that clash with S3's architectural strengths. Object storage performs optimally with fewer request counts, larger data transfers, immutable objects, and highly parallelized operations where network bandwidth typically becomes the bottleneck. The architectural approach draws direct inspiration from turbopuffer's native S3-optimized design principles.
The target scenario centers on managing numerous predominantly dormant SQLite databases—common in database-per-tenant, database-per-session, or database-per-user architectural patterns—where maintaining dedicated attached volumes for inactive databases represents inefficient resource allocation. The turbolite design assumes a single-writer model and targets "many databases experiencing intermittent cold reads" rather than "one continuously active hot database" use cases.
Rather than implementing naive page-by-page retrieval from raw SQLite files, turbolite performs deep introspection of SQLite B-tree structures, consolidates related pages into compressed page groups, and maintains a manifest serving as the authoritative source for page location mapping. Cache misses leverage seekable zstd compression frames combined with S3 range GET requests for search operations, enabling selective page retrieval without downloading entire objects.
During query execution, turbolite can propagate storage operation hints from the query planner down to the VFS layer, enabling proactive prefetching of indexes and large scan operations in their anticipated access sequence.
The prefetching aggressiveness is configurable based on query patterns. For point queries and compact joins, the system maintains conservative prefetching to avoid unnecessary table downloads. For full table scans, it adopts substantially more aggressive prefetching strategies.
The implementation also employs page-type-based grouping in S3 storage. Interior B-tree nodes are bundled separately and loaded eagerly. Index pages trigger aggressive prefetching. Data pages are organized by table. This architecture aims to deliver acceptable performance for cold point queries and joins while mitigating the performance degradation that naive remote paging would impose on scan operations.
Benchmark results on a 1-million-row, 1.5GB dataset running on EC2 with S3 Express show promising metrics: cold point lookups completing in under 100 milliseconds, cold 5-table join profile queries executing in under 200 milliseconds, and full table scans from an empty cache completing in under 600 milliseconds with the 1.5GB database. Performance degrades somewhat when using standard S3 or Tigris storage.
Current constraints are clearly defined: the system supports only single-writer configurations and remains firmly positioned as a systems research experiment rather than production-ready infrastructure.
The project seeks input from practitioners with experience in SQLite-over-network implementations, storage engine development, VFS architectures, or object-storage-backed database systems. Particular interest lies in validating whether the B-tree-aware grouping, manifest-based tracking, and seekable-range-GET approach represents a promising direction for continued development.
Comments URL: https://news.ycombinator.com/item?id=47534283
Points: 120
# Comments: 25
