Skip to content

AdrielC/graviton

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

505 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Graviton

DOCS

Graviton is a modular content-addressable storage runtime built on the ZIO ecosystem. The repository is structured as a multi-module build so that pure data types, streaming utilities, runtime ports, transport layers, and backend implementations evolve independently.

Building locally

./sbt compile

To execute the formatter and the unit suites:

TESTCONTAINERS=0 ./sbt scalafmtAll test

The documentation site is powered by VitePress and includes an interactive Scala.js frontend. Run it locally with:

# Build the interactive frontend
sbt buildFrontend

# Start the documentation server
cd docs
npm install
npm run docs:dev

Try the CAS Playground on the documentation site to explore chunking, hashing, and deduplication in your browser. Root-relative links such as /demo in a GitHub README resolve inside the repository tree (and 404); use the docs URL above instead.

Module map

graviton/
├─ modules/
│  ├─ graviton-core/        # pure domain types: hashing, keys, locators, ranges, manifests
│  ├─ graviton-streams/     # ZIO stream combinators, chunkers, hashing pipelines, scan helpers
│  ├─ graviton-runtime/     # runtime ports (BlobStore, BlockStore, policies, metrics)
│  ├─ graviton-cli/         # SBT runnable CLI: ingest / stat / get / verify (filesystem blocks + in-memory manifests)
│  ├─ protocol/
│  │  ├─ graviton-proto/    # protobuf definitions for the public RPC APIs
│  │  ├─ graviton-grpc/     # zio-grpc service stubs (server wiring to the runtime is still in progress)
│  │  └─ graviton-http/     # zio-http routes (blob upload/download, dashboard helpers)
│  ├─ backend/
│  │  ├─ graviton-s3/       # AWS SDK v2 backed object store bindings
│  │  ├─ graviton-pg/       # PostgreSQL powered manifest/metadata stores
│  │  └─ graviton-rocks/    # RocksDB backed key-value primitives with metrics adapters
│  └─ server/
│     └─ graviton-server/   # HTTP server, backend selection (fs / S3|MinIO), Postgres manifests, metrics
└─ docs/                    # VitePress documentation (architecture, manifests, API surface, operations)

The modules/zio-blocks directory is a git submodule that hosts the content-defined chunking primitives consumed by graviton-streams.

Contributing

See CONTRIBUTING.md for workflow guidance, coding standards, and required validation steps before opening a pull request.

High level flow

  1. Chunk and hash – chunkers in graviton-streams split byte streams into blocks; graviton-core supplies hashers and BinaryKey derivation.
  2. Derive locators – pure locator strategies in graviton-core map keys to storage paths (S3 prefixes, filesystem layout, etc.).
  3. PersistBlockStore implementations write canonical blocks; CasBlobStore builds manifests and uses BlobManifestRepo (Postgres in the server, or in-memory for the CLI / Graviton helpers).
  4. Index and trackReplicaIndex and related ports exist for replication metadata; several paths are still stubs or roadmap (see Replication).
  5. Servegraviton-server exposes an HTTP API for blobs, health, and dashboard endpoints; gRPC is defined in protos and libraries but not yet exposed as a full production server entrypoint alongside HTTP.

Consult the architecture document for a detailed walkthrough of the modules, runtime wiring, and extension points.

About

Unified framework for working with Binary Files

Resources

License

Contributing

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors