Platform
ScaiWave ScaiGrid ScaiCore ScaiBot ScaiDrive ScaiKey Models Tools & Services
Solutions
Organisations Developers Internet Service Providers Managed Service Providers AI-in-a-Box
Resources
Support Documentation Blog Downloads
Company
About Research Careers Investment Opportunities Contact
Log in

Storage & distribution

How model bytes move in and out of ScaiAtlas, and how clusters share them.

Storage backends#

ScaiAtlas stores metadata in a database and bytes in an object store. Two backends:

  • S3-compatible (default) — production deployments. Multipart uploads and range reads happen against the bucket, mediated by the API.
  • Filesystem — single-node deployments. Files are served through signed, time-limited URLs (/api/v1/files/{key}) validated by HMAC signature and expiry, so no auth token is needed for the actual byte transfer.

Either way, clients never hold storage credentials. The API proxies or signs every transfer.

Uploads are chunked and verified#

Uploads are multipart. The lifecycle:

  1. Create the version — POST …/versions puts it in uploading state.
  2. Init a file — POST …/upload/init with the file's path, size, and SHA-256; the server returns an upload_id, a recommended part_size, and part_count.
  3. Push parts — PUT …/upload/{upload_id}/parts/{n} for each chunk; each returns an etag.
  4. Complete — POST …/upload/{upload_id}/complete with the part list; the file is assembled and checksum-verified.

The SDK's upload session does all of this for you — see Uploading a model. Default chunk size is 10 MB; you can override it.

Downloads resume and verify#

Downloads support HTTP range requests, so an interrupted transfer restarts from where it stopped rather than from zero:

  • GET …/download returns file metadata and URLs for a version (optionally a specific variant or file).
  • GET …/download/files/{path} streams a file, honoring a Range: bytes=start-end header and replying 206 Partial Content with a Content-Range. It also sets X-Checksum-SHA256 and Accept-Ranges: bytes.

The SDK verifies each file's SHA-256 after download and skips files already present with a matching checksum. See Downloading a model.

Variants: many formats, one version#

A version's original upload is its primary variant. You can request additional variants — e.g. a GGUF quantization of a safetensors original — with POST …/variants, which returns 202 Accepted and creates the variant in pending state while conversion runs (pending → converting → available). Downloads can target a specific variant with the variant parameter.

Peer-to-peer distribution#

For clusters that pull the same model repeatedly, ScaiAtlas can distribute via P2P instead of every node hitting origin storage.

Manifests#

A distribution manifest breaks a version into fixed-size chunks (default 64 MB), each with its own checksum:

bash
1
2
GET  …/versions/{version}/manifest    # fetch
POST …/versions/{version}/manifest    # 501 — generation is not built

Chunked distribution is not implemented today. No manifest is generated, so GET answers 404 and POST answers 501. Clusters that share model bytes between nodes currently fetch a whole version from one peer and verify it against the per-file SHA-256 checksums the version already carries. The chunked form below is the design, not the running behaviour.

The manifest lists every file, its chunks, and per-chunk offsets/sizes/checksums — enough for a peer to fetch chunks from wherever they're available and reassemble with integrity.

The tracker#

When P2P is enabled, peers announce what they hold and discover each other through the tracker under /api/v1/p2p:

  • POST /p2p/announce — a peer advertises its address and which version chunks it has (seeding vs. still leeching).
  • GET /p2p/peers/by-version?version_id=… — find peers holding a version, split into seeders and leechers.
  • GET /p2p/peers and GET /p2p/stats — the full roster and tracker-wide counts. Administrators only: together they list every node's address and port. A peer looking for a swarm uses by-version, which answers for one version at a time.

If P2P is disabled on a deployment, these endpoints return 503.

A node id has an owner#

The first announce of a node_id claims it for the caller that made it. From then on only that caller can move the peer — change its address, port or chunk list — and anyone else gets 403. Since clients resolve peer addresses through by-version, an unowned node id would let anyone redirect a swarm.

Ownership is tied to the credential, not kept forever. Revoking the key that claimed a peer releases it, so a machine rebuilt under a new worker key simply reclaims its node id on the next announce. An administrator can also take a node over, and a node can delete its own peer when it retires.

P2P is an optimization layer. Correctness never depends on it — origin storage is always authoritative, and every chunk is checksum-verified on arrival.

Updated 2026-10-18 01:55:51 View source (.md) rev 3