Storage & distribution

How model bytes move in and out of ScaiAtlas, and how clusters share them.

Storage backends#

ScaiAtlas stores metadata in a database and bytes in an object store. Two backends:

• S3-compatible (default) — production deployments. Multipart uploads and range reads happen against the bucket, mediated by the API.
• Filesystem — single-node deployments. Files are served through signed, time-limited URLs (/api/v1/files/{key}) validated by HMAC signature and expiry, so no auth token is needed for the actual byte transfer.

Either way, clients never hold storage credentials. The API proxies or signs every transfer.

Uploads are chunked and verified#

Uploads are multipart. The lifecycle:

1. Create the version — POST …/versions puts it in uploading state.
2. Init a file — POST …/upload/init with the file's path, size, and SHA-256; the server returns an upload_id, a recommended part_size, and part_count.
3. Push parts — PUT …/upload/{upload_id}/parts/{n} for each chunk; each returns an etag.
4. Complete — POST …/upload/{upload_id}/complete with the part list; the file is assembled and checksum-verified.

The SDK's upload session does all of this for you — see Uploading a model. Default chunk size is 10 MB; you can override it.

Downloads resume and verify#

Downloads support HTTP range requests, so an interrupted transfer restarts from where it stopped rather than from zero:

• GET …/download returns file metadata and URLs for a version (optionally a specific variant or file).
• GET …/download/files/{path} streams a file, honoring a Range: bytes=start-end header and replying 206 Partial Content with a Content-Range. It also sets X-Checksum-SHA256 and Accept-Ranges: bytes.

The SDK verifies each file's SHA-256 after download and skips files already present with a matching checksum. See Downloading a model.

Variants: many formats, one version#

A version's original upload is its primary variant. You can request additional variants — e.g. a GGUF quantization of a safetensors original — with POST …/variants, which returns 202 Accepted and creates the variant in pending state while conversion runs (pending → converting → available). Downloads can target a specific variant with the variant parameter.

Peer-to-peer distribution#

For clusters that pull the same model repeatedly, ScaiAtlas can distribute via P2P instead of every node hitting origin storage.

Manifests#

A distribution manifest breaks a version into fixed-size chunks (default 64 MB), each with its own checksum:

1
2

POST …/versions/{version}/manifest    # generate (201)
GET  …/versions/{version}/manifest    # fetch

The manifest lists every file, its chunks, and per-chunk offsets/sizes/checksums — enough for a peer to fetch chunks from wherever they're available and reassemble with integrity.

The tracker#

When P2P is enabled, peers announce what they hold and discover each other through the tracker under /api/v1/p2p:

• POST /p2p/announce — a peer advertises its address and which version chunks it has (seeding vs. still leeching).
• GET /p2p/peers/by-version?version_id=… — find peers holding a version, split into seeders and leechers.
• GET /p2p/stats — tracker-wide counts.

If P2P is disabled on a deployment, these endpoints return 503.

P2P is an optimization layer. Correctness never depends on it — origin storage is always authoritative, and every chunk is checksum-verified on arrival.