Sabitlenmiş Tweet
Introducing Crab: a serverless Git remote storage solution for teams working with large files.
Git is still the best collaboration model we have for software: branches, commits, review, rollback, CI, and history.
But modern repositories no longer contain only code.
They contain model weights, datasets, media assets, simulation outputs, build artifacts, checkpoints, design files, and other large binary payloads that can easily reach gigabytes or terabytes.
That creates a painful tradeoff.
Plain Git bloats history.
Git LFS adds another server, endpoint, quota model, and operational surface.
Manual object-storage buckets keep bytes somewhere durable, but they separate the data from Git history and make collaboration harder to reason about.
Crab is built around a simpler idea:
Keep Git as the interface.
Use your cloud object storage as the backend.
Remove the server layer in between.
With Crab, developers keep using familiar Git workflows. Commits, branches, reviews, worktrees, CI, and rollback still behave like Git. Large-file bytes are routed into customer-owned object storage such as S3, GCS, or Azure Blob.
The repository stays lightweight because Git carries metadata and tiny pointer files, while the real payloads live durably in your bucket.
A normal workflow looks like:
crab init
crab add models/
git commit
crab push
Or, where configured, teams can keep using familiar Git commands like `git add` and `git push` through Crab’s Git integration.
Under the hood, Crab uses Git remote-helper and filter-process concepts to make large-file handling feel native. The developer works in Git, while Crab manages chunking, deduplication, upload, checkout, hydration, and storage layout.
The technical model is designed for large binary data.
Large files are split into content-defined chunks, so small edits do not necessarily rewrite the entire file. Crab can identify which chunks are already known and which chunks are new.
Known chunks skip upload.
New chunks upload once.
Chunks are packed into xorbs to reduce duplicate storage, bandwidth, and object request volume.
That matters when teams are moving ML checkpoints, datasets, media, or build artifacts repeatedly across branches and CI jobs.
Clone and checkout are also designed for large repos.
Instead of forcing every user or CI job to materialize terabytes immediately, Crab can clone metadata first. Teams can inspect branches, review code, and start work quickly with lightweight pointer checkout, then hydrate real bytes only when needed.
Access modes are flexible:
crab hydrate downloads selected content.
crab dehydrate frees local disk by returning files to pointer form.
crab mount supports on-demand reads through a virtual filesystem.
CI and desktop workflows can hydrate only the files they actually need.
For enterprise teams, the important part is ownership and control.
Crab does not require a central LFS server or a new database service. Storage stays in your cloud account. Your buckets, IAM, lifecycle rules, encryption posture, and identity model remain the control plane.
Crab is designed to fit enterprise auth and operational requirements: customer-owned buckets, IAM, OIDC, Entra ID, Workload Identity, encrypted local token cache, structured CLI output, and operational commands such as doctor, status, fsck, and gc.
This gives infra teams fewer services to run and gives developers a workflow that still feels like Git.
The business benefits are straightforward:
Less Git bloat.
No separate LFS server to operate.
Large files stay versioned with the repo.
Object storage remains customer-owned.
CI can avoid unnecessary downloads.
Teams can adopt gradually instead of rewriting everything at once.
Costs can be managed with lifecycle policies and storage tiering.
Existing Git habits remain intact.
Crab is especially useful for teams building with:
ML models and checkpoints
Training and evaluation datasets
Media and creative assets
Large generated artifacts
Scientific or simulation outputs
Game assets
Build cache outputs
Enterprise repos that have outgrown normal Git storage
The goal is not to replace Git.
The goal is to let Git keep doing what it is excellent at: collaboration, review, history, and coordination.
Crab turns object storage into an enterprise Git remote for large files.
Git-native large files.
Your cloud.
No server layer.
DM open for early users ;)
Powered by @huggingface xet protocol github.com/huggingface/xe…, @criccomini 's SlateDB github.com/slatedb/slatedb, and @theelbasian 's Gitoxide github.com/gitoxidelabs/g…
English
