Publishing Datasets

A dataset on ChipAI.dev has three parts:

• Metadata — name, modality, license, tags (shown on cards and the detail page)
• Code & docs — README, dataset card, loading script (in an auto-provisioned git repo)
• Data files — Parquet, CSV, WebDataset, audio/images (in object storage)

Before you start

• You need to be signed in with at least the contributor role.
• Pick a slug ahead of time — it's permanent, used in the URL (/datasets/<slug>) and the git repo name.

1. Create the listing

Open Create Dataset on the Datasets page (or Create → Dataset from the profile menu), then fill in:

You can also publish under an organization you belong to.

2. Pending review

New datasets are created with pending status and reviewed by a moderator before appearing in public listings. You can still see and work on yours from the My Datasets strip on the Datasets page while it's pending.

3. Push code & docs (git)

Creating a dataset auto-provisions a git repository. Open the clone panel on the dataset page to copy its URL:

git clone <clone-url>
cd <repo>

# add a dataset card + loading script
$EDITOR README.md
git add .
git commit -m "Add dataset card"
git push

Text and code (README, loading scripts, dataset_infos.json) live here. You can also edit files directly in the in-browser code editor on the dataset page.

4. Upload data files

Large data files don't go in git — they stream to object storage:

• In the browser: use the file browser on the dataset page (upload / download / delete). Files over ~100 MiB automatically switch to a presigned direct upload.
• Object keys follow datasets/<slug>/<path>.

Programmatic upload

# $API = https://api.chaimind.in (or your local backend)
curl -X PUT \
  -H "Authorization: Bearer <token>" \
  --data-binary @train.parquet \
  "$API/api/v1/datasets/<slug>/files/data/train.parquet"

For very large files, request a presigned URL first (POST /api/v1/datasets/<slug>/upload-url) and PUT the bytes directly to it.

5. Write a good dataset card

Put these in your README.md:

• What's in it — fields, splits, size
• Collection & preprocessing methodology
• Intended use and out-of-scope uses
• Licensing, consent and PII handling
• Limitations & biases

Versioning

Use the git repo's history and tags for code/card versions, and upload new revisions under clear paths (e.g. data/v2/). The dataset page always serves the current files from storage.

Other ways in

• Import an existing GitHub repo instead of starting from scratch — see Import from GitHub.
• Consuming a dataset (browse, download, clone)? See Using & Downloading Datasets.

Next: Using & Downloading Datasets · Browsing & Discovery · Publishing Models.