Skip to main content

Datasets

A dataset is a structured collection of items (e.g. questions, prompts, or text samples) that will be used to evaluate and compare AI models. Each row represents one item to be scored.

QuickCompare accepts datasets in two formats: JSONL and CSV. Size limits depend on your plan; please consult your profile to know the limits of your plan.

JSONL​

We support JSONL files that contain one JSON object per line, flat (no nested fields), with the same keys on every line.

{"question": "What is the capital of France?", "expected_answer": "Paris", "category": "geography"}
{"question": "Who wrote Hamlet?", "expected_answer": "Shakespeare", "category": "literature"}
{"question": "What is 2 + 2?", "expected_answer": "4", "category": "math"}

CSV​

We support CSV files with a single header row, comma-separated, with fields containing commas, quotes, or newlines wrapped in double quotes.

question,expected_answer,category
"What is the capital of France?","Paris","geography"
"Who wrote ""Hamlet""?","Shakespeare","literature"
"What is 2 + 2?","4","math"

Column names​

You can access columns simply by selecting them in the UI if you use the simple column input mode or static metrics.

Column names become {{ row.<name> }} references in Jinja templates and LLM-as-Judge rubrics, so we recommend you to descriptive names. Names with spaces or other non-identifier characters still work via bracket syntax — {{ row['field name'] }}. response and row are reserved.

A single-column dataset (just input, for example) is valid. You don't need a reference answer unless your evaluation uses one.

Formatting with an agent​

If you have raw data in some other shape (a spreadsheet, a scraped corpus, a mix of files), we released an Agent Skill-based skill that walks an agent through preparing an upload-ready file.

Claude Code​

Install the skill with:

/plugin marketplace add trismik/skills
/plugin install quickcompare-format@trismik-skills
/reload-plugins

Invoke the skill with /quickcompare-format, or just describe your task and Claude will load it when the description matches.

Codex:​

Install the skill with:

$skill-installer install https://github.com/trismik/skills/tree/main/skills/quickcompare-format

Restart Codex after install; then you can just invoke the skill with

$quickcompare-format

Other Agent CLIs​

If you have other agent CLIs (Cursor, Gemini), you can find point them to the trismik/skills GitHub repo.