Suite v0.1 · 2026-10-15
Methodology
ShipRanked ranks tools by the job, not by the size of a catalog. A score exists only after an agent runs the tool on a fixed task and the artifacts are kept. This page is the method. The launch set has not been scored yet.
What a run is
- Spec. A versioned task: the prompt, the starting repo commit, the acceptance tests, and a time and cost cap.
- Execute. CLI agents run headless in a fresh container. Web builders are driven in a clean browser profile with the screen recorded. API and SDK tools are integrated by a coding agent that is limited to the official docs.
- Collect. The final repo, build log, deployed URL, screenshots or video, token and credit cost, wall time, and the number of follow-up prompts.
- Score. Code first, then two LLM judges from different model families. A gap over 15 points on the judge rubric is flagged for a person.
- Repeat. Three runs. The published score is the median. The spread is the consistency number.
Weights
| Dimension | Weight | How it is measured |
|---|---|---|
| Task success | 40% | Hidden acceptance tests: Playwright end-to-end checks, unit tests, and HTTP checks. The number is the pass rate. |
| Output quality | 20% | Lint, typecheck, Lighthouse, and a security scan. Then two LLM judges from different model families, averaged. A disagreement over 15 points is flagged for a person to review. |
| Speed | 10% | Wall time from the start of the run to a passing state. |
| Cost | 10% | Dollars per successful task at the plan price used for that run. |
| Effort | 10% | Follow-up prompts or other interventions needed to reach a passing state. |
| Consistency | 10% | Spread across three runs. The published score is the median, not the best run. |
Rules that keep a rank honest
- The same plan tier for every tool in a job: the cheapest paid plan, or free if that is all there is. The tier is printed next to the score.
- The top 3 only includes runs from the last 60 days. Older scores are marked stale and stay visible, but they do not hold a slot.
- Tools are re-run every 30 to 60 days, and earlier when the release feed sees a major version.
- Paid placement never changes a rank. A sponsored slot, if one is sold, sits outside the top 3 and is labeled. Read the disclosure.
- A vendor can request a re-test. A vendor cannot buy a score.
- Before any automated run, a person checks that the vendor’s terms allow the test and the publication of the result. Tools that forbid it are skipped or tested with a person in the loop.
What is not live yet
The reference app, the harness, and the raw logs are not in this site yet. The launch pages show the queue and the method so the scores have a place to land. Dropping a real score into data/scores.json is what fills a top-3 slot. Until that file says scored, the site will not invent a number.
Out of scope for this version: accounts, reviews, the other eight builder categories, paid listings, and the X firehose.
Questions
Has ShipRanked published scores?
No. As of October 15, 2026 every tool in the launch set is testing in progress. Lists are alphabetical. That order is not a rank, and the top 3 slots are empty.
How will a score be calculated?
A score is 0 to 100. Task success is 40%, output quality is 20%, and speed, cost, effort, and consistency are 10% each. Each tool runs the task three times. The published number is the median.
Does an affiliate link change a rank?
No. Ranks ignore commission. An affiliate link is labeled and uses rel=sponsored. If featured slots are sold later, they sit outside the top 3.
Which plan is tested?
The cheapest paid plan, or free when that is all the tool sells. The plan is named on each tool page before the run, and again next to the score after it.
When does a score go stale?
A score leaves the top 3 after 60 days, and earlier when the release feed sees a major new version. The old number stays on the tool page and is marked stale until a re-test.
Can a vendor buy a better score?
No. A vendor can ask for a re-test. They cannot buy the outcome. A paid priority test, if it is offered later, still publishes the result with no vendor control over the number.