repo·evals
· 2026-06-27 ·main@e9d45a19+credibility-pass

repo-evals

zinan92/repo-evals

🛠

80 / 100Available

🛠
🗺
01Research调研02Plan & design计划与设计03Code & review开发与评审04Package打包发布05Maintain维护
📍
⚛
→
⚗
→
🧬

🛑
0–29
⚠️
30–49
🛠
50–79
🏭
80–100
▼
80
🛠· 80 / 100
  • ✓10 claims passed, no critical failures
  • ✓MIT / Apache / etc., installable per deployment.install_methods
  • ◐release_pipeline=1, recently_active=True
  • ⚪EN-only or ZH-only README
  • ◐evidence_completeness=portable

#1👤
#2🎯
#3🧭
#4⇄

new-repo-eval.shnew-repo-eval.sh(scaffold dirs)(生成目录)Author claim-map.yaml手写 claim-map.yaml(human, 30-60 min)(人工 30-60 分钟)Static checks静态检查(verify each claim)(逐条 claim 验证)verdict_calculator.pyverdict_calculator.py(0-100 + category)(0-100 + 类别)render_verdict_html.pyrender_verdict_html.py(bilingual dossier)(双语 dossier)build_master_dashboard.pybuild_master_dashboard.py(corpus index)(corpus 总目录)

git clone + python3 -m pip install pyyamlany (Python 3.11+)easy
  • 📡
GitHub (target repos to evaluate)
Source of repos being evaluated; needed only for cloning the target
Public repos only — no auth required for the eval flow itself
Python ecosystem (PyYAML)
YAML parsing for repo.yaml + claim-map.yaml
pip install pyyaml — single dependency
· 10
9 1
+40
+30
+10
0
0
0

9 / 10
passed claim-001

passed claim-002

passed claim-003

passed claim-004

passed claim-005

passed claim-006

passed claim-007

passed claim-008

passed claim-009

untested claim-010

input_contract
output_contract
determinism
idempotence
no_skill_callouts
failure_mode_clarity

workflow_correctness
declared_call_graph
stop_conditions
handoff_points
atom_evidence
error_propagation
partial_failure_handling

  • core user-facing layer untested → capped at 'usable'
  • evidence_completeness='portable' → capped at 'reusable'

archetype: pure-cli→core_layer_tested? False→evidence: portable→recommended: usable→final: usable
ceiling 1 · core user-facing layer untested → capped at 'usable'
ceiling 2 · evidence_completeness='portable' → capped at 'reusable'
claim-0010-100 score is auditable — every point traces to a named bucketcriticalscore-auditability● passed
claim-002Dossier is bilingual EN/ZH with runtime togglecriticali18n● passed
claim-0034-category collapse (Production / Available / Risky / Don't use) is tested at every boundarycriticalcategory-correctness● passed
claim-004Workflow diagram renders all 3 layouts (io / linear / tree)highvisualization● passed
claim-005Similar-repos comparison computes peer scores live, not from stored sidecarshighcomparison-freshness● passed
claim-006A 30-repo corpus already exists for cross-comparisonhighcorpus-size● passed
claim-007Test suite covers the new code paths and passeshightest-discipline● passed
claim-008Repository has a LICENSE filecriticallicensing● passed
claim-009README reflects the current scoring + category modelhighdocs-currency● passed
claim-010End-to-end: a new evaluator can clone, scaffold, fill, rendercriticalend-to-end○ untested

100%
0.00s
0

run-live-e2e

2026-06-27
100% — tokens in ? / out ?
  • claim-010 · passed

run-static-checks

2026-05-05
90% — tokens in ? / out ?
  • claim-001 · passed
  • claim-002 · passed
  • claim-003 · passed
  • claim-004 · passed
  • claim-005 · passed
  • claim-006 · passed
  • claim-007 · passed
  • claim-008 · passed
  • claim-009 · passed

run-live-e2e

2026-06-27
100% — tokens in ? / out ?
  • claim-010 · passed

run-static-checks

2026-05-05
90% — tokens in ? / out ?
  • claim-001 · passed
  • claim-002 · passed
  • claim-003 · passed
  • claim-004 · passed
  • claim-005 · passed
  • claim-006 · passed
  • claim-007 · passed
  • claim-008 · passed
  • claim-009 · passed
# zinan92/repo-evals — final verdict (2026-06-27)

## Repo

- **Name:** zinan92/repo-evals · **Stars:** 0 (private/personal)
- **Archetype:** pure-cli · **Layer:** **molecule**
- **License:** MIT LICENSE file present
- **Pushed baseline:** `e9d45a19`; this pass validates the current credibility fixes

## What was evaluated

| Claim | Status | Notes |
|---|---|---|
| 001 score is auditable | passed | 6 named breakdown buckets, math tested |
| 002 bilingual EN/ZH | passed | Rendered dossiers carry EN/ZH fields and runtime language toggle |
| 003 4-category mapping | passed | Boundary tests cover Production / Available / Risky / Don't use |
| 004 3-layout workflow diagrams | passed | io / linear / tree branches exist and render |
| 005 similar-repos live scores | passed | Peer cards compute current verdict data at render time |
| 006 corpus exists for cross-comparison | passed | The corpus is past cold start and dashboard-indexed |
| 007 tests pass | passed | `153 passed` on 2026-06-27 |
| 008 LICENSE | passed | MIT LICENSE exists at repo root |
| 009 README is current | passed | README describes the current 0-100 + 4-category model |
| 010 live e2e onboarding | passed | Fresh-clone style scaffold → fill → render → dashboard smoke test logged |

## Real findings

1. **The P0 test failures were real and fixed.** Four archetype starter
   claim maps had unescaped Chinese double quotes that made PyYAML fail.
   The generic claim template also drifted from its scaffold test contract.
   After the fix, the focused archetype suite passes and the full suite is
   green.

2. **The self-eval evidence chain is now explicit.** Claims 001-009 now state
   `evidence_needed`, so the coverage gap detector no longer has to infer what
   would prove each claim. Claim 010 has a dated live e2e run with preserved
   command log and rendered HTML artifact.

3. **The test output is now clean.** The prior pytest warnings came from
   unregistered custom marks (`unit`, `integration`) in `tests/test_layers.py`.
   `pytest.ini` now registers both marks.

## Why the verdict improved

- All 10 self-eval claims are now passed.
- The full test suite is green: `153 passed`.
- License and README drift were already fixed; the stale self-eval wording has
  been replaced by this fresh verdict.
- Live e2e onboarding evidence now exists under
  `runs/2026-06-27/run-live-e2e/`.

The honest status is **Available and close to team-ready**: the framework is
usable and internally auditable, but still needs a public packaging pass
before broad distribution as an installable skill.

## Path to higher score

1. Add public-skill packaging: demo GIF/video, `.claude-plugin/marketplace.json`,
   one-line `npx skills add` path, and skills.sh badge if this will be
   distributed as a public skill.
2. Run the same live e2e on a genuinely unseen third-party repo, not only a
   smoke fixture.

## Recommended

```yaml
status: evaluated
category: available
next_step: register_pytest_marks_then_package_public_skill
```