content-toolkit

# Final Verdict

## Repo

- Name: zinan92/content-toolkit
- Date: 2026-04-13
- Archetype: orchestrator
- Final bucket: **usable**
- Confidence: medium

## Verdict Rationale

### Baseline: usable

Per verdict calculator rules:
- Critical claims **claim-001 through claim-006** all PASSED (routing, help, auto-install, smart hints)
- But critical downstream coverage is partial — test suite is 100% broken (claim-014),
  subtitle silently fails (claim-009), intelligence capability degraded (claim-015)
- Error propagation is inconsistent (claim-012: partial)

### Ceiling applied: none

The orchestrator archetype has no default ceiling. However, the broken test suite
and silent failures effectively self-cap at `usable` — you can't recommend something
where automated verification is completely absent and some capabilities fail silently.

## Evaluation Dimensions (Orchestrator-Specific)

| Dimension | Rating | Notes |
|-----------|--------|-------|
| **Routing correctness** | ★★★★☆ | Excellent. All tested routes work. Aliases normalize correctly. Smart input detection is a nice touch. |
| **Error propagation** | ★★☆☆☆ | Inconsistent. `download` passes through errors, but `videocut subtitle` exits 0 with empty output. |
| **Partial failure handling** | ★★★☆☆ | Not tested deeply, but auto-install→degraded is handled well (health reports it). |
| **End-to-end happy path** | ★★★☆☆ | transcribe and autocut work. subtitle fails silently. Pipeline untested in this run but TEST-REPORT.md says it passes. |
| **Per-area coverage** | ★★☆☆☆ | 7 capabilities claimed, only download+videocut(2/7 subs) verified working. Intelligence degraded. publish/xiaohongshu untested (require auth/external services). |
| **Observability** | ★★★★☆ | `content health` is genuinely useful — shows per-capability status, git ref, known issues. |

## Score Summary

| Category | Passed | Failed | Partial | Total |
|----------|--------|--------|---------|-------|
| Critical | 6 | 0 | 0 | 6 |
| High | 4 | 2 | 1 | 7 |
| Medium | 2 | 1 | 0 | 3 |
| **Total** | **12** | **3** | **1** | **16** |

## What I Would Say In Plain English

**content-toolkit's orchestration layer is well-designed — the routing, help, aliases,
and smart input detection are genuinely good.** If you already know which commands work,
it's a useful tool.

**But it's not reliable enough to recommend.** The test suite is 100% broken (all 80+ tests
fail on import), some capabilities silently fail (subtitle exits 0 with empty output),
and the intelligence capability auto-installs into a degraded state. The repo's own
TEST-REPORT.md honestly documents a 7/20 pass rate from March 31 — and nothing has
been fixed since.

**The gap is not in design but in execution quality.** The architecture is sound, the
skill system is thoughtful, and the health reporting is better than most. What's missing
is: fix the test suite, fix silent failures, fix the 13 known issues in TEST-REPORT.md.

## Path to `reusable`

1. **Fix test suite** — export functions from cli.js, prevent help side effect on import.
   Currently zero automated verification of routing logic.
2. **Fix silent failures** — videocut subtitle (and likely clip, cover, speed per TEST-REPORT.md)
   must either produce output or surface a clear error. Exit 0 + empty dir is unacceptable.
3. **Fix intelligence capability** — pyproject.toml module path so auto-install produces
   a working capability, not degraded.
4. **Address TEST-REPORT.md backlog** — at least the 4 MEDIUM bugs (BUG-3/4/5/6) and
   2 HIGH UX issues (UX-1/2).

## Path to `recommendable`

Everything in `reusable` plus:
5. **Per-area claim maps** — each downstream capability (download, extract, rewrite,
   videocut, publish, xiaohongshu) gets its own eval under `areas/<slug>/`
6. **End-to-end workflow verification** — the douyin-to-xhs and pipeline presets
   tested with real content
7. **Consistent error propagation** — every downstream failure surfaces at the
   orchestrator boundary
8. **CI integration** — test suite runs on push, catches regressions

## Remaining Risks

- **Silent failure pattern may be systemic.** We only tested 3 of 7 videocut subcommands
  and 1 already silently fails. TEST-REPORT.md documents similar issues in clip, cover, speed.
- **No CI.** Regressions accumulate silently. The test suite broke and nobody noticed.
- **External capabilities (publish, xiaohongshu) are untested** — they require auth
  and external services, making them hard to evaluate without credentials.
- **intelligence capability has a packaging bug** in the upstream repo, but content-toolkit
  claims it as a capability. Users will encounter a broken experience.