Shaduf.Research preview
Marketing Skills Catalog/Original context eval definitions: expectation versus observation

Research report · 10 November 2026

Context eval fixtures state expectations, not results

Three original evals/evals.json files contain seven author-written cases each. They describe what an assistant should do. They do not show what an assistant did.

What the files contain

Each file has a skill name and seven cases with an input prompt, expected output, natural-language assertions and an empty files array. The empty arrays supply no context-document contents or path-specific permission policy. None of the files contains an assistant response, read/write trace, evaluator implementation, pass/fail result or result date.

Original fileContext expectationMissing observation
Copywriting evalsOne homepage-copy case expects a check for product-marketing.md.No authorized path list, actual read, Learnings check or result.
emails evalsOne welcome-sequence case expects the same file check.No authorized path list, actual read, Learnings check or result.
Product Marketing evalsCases expect existence checks, creation or scoped updates, version and Changelog handling. One auto-draft case expects a draft shown for review before saving.No exact-file approval before a scan, stale-learning check, review before downstream reuse, trace or result.

Where the evidence stops

“Check the file” is an assertion in a test definition. It does not show which file a host permitted the assistant to read, whether the read happened, or whether the content was current. Product Marketing's review-before-save assertion concerns the first draft of a shared file; it is not a check on a later Copywriting or email task that reuses that file. The prior selected-file review describes source instructions for optional project scans and shared context reads. Source instructions and eval assertions together still do not constitute an observed host run.

All three URLs point to mutable main files. This review did not establish an immutable eval-file commit or content hash. These definitions cannot establish package inclusion, host permission enforcement, output quality, marketing results or whether tests were run elsewhere. No eval, package or live workflow was run for this review.

What would establish behavior

A future, separately authorized test would need a fixed package and host route, an explicit file permission policy, a controlled dated context document with Learnings, captured reads and writes, evaluation logic, outputs and recorded results. Until then, pass an approved brief for one local draft, approve exact files and sections before a broader read, review saved context before reuse, and separately approve publication or sending.

Research library · Choose a route

Search published pools, pages, reports, and evidence.