The Same Legal Skill Narrowed the Gap With Claude, But Did Not Close It
Our first ScaffoldBench showed the same model performs differently across scaffolds. The market asked a sharper question: was Claude’s lead really about the scaffold, or just its domain Legal Skill? So we gave MikeOSS the same Privacy Skills and re-ran the tasks to isolate one variable.
In our first ScaffoldBench, we tested a simple thesis: the same model can perform differently depending on the scaffold around it. We took one LLM, Opus 4.8, and ran it across three legal AI environments (Claude Chat, Cowork with Legal Plugin, and MikeOSS) on a 40-task legal eval focused on data protection and operational resilience.
The result confirmed the thesis: the same model produced different results across different scaffolds. But after publishing, we got useful feedback from the market.
Was Cowork ahead because of the full scaffold architecture, or just because it had a domain-specific Legal Skill?
Why we ran a second ScaffoldBench
Some people pointed out that Cowork with Legal Plugin may have outperformed MikeOSS not because of the full scaffold architecture, but because it carried a domain-specific Legal Skill. That was a fair point, and a testable one. So we decided to run a follow-up experiment.
The first benchmark established that the scaffold around a model materially affects performance on legal tasks. The same Opus 4.8 scored differently in plain chat, in Cowork, and in MikeOSS.
Read the first ScaffoldBench→Which part of the scaffold matters most?
Part 1 showed that scaffold matters. The next question was narrower: which part of it? To answer that, we decomposed the scaffolds into their components, then focused this benchmark on just one.
For this follow-up, we isolated the component the market had flagged: Skills, specifically the Privacy Skills that ship inside Claude’s Legal Plugin.
The experiment
We held everything constant except one variable. Same two scaffolds, same model, same evaluation dataset, then we added the same Privacy Skills to MikeOSS. The goal was not to prove which scaffold is universally better; it was to measure how much MikeOSS improves when it carries the same Skills as Claude’s Legal Plugin.
Adding Skills narrowed the gap, but did not close it
Privacy Skills improved MikeOSS by +3.0 points overall. But Claude Cowork still led by around 5 points. The Skill mattered. It was not the whole story.
Skills improved MikeOSS by +3.0 points. Claude Cowork stayed ahead by about 5.
The Skill helped most on reasoning
The most important signal was not the overall score. It was where the improvement happened. Privacy Skills had by far the strongest impact on how the model reasoned through privacy-specific legal tasks.
This suggests the Privacy Skills did not mainly improve document finding or evidence extraction. They improved the way the model reasoned through privacy tasks. A good legal Skill is not just a folder of legal knowledge. It can shape how the model approaches the task, what risks it looks for, and how it structures the analysis.
Skills work best when they match the task
On tasks with high Skill-relevance, MikeOSS with Privacy Skills came much closer to Claude Cowork, a near-dead heat. This is probably the strongest result from the follow-up.
Domain-specific Skills can make a substantial impact when they are properly matched to the legal task. This supports the point OpenAI recently made about evaluations. Domain-specific scaffolding can help pull more capability out of the model.
Same improvement, without losing the cost edge
Adding Privacy Skills raised MikeOSS cost by about 15%, but it stayed roughly 57% cheaper than Claude Cowork. In real legal workflows, cost per task is an evaluation criterion too.
If a system can improve performance while keeping a material cost advantage, that is an important design signal. Quality is not the only criterion that matters.
Why the cost difference remained
To understand the gap, we looked deeper into the scaffold architecture. Cowork has more built-in capability; MikeOSS is more lightweight. Both facts show up across the same components.
More scaffold capability does not automatically mean better performance for every task. Cowork can run extra checks, use more tools, and produce longer reasoning, useful for complex or ambiguous work, but it also raises token spend. For these tasks, some of that spend did not materially improve the final result. Sometimes the best scaffold is not the most complex one. It is the one with the right level of capability for the task.
What we learned
We expected that adding the same Privacy Skills to MikeOSS might almost fully close the gap with Claude Cowork. It did not, but it narrowed it in a meaningful way. The strongest improvement was in reasoning, and the best result appeared where the Skills were directly relevant to the tasks. At the same time, MikeOSS stayed materially more cost-efficient.
Domain-specific Skills are a major performance driver inside legal AI scaffolds, especially for the tasks they are designed for. But Skills alone do not explain the entire gap.
The broader scaffold architecture still matters, especially for evidence handling, verification, and consistency.
For legal AI teams, this reframes the question. Not only which model should we use and not even only which scaffold, but:
Which scaffold components actually improve performance for this specific legal workflow, and at what cost?
The future of legal AI will not be only about stronger models. It will also be about better legal Skills, better scaffold architecture, and better evaluation infrastructure to understand what actually works. That is the direction we want to keep exploring with ScaffoldBench.
Legal AI needs benchmarks that measure Skills, scaffolds, and cost, not just models.
If you are building legal AI evals, legal-agent workflows, or applied AI systems for law firms and legal teams, we would be happy to compare notes.
Book a Discovery Call