Coding agents now decide which SDK to use and which tools to call, though something that works smoothly for a human developer can still leave an agent stumbling.
Microsoft built Scope to find those failures. It runs the same task through different agents and scores what each one produces. The code for it is provided under the MIT license.
They are calling it the field Developer Agent Experience, or DevAx, though its 'scope' is to evaluate existing coding agents.
How does it work?
You give Scope a task, a starting codebase if needed, and an agent setup. It then drives existing agents such as GitHub Copilot and Claude Code through that task.
Success is defined through criteria that form a dependency graph. One check can gate another. A simple case is asking an agent for a Node.js server and checking for a GET /health route that returns JSON.
An LLM-based Judge service scores each run against those criteria. Keep in mind that it can make mistakes, and Microsoft says important conclusions still warrant human review.
Each run keeps live logs, generated files, workspace snapshots, and tool calls. A failed criterion can be traced back through those steps. Versioned profiles save the agent, model, skills, MCP servers, and extensions.
You can also swap one, rerun the same task, and compare. Results vary from attempt to attempt, and repeating a task helps separate a recurring problem from a one-off issue.
Then there are the configurable personas that let you set the viewpoint and tone of the simulated user. You can review results in the web portal or use the CLI and REST API to feed them into scripts and pipelines.
Where would you use it?
Docs teams can watch for an instruction that agents miss run after run. Fix the page, then run the scenario again.
SDK maintainers can check whether an agent picks their SDK in the first place. They can also test whether new sample code or a skill changes that.
For CLIs and MCP servers, the traces show where an agent stalled. That can point to a vague tool description or an error message that leaves the agent guessing.
What's the catch?
You can't just sign up and start testing. Scope has no hosted version, so you run it yourself.
A local setup requires these components: Docker with Compose v2, Node 22, pnpm, mkcert, and the GitHub CLI. Microsoft has a local development guide for guiding you through that.
The default worker also calls for an active GitHub Copilot entitlement, and the Judge needs credentials for a model provider. Teams that want a shared deployment can deploy it on Kubernetes with the Helm chart.
Scope's code is on GitHub. Though if you were looking to contribute, you will most likely need to agree to the Microsoft Contributor License Agreement.
Suggested Read 📖: You can use AI to get rid of ads on your smart TV.
from It's FOSS https://ift.tt/g9uQGXc
via IFTTT
Tidak ada komentar:
Posting Komentar