Review an agent skill before you enable it

Write down the exact combination under review: skill name and revision, recurring task, model and version, agent host and version, repository state, and permissions. A result from another model or a broad label such as “code review” is not the same decision.

Microsoft’s July 14, 2026 Visual Studio announcement says its built-in .NET and Azure skills are off by default while Microsoft evaluates effectiveness and cost. The company also links to a public evaluation dashboard that compares work with and without skill plugins. That is useful project evidence, but it does not replace a test against your repository, acceptance criteria, permissions, and review burden.

The approval options are deliberately narrow:

Approve

enable the named revision for the recorded scope.

Limit

allow it only for specified users, repositories, tasks, or manual invocation while evidence develops.

Hold

do not load it until an open security, quality, or measurement question is resolved.

Disable

remove an active skill because a recorded stop condition has occurred.

The Agent Skills specification defines a skill as a directory with a required SKILL.md file and optional scripts, references, assets, and other files. Review the whole directory, not only the visible instructions. Record the maintainer and revision, then inspect any script, command, network request, secret access, permission request, encoded content, or persistence behavior.

A static scan can help find known patterns, but a clean scan is one security signal. It does not show that the skill will improve your task or stay inside the intended access boundary at runtime. The SkillsGuard security guide explains the deeper pre-load review, including scripts, network and secret reach, encoded content, persistence, suppressions, and re-scan dates. Use the AI workflow security review worksheet when the surrounding workflow also needs a full data, credential, approval, retention, and rollback review.

Hold the activation if the package contains an unexplained file, unread encoded content, broader permissions than the task requires, or a finding no reviewer owns. Testing usefulness does not clear a security finding.

Choose a task the skill is expected to improve and define success before running it. Use the same prompt, input files, repository commit and working state, model, agent host, permissions, tool availability, limits, and acceptance checks for both runs. Start each run with a fresh context so earlier work does not influence the comparison.

The Agent Skills project’s evaluation guidance recommends a with-skill and without-skill baseline, realistic prompts, clean contexts, observable assertions, human review, and timing data. If you are reviewing a new revision of an existing skill, compare the new revision with the approved revision as well as the no-skill baseline when that distinction affects the decision.

Run enough repetitions to expose inconsistent behavior when the task has meaningful variability. Do not choose a sample size only because one run produced the preferred result. For a broader model or workflow evaluation, use the AI model bake-off guide to check whether the harness, tool schema, budgets, fixtures, and adapter assumptions treat both conditions fairly.

Review the produced files, tests, citations, commands, and tool traces. Record whether each run completed the task correctly, introduced regressions, stayed within the requested scope, and needed reviewer edits. A polished explanation is not evidence that the code, configuration, or artifact works.

Then compare the operating evidence your host exposes: input and output tokens, total tokens, tool calls, elapsed time, retries, timeouts, and metered cost. Keep unavailable fields blank rather than estimating them. More tool calls or tokens may be justified by a material quality gain, but the reviewer should be able to name the gain.

The public .NET Skills evaluation dashboard tracks quality with and without skills alongside token usage, model, timeouts, and overfitting signals. In maintainer issue #886, the project reports different outcomes across model families, added tool-call overhead, and rubric overfitting risk for analyzing-dotnet-performance. Treat those results as evidence about that project’s tests, not as a benchmark for your skill or workload.

Use the Model Facts Register when the decision depends on provider, model version, region, quota, pricing, retention, SDK behavior, or a fallback path that can change after approval.

An approval should name the users, repositories, tasks, environments, and invocation mode covered by the evidence. Specify whether the skill may load automatically or only when a developer selects it. If it can call another agent or service, use the Agent Access Matrix to name allowed callers, scopes, operations, credentials, logs, limits, and the person who can close the path.

Assign an owner and an expiry or retest trigger. Common triggers include a skill revision, maintainer change, model or host update, permission change, repository policy change, new bundled script, quality regression, unexpected tool use, or cost above the recorded ceiling.

Write the deactivation condition as an observable result. Examples of condition shape include “disable if required tests regress,” “return to manual invocation after an unexplained network call,” or “retest before using a new model version.” These are examples of wording, not recommended thresholds; the team must set the actual condition for its task.