4 - Perform evaluation, error analysis, and tuning (GH-600)

What is the primary benefit of defining agent tasks in terms of inputs, outputs, and success criteria?. It focuses solely on the speed of the agent's execution. It ensures that outcomes are predictable. It allows the agent to modify its own workflow constraints. It eliminates the need for human review of pull requests.

Which of the following is categorized as an 'Input' for an agent task?. The resolution of a security signal. A structured plan in the pull request description. Issue context and boundary constraints. Evidence links to workflow runs.

In the context of agent outputs, what is the purpose of providing 'evidence'?. To replace the need for a bounded changeset. To list the constraints that the agent ignored during the task. To define the repository scope for the next agent task. To prove that the task was executed and verified, such as via workflow run links.

What is a common risk when agent tasks have 'vague or missing' success criteria?. The agent will expand the repository scope to include infrastructure files. The agent may produce changes that look plausible but do not solve the underlying problem. The CI system will automatically generate the missing criteria. The agent will refuse to start the task.

Why is 'CI passed' described as 'necessary, but not always sufficient' as a success criterion?. Because agents should be trusted to verify their own work without external checks. Because it might not reflect the real intent, such as whether a specific vulnerability was actually resolved. Because CI checks are often slower than the agent's reasoning process. Because CI systems cannot run on Ubuntu-latest environments.

According to the implementation example, how is success 'enforced by the system' rather than 'assumed by the agent'?. By requiring specific status checks (like security analysis) to pass before a merge is allowed. By removing the 'test' step from the YAML workflow to speed up validation. By allowing the agent to write its own security analysis rules. By using GitHub Actions to automatically merge every pull request.

In a vulnerability remediation task, what is an example of a 'Constraint' provided as an input?. A link to the security alert. A structured plan in the pull request. No direct-to-main pushes allowed. The successful replacement of a vulnerable version.

If an agent updates a direct dependency but leaves the vulnerable version reachable through a transitive dependency, it has met a high-quality success criterion. True. False.

Which component of the agent's output is intended to provide a 'bounded changeset'?. Commits on an agent branch. The review outcomes from a security scan. Review outcomes are used to evaluate the work (success criteria) rather. The 'runs-on: ubuntu-latest' command. The security alert link.