Skip to content

Agents Package Cheatsheet

Root namespace: Cognesy\Agents

This file is a quick, code-aligned map of the package surface. For narrative guidance and examples, use packages/agents/docs/*.md.

1. Core Loop

  • AgentLoop (readonly)
  • main orchestrator, implements CanControlAgentLoop and CanAcceptEventHandler
  • key API: default(), execute(), iterate()
  • accessors: tools(), toolExecutor(), driver(), eventHandler(), interceptor()
  • composition API: withTool(), withTools(), withDriver(), withToolExecutor(), withInterceptor(), withEventHandler(), with()
  • event API: wiretap(), onEvent()
  • note: terminal executions are auto-reset on entry to execute() / iterate()
  • CanControlAgentLoop
  • contract: execute(AgentState): AgentState, iterate(AgentState): iterable

2. State Model

  • Data\AgentState
  • immutable runtime state
  • factories: empty(), fromArray()
  • identity: agentId(), parentAgentId(), llmConfig(), executionCount(), createdAt(), updatedAt()
  • common mutators: withUserMessage(string|\Stringable|Message), withSystemPrompt(string|\Stringable), withMetadata(), withMessages(), withMessageStore(), withLLMConfig(), with()
  • step mutators: withCurrentStep(), withCurrentStepCompleted(), withExecutionCompleted(), withExecutionContinued()
  • stop/failure: withStopSignal(), withFailure(), withExecutionStatus()
  • context access: context(), store(), messages(), metadata()
  • result access: finalResponse(), currentResponse(), hasFinalResponse()
  • execution access: execution(), status(), stepCount(), steps(), usage(), errors(), hasErrors()
  • last-step accessors: lastStep(), lastStepExecution(), lastStepToolExecutions(), lastToolExecution(), lastStepErrors(), lastStepType(), lastStepUsage(), lastStepDuration()
  • stop accessors: stopSignal(), stopReason(), stopSource()
  • control: shouldStop(), forNextExecution()
  • serialization: debug(), toArray(), fromArray()
  • Data\ExecutionState
  • per-execution transient state (executionId, status, steps, continuation)
  • Data\AgentStep
  • one loop step snapshot (inputMessages, outputMessages, inferenceResponse, toolExecutions, errors)
  • Data\StepExecution
  • completed step wrapper with timing
  • Data\ToolExecution
  • one executed tool call (value(), hasError(), errorAsString(), wasBlocked())
  • Data\ExecutionBudget
  • optional limits: maxSteps, maxTokens, maxSeconds, maxCost, deadline
  • factories: unlimited()
  • queries: isEmpty(), isExhausted()
  • Data\AgentId, Data\ExecutionId, Data\AgentStepId, Data\ToolExecutionId
  • typed ID value objects

3. Enums

  • Enums\ExecutionStatus -- Pending, InProgress, Completed, Stopped, Failed
  • Enums\AgentStepType -- ToolExecution, FinalResponse, Error

4. Collections

  • Collections\Tools
  • immutable named tool collection
  • key API: has(), get(), names(), all(), count(), isEmpty(), descriptions(), withTool(), withTools(), withToolRemoved(), merge(), toToolSchema(): ToolDefinitions
  • Collections\AgentSteps
  • Collections\StepExecutions
  • Collections\ToolExecutions
  • Collections\NameList

5. Tools

Contracts

  • Tool\Contracts\ToolInterface
  • use(mixed ...$args): Result
  • toToolSchema(): ToolDefinition
  • descriptor(): CanDescribeTool
  • Tool\Contracts\CanDescribeTool
  • name(), description(), metadata(), instructions()
  • Tool\Contracts\CanExecuteToolCalls
  • executeTools(ToolCalls, AgentState): ToolExecutions
  • Tool\Contracts\CanAccessAgentState
  • Tool\Contracts\CanAccessToolCall
  • Tool\Contracts\CanManageTools

Base classes

  • Tool\Tools\SimpleTool
  • Tool\Tools\ReflectiveSchemaTool
  • Tool\Tools\FunctionTool
  • Tool\Tools\StateAwareTool
  • Tool\Tools\BaseTool
  • Tool\Tools\ContextAwareTool
  • Tool\Tools\FakeTool

Runtime

  • Tool\ToolExecutor
  • Tool\ToolRegistry
  • Tool\ToolDescriptor

6. Drivers

  • Drivers\CanUseTools
  • Drivers\CanAcceptToolRuntime
  • Drivers\ToolCalling\ToolCallingDriver (default)
  • Drivers\ToolCalling\ToolExecutionFormatter
  • Drivers\ReAct\ReActDriver
  • Drivers\Testing\FakeAgentDriver
  • Drivers\Testing\ScenarioStep

7. Context

  • Context\AgentContext
  • Context\CanCompileMessages
  • Context\CanAcceptMessageCompiler
  • Context\ContextSections

Compilers:

  • Context\Compilers\ConversationWithCurrentToolTrace (default)
  • Context\Compilers\AllSections
  • Context\Compilers\SelectedSections

8. Continuation / Stop

  • Continuation\StopReason
  • Continuation\StopSignal
  • Continuation\StopSignals
  • Continuation\ExecutionContinuation
  • Continuation\AgentStopException

9. Hooks / Interception

  • Hook\Contracts\HookInterface
  • Hook\Data\HookContext
  • Hook\Data\RegisteredHook
  • Hook\Collections\RegisteredHooks
  • Hook\Enums\HookTrigger
  • values: BeforeExecution, BeforeStep, BeforeToolUse, AfterToolUse, AfterStep, OnStop, AfterExecution, OnError
  • Hook\Collections\HookTriggers
  • Hook\HookStack

Built-in hooks:

  • Hook\Hooks\CallableHook
  • Hook\Hooks\StepsLimitHook
  • Hook\Hooks\TokenUsageLimitHook
  • Hook\Hooks\ExecutionTimeLimitHook
  • Hook\Hooks\FinishReasonHook
  • Hook\Hooks\ApplyContextConfigHook

Interception:

  • Interception\CanInterceptAgentLifecycle
  • Interception\PassThroughInterceptor

10. Builder / Capabilities

  • Builder\AgentBuilder
  • Builder\AgentConfigurator
  • Builder\Contracts\CanProvideAgentCapability
  • Builder\Contracts\CanConfigureAgent
  • Builder\Contracts\CanComposeAgentLoop
  • Builder\Contracts\CanProvideDeferredTools
  • Builder\Collections\DeferredToolProviders
  • Builder\Data\DeferredToolContext

Capability registry:

  • Capability\AgentCapabilityRegistry
  • Capability\CanManageAgentCapabilities

Core capabilities:

  • Capability\Core\UseLLMConfig
  • Capability\Core\UseGuards
  • Capability\Core\UseTools
  • Capability\Core\UseToolFactory
  • Capability\Core\UseHook
  • Capability\Core\UseDriver
  • Capability\Core\UseDriverDecorator
  • Capability\Core\UseContextCompiler
  • Capability\Core\UseContextCompilerDecorator
  • Capability\Core\UseContextConfig
  • Capability\Core\UseReActConfig

Domain capabilities:

  • Capability\Bash\UseBash
  • Capability\Cancellation\UseCooperativeCancellation
  • adds checkpoint-based cooperative cancellation to the loop
  • cancellation is cooperative: stops at BeforeExecution / BeforeStep checkpoints only — does not interrupt in-flight HTTP or tool calls
  • requires a CanProvideCancellationSignal implementation; built-in: InMemoryCancellationSource
  • stop reason reported as StopReason::UserRequested

use Cognesy\Agents\Capability\Cancellation\UseCooperativeCancellation;
use Cognesy\Agents\Capability\Cancellation\InMemoryCancellationSource;

$source = new InMemoryCancellationSource();

$agent = AgentBuilder::base()
    ->withCapability(new UseCooperativeCancellation($source))
    ->build();

// cancel from outside (e.g. signal handler, HTTP request, timer):
$source->cancel('user pressed stop');

// custom source (Redis key, DB flag, HTTP endpoint, …):
$agent = AgentBuilder::base()
    ->withCapability(new UseCooperativeCancellation(
        new class implements CanProvideCancellationSignal {
            public function cancellationSignal(AgentState $state): ?StopSignal {
                return redis_get("cancel:{$state->agentId()}")
                    ? StopSignal::userRequested('cancelled via redis')
                    : null;
            }
        }
    ))
    ->build();
- Capability\File\UseFileTools - installs: read_file, write_file, edit_file - standalone file tools also available: SearchFilesTool, ListDirTool - Capability\Metadata\UseMetadataTools - Capability\Subagent\UseSubagents - Capability\PlanningSubagent\UsePlanningSubagent - Capability\StructuredOutput\UseStructuredOutputs - Capability\Summarization\UseSummarization - Capability\SelfCritique\UseSelfCritique - Capability\Skills\UseSkills - Capability\Tasks\UseTaskPlanning - Capability\Tools\UseToolRegistry - Capability\ExecutionHistory\UseExecutionHistory - Capability\Retrospective\UseExecutionRetrospective - Capability\Broadcasting\UseAgentBroadcasting

11. Broadcasting

  • Broadcasting\AgentEventBroadcaster
  • Broadcasting\AgentBroadcastObserver
  • Broadcasting\BroadcastConfig
  • Broadcasting\CanBroadcastAgentEvents

12. Templates

  • Template\Data\AgentDefinition
  • core fields: name, description, systemPrompt, label, llmConfig, capabilities, tools, toolsDeny, skills, budget, metadata
  • tool semantics: tools === null means inherit all available tools
  • Template\AgentDefinitionLoader
  • Template\AgentDefinitionRegistry
  • Template\Contracts\CanManageAgentDefinitions
  • Template\Contracts\CanInstantiateAgentLoop
  • Template\Contracts\CanInstantiateAgentState
  • Template\Parsers\CanParseAgentDefinition
  • Template\Factory\DefinitionStateFactory
  • Template\Factory\DefinitionLoopFactory
  • parsers: Template\Parsers\MarkdownDefinitionParser, JsonDefinitionParser, YamlDefinitionParser

13. Sessions

Core:

  • Session\Data\SessionId
  • Session\Data\AgentSessionInfo
  • Session\Data\AgentSession
  • access: info(), definition(), state(), sessionId(), status(), version()
  • Session\SessionRuntime -- preferred API for creating new sessions and applying actions to persisted sessions
  • Session\SessionRepository -- low-level persistence boundary over a store implementation
  • Session\SessionFactory -- low-level helper for constructing AgentSession instances before manual persistence

Contracts:

  • Session\Contracts\CanManageAgentSessions
  • Session\Contracts\CanExecuteSessionAction
  • Session\Contracts\CanStoreSessions
  • Session\Contracts\CanControlAgentSession

Stores:

  • Session\Store\InMemorySessionStore
  • Session\Store\FileSessionStore

Actions:

  • Session\Actions\SendMessage (accepts string|\Stringable|Message)
  • Session\Actions\ForkSession (returns a new branch session object; persist that fork via repository create(); for brand-new root sessions prefer SessionRuntime::create())
  • Session\Actions\ResumeSession
  • Session\Actions\SuspendSession
  • Session\Actions\ClearSession
  • Session\Actions\ChangeModel
  • Session\Actions\ChangeSystemPrompt (accepts string|\Stringable)
  • Session\Actions\WriteMetadata
  • Session\Actions\UpdateTask

Enums:

  • Session\Enums\SessionStatus -- Active, Suspended, Completed, Failed, Deleted
  • Session\Enums\AgentSessionStage -- AfterLoad, AfterAction, BeforeCreate, AfterCreate, BeforeSave, AfterSave

Session hooks:

  • Session\SessionHookStack
  • Session\RegisteredSessionHook
  • Session\PassThroughSessionController
  • Session\Collections\SessionInfoList

Exceptions:

  • Session\Exceptions\SessionNotFoundException
  • Session\Exceptions\SessionConflictException
  • Session\Exceptions\InvalidSessionFileException

14. Events

Agent events include:

  • AgentExecutionStarted, AgentStepStarted, AgentStepCompleted
  • AgentExecutionStopped, AgentExecutionCompleted, AgentExecutionFailed
  • AgentStateUpdated
  • ContinuationEvaluated, StopSignalReceived, TokenUsageReported
  • ToolCallStarted, ToolCallCompleted, ToolCallBlocked
  • InferenceRequestStarted, InferenceResponseReceived
  • SubagentSpawning, SubagentCompleted
  • HookExecuted, DecisionExtractionFailed, ValidationFailed
  • Events\AgentEvent (base class)

Event support:

  • Events\Support\AgentEventConsoleFormatter
  • Events\Support\AgentEventConsoleObserver

Session events include:

  • SessionLoaded, SessionActionExecuted, SessionSaved
  • SessionLoadFailed, SessionSaveFailed

15. Exceptions

  • Exceptions\AgentException (base)
  • Exceptions\AgentNotFoundException
  • Exceptions\InvalidToolException
  • Exceptions\InvalidToolArgumentsException
  • Exceptions\ToolCallBlockedException
  • Exceptions\ToolExecutionBlockedException
  • Exceptions\ToolExecutionException

16. Skills

  • Capability\Skills\Skill
  • immutable skill value object
  • standard fields: name, description, license, compatibility, metadata, allowedTools, body, path, resources
  • extension fields: disableModelInvocation, userInvocable, argumentHint, model, context, agent
  • key API: render(?string $arguments), renderMetadata(), toArray()
  • argument substitution: $ARGUMENTS, $ARGUMENTS[N], $N placeholders
  • Capability\Skills\SkillLibrary
  • discovers SKILL.md files in <path>/<skill-name>/SKILL.md
  • lazy-loads skill content on first access, caches result
  • key API: listSkills(modelInvocable, userInvocable), hasSkill(), getSkill(), renderSkillList()
  • resource discovery: scans scripts/, references/, assets/, examples/ subdirs
  • Capability\Skills\LoadSkillTool
  • tool exposed to LLM: load_skill(skill_name, list_skills, arguments)
  • user-invocable filtering on list mode
  • Capability\Skills\AppendSkillMetadataHook
  • injects skill names/descriptions as system message before first step
  • filters out disable-model-invocation: true skills
  • Capability\Skills\TrackActiveSkillHook
  • tracks active skill metadata (allowed-tools, model) in state after load_skill completes
  • Capability\Skills\SkillToolFilterHook
  • enforces allowed-tools restrictions; blocks non-allowed tools (except load_skill itself)
  • Capability\Skills\SkillModelOverrideHook
  • overrides LLMConfig when a skill with a model field is active
  • Capability\Skills\SkillForkExecutor
  • executes skills in a forked agent loop context
  • Capability\Skills\SkillPreprocessor
  • executes !command`` patterns in skill body before argument substitution
  • configurable working directory and timeout
  • opt-in: pass to UseSkills or LoadSkillTool constructor
  • Capability\Skills\UseSkills
  • capability that wires LoadSkillTool + hooks into agent
  • optional ?SkillPreprocessor for shell preprocessing
  • follows Agent Skills Open Standard (30+ tools)

17. Testing

  • Drivers\Testing\FakeAgentDriver
  • scripted loop steps via ScenarioStep
  • best for most deterministic agent-loop tests
  • Tests\Support\FakeInferenceDriver
  • queued raw InferenceResponse or streaming PartialInferenceDelta fixtures
  • use when the test sits closer to the inference boundary
  • Tool\Tools\FakeTool
  • deterministic tool double with fixed or callable-backed results
  • Tests\Support\FakeSubagentProvider
  • in-memory subagent definition registry for capability tests
  • Tests\Support\TestAgentLoop
  • small loop harness with explicit max-iteration stop behavior
  • Cognesy\Sandbox\Testing\FakeSandbox (from packages/sandbox, not agents)
  • deterministic process-execution seam for bash-backed tools

18. Docs Index

Read in this order:

  1. packages/agents/docs/01-introduction.md
  2. packages/agents/docs/testing-doubles.md
  3. packages/agents/docs/02-basic-agent.md
  4. packages/agents/docs/05-tools.md
  5. packages/agents/docs/06-building-tools.md
  6. packages/agents/docs/13-agent-builder.md
  7. packages/agents/docs/14-agent-templates.md
  8. packages/agents/docs/15-subagents.md
  9. packages/agents/docs/16-session-runtime.md
  10. packages/agents/docs/19-skills.md
  11. packages/agents/docs/21-evals.md
  12. packages/agents/docs/22-eval-assertions.md
  13. packages/agents/docs/23-eval-judges.md
  14. packages/agents/docs/24-eval-traces-and-artifacts.md
  15. packages/agents/docs/25-running-evals.md

19. Evals

Behavioral evals that grade an agent target with deterministic assertions and semantic judges. Narrative docs: docs/21-evals.md through docs/25-running-evals.md.

Case definition:

  • Evals\AgentEval (readonly)
  • immutable definition of one eval case
  • factories: define(description, Closure(EvalContext): void $test, ?tags, ?judge)
  • key API: withId()
  • accessors: description(), test(), tags(), id(), judge()
  • Evals\AgentEvals (readonly, Countable, IteratorAggregate)
  • immutable collection of AgentEval
  • factories: none()
  • key API: with(), filtered(?glob, ?required, ?excluded)
  • accessors: all(), count()
  • Evals\AgentEvalSet (readonly)
  • groups evals built from a dataset
  • factories: fromDataset(EvalDataset, Closure(EvalDatasetRow): AgentEval $factory), of(AgentEval ...$evals)
  • accessors: evals(): AgentEvals
  • Evals\EvalTags (readonly, Countable, IteratorAggregate)
  • normalized (trimmed, deduped, sorted) tag set
  • factories: of(), none()
  • key API: has()
  • accessors: all(), count()
  • Evals\EvalDataset (readonly, Countable, IteratorAggregate)
  • list of EvalDatasetRow
  • factories: fromJson(), fromYaml()
  • Evals\EvalDatasetRow (readonly)
  • one dataset row
  • key API: value(key), string(key)
  • accessors: toArray()
  • Evals\EvalDiscovery (readonly)
  • finds *.eval.php files under a root and assigns ids
  • factories: in(root)
  • key API: discover(): AgentEvals
  • note: an eval file must return AgentEval|AgentEvalSet|array<AgentEval>; ids are the file's path relative to the root (with a /NNNN suffix appended when one file yields more than one eval)
  • Evals\EvalCount (readonly)
  • count predicate for calledTool()/calledSubagent()/event() assertions
  • factories: atLeast(), atMost(), between(), satisfies(Closure(int): bool)
  • key API: matches(int)
  • Evals\EvalMatch (readonly)
  • value matcher for outputMatches() and ValueExpectation::matches()
  • factories: partial(array), regex(pattern), satisfies(Closure)
  • key API: matches(mixed)
  • Evals\EvalMatcher (readonly)
  • static matching helpers used throughout: matches() (exact equality unless given an EvalMatch or array), partial() (recursive partial-array match; lists require equal length, maps require only the listed keys)

Execution context & assertions:

  • Evals\EvalContext
  • passed into every eval's test closure; owns the session, assertion collector, and log collector for one eval run
  • key API: send(), run(): AgentRun, expect(mixed): ValueExpectation, judge(): AgentJudgeAssertions, check(), require() (throws EvalRequirementFailed on failure), skip() (throws EvalSkipped), log(), newSession()
  • built-in assertions, each returning AssertionHandle: succeeded(), stopped(), messageIncludes(), outputEquals(), outputMatches(), calledTool(), notCalledTool(), toolOrder(), usedNoTools(), maxToolCalls(), stepCount(), maxSteps(), totalTokensAtMost(), noFailedActions(), calledSubagent(), event(), notEvent(), eventOrder(), eventsSatisfy()
  • accessors: assertions(), logs()
  • note: newSession() shares this context's collectors with the new session -- EvalRunner's repeated trials deliberately construct a brand-new EvalContext instead, so trials never share collectors
  • Evals\AssertionCollector
  • records and defer-resolves assertion results for one eval run
  • key API: record(), recordLazy(placeholder, Closure(): AssertionResult $resolve), replace(), at(index), results(): AssertionResults
  • note: recordLazy()'s resolver runs at most once, at first read via at() or results() -- this is the mechanism that makes a judge run at most once
  • Evals\AssertionHandle (readonly)
  • fluent handle returned by every EvalContext/ValueExpectation assertion
  • key API: gate(), soft(), atLeast(threshold), label(), result(): AssertionResult, replace()
  • Evals\AssertionResult (readonly)
  • one assertion's outcome
  • factories: pass(), fail()
  • key API: withSeverity(), withScore(), withThreshold(), withLabel(), withJudgeScore(), passed(): bool (score >= threshold ?? 1.0)
  • accessors: name(), score(), severity(), threshold(), message(), label(), judgeScore(), judgeClass(), toArray()
  • Evals\AssertionResults (readonly, Countable, IteratorAggregate)
  • immutable collection of AssertionResult
  • key API: with(), hasFailedGate(), hasFailedSoft()
  • accessors: all(), count()
  • Evals\AssertionSeverity -- Gate, Soft
  • Evals\ValueExpectation
  • fluent value assertion returned by EvalContext::expect()
  • key API: includes(), equals(), matches(string|EvalMatch), similarity() (Levenshtein-based, always Soft), satisfies(Closure); chain modifiers gate(), soft(), atLeast(), label()
  • note: the chain modifiers apply only to the LAST assertion this expectation recorded, not to every assertion the expectation has made
  • Evals\EvalRequirementFailed (extends RuntimeException) -- thrown by EvalContext::require() on failure; caught internally by EvalRunner, not user-visible
  • Evals\EvalSkipped (extends RuntimeException) -- thrown by EvalContext::skip(); caught internally by EvalRunner and turned into EvalVerdict::Skipped

Target & sessions:

  • Evals\CanRunAgentEvalTarget -- contract: open(?EvalSessionRequest): CanUseAgentEvalSession
  • Evals\LocalAgentTarget (readonly)
  • runs eval sessions in-process
  • factories: fromFactory(Closure(): CanControlAgentLoop $factory, ?EvalTracePolicy)
  • Evals\HttpAgentTarget (readonly)
  • runs eval sessions against a remote agent server over HTTP
  • key API: open(), attach(sessionId), sendTurn(), policy()
  • note: applies its EvalTracePolicy (default safe()) to whatever the remote server sends, so the HTTP path is safe by default rather than degrading to verbatim serialization
  • Evals\CanUseAgentEvalSession -- contract: send(message): EvalTurn, run(): AgentRun
  • Evals\LocalEvalSession -- CanUseAgentEvalSession over an in-process CanControlAgentLoop
  • Evals\HttpEvalSession -- CanUseAgentEvalSession over HttpAgentTarget; accessor: sessionId()
  • Evals\HttpTargetException (extends RuntimeException) -- thrown on a non-2xx response, malformed JSON, or a missing sessionId
  • Evals\EvalSessionRequest (readonly) -- optional caseId/description passed to CanRunAgentEvalTarget::open()
  • Evals\EvalTurn (readonly)
  • one turn of an eval session
  • accessors: index(), message(), run(): AgentRun, reply()

Run trace:

  • Evals\AgentRun (readonly)
  • immutable accumulated projection of an eval session, across turns
  • factories: fromState(), empty(), fromArray()
  • accessors: reply(), status(), succeeded(), tools(), events(), turns(), errors(), steps(), usage(), duration(), stepCount(), stopSignal(), llmProfile()
  • note: stopSignal() is the LAST turn's resolved signal and does NOT aggregate across turns -- per-turn signals live on EvalStep::stopSignal()
  • Evals\EvalStep (readonly)
  • immutable safe projection of one StepExecution
  • factories: fromStepExecution(), fromArray()
  • accessors: id(), turn(), index(), type(), outputMessages(), requestedToolCalls(), toolExecutions(), finishReason(), usage(), duration(), stopSignal(), errors(), hasErrors(), toArray()
  • note: carries no input messages and never serializes the raw InferenceResponse
  • Evals\EvalSteps (readonly, Countable, IteratorAggregate) -- ordered EvalStep collection; key API: with(), last(), usage(), duration(), toArray()/fromArray()
  • Evals\EvalToolExecutions (readonly, Countable, IteratorAggregate) -- collection of Data\ToolExecution
  • Evals\EvalEvents (readonly, Countable, IteratorAggregate) -- collection of arbitrary agent event objects captured during a run
  • Evals\EvalTracePolicy (readonly)
  • controls how much of a tool payload lands in a serialized trace
  • factories: safe() (default everywhere), full() (explicit opt-in, never a default)
  • key API: digest(mixed): array{hash, bytes, preview}, isDigest(), withPreviewBytes(), toArray()/fromArray()
  • accessors: isFull(), previewBytes() (DEFAULT_PREVIEW_BYTES = 120)
  • note: safe() digests tool call arguments, tool results, AND error messages -- there is no size threshold, short values are digested too; preview renders the value's SHAPE (<string:N>, <int>, <array:N>, <object:N> past MAX_PREVIEW_DEPTH = 6), never the payload itself

Judging:

  • Evals\CanJudgeAgentEval -- contract: judge(JudgeRequest): JudgeScore
  • Evals\AgentLoopJudge
  • agentic judge: runs a bounded AgentLoop that inspects the target's AgentRun and submits a verdict via the submit_judgment terminal tool
  • factories: fromBuilder(callable(): CanComposeAgentLoop $builderFactory) -- the factory must return a FRESH, not-yet-built builder on every call
  • accessors: llmProfile(), guardProfile(): array{configured, hooks}
  • note: every judge() call gets a fresh builder/loop/state/event-list/JudgeSubmissionInbox -- nothing leaks between calls, even repeated calls on the same instance
  • note: installs NO guards of its own -- warnIfGuardsMissing() only inspects the built loop's profile for UseGuards and, if absent, dispatches Events\JudgeGuardsNotConfigured at most once per instance; it never substitutes a limit. Install Capability\Core\UseGuards explicitly
  • Evals\PolyglotAgentJudge (readonly) -- lightweight judge backed by a raw LLM call expected to return {"score":..,"reason":..} JSON; factories: fromInference(Inference), fromInvoker(Closure(string): string)
  • Evals\FakeAgentJudge (readonly) -- deterministic judge double; factories: fromScore(), fromClosure(Closure(JudgeRequest): JudgeScore)
  • Evals\JudgeRequest (readonly) -- criterion, output, run: AgentRun (required, not optional), input, reference
  • Evals\JudgeScore (readonly) -- score (validated [0,1]), reason (non-empty), evidence: JudgeEvidence, ?run: AgentRun
  • Evals\JudgeEvidence (readonly, Countable, IteratorAggregate) -- ordered evidence strings backing a JudgeScore; factories: none(), of(); note: developer-visible support for the score, never hidden model reasoning
  • Evals\JudgeCriterion -- Factuality, Summarizes, ClosedQa, Sql
  • Evals\AgentJudgeAssertions (readonly)
  • returned by EvalContext::judge(); built-in criteria
  • key API: factuality(reference), summarizes(source), closedQa(question), sql(reference) -- each returns JudgeExpectation
  • Evals\JudgeExpectation
  • fluent judge assertion chain
  • key API: on(output) (replaces only the graded output; retains the run), gate(), soft(), atLeast(), label()
  • note: the chain only accumulates state -- the judge runs AT MOST ONCE, on first read of the recorded result (AssertionCollector::results()/at()); .on() never re-runs or re-judges. Severity defaults to Gate with no judge configured, Soft with one; a judge exception always forces Gate regardless of prior gate()/soft() calls
  • Evals\SubmitJudgmentTool (extends Tool\Tools\SimpleTool)
  • the judge's terminal tool (submit_judgment); validates score/reason/evidence and records a JudgeSubmission into its JudgeSubmissionInbox
  • constant: TOOL_NAME
  • Evals\JudgeSubmission (readonly) -- one validated submit_judgment call: score, reason, evidence
  • Evals\JudgeSubmissionInbox
  • mailbox shared between SubmitJudgmentTool and JudgeProtocolHook for one judge() call
  • key API: submit(), has(), get(), attempts()
  • note: holds at most one submission -- submit() never overwrites; attempts() counts only tool-body invocations, so a call blocked by JudgeProtocolHook does NOT increment it
  • Evals\JudgeProtocolHook (readonly, implements Hook\Contracts\HookInterface)
  • enforces the terminal-submission protocol on BeforeToolUse/AfterStep: blocks a second submit_judgment call, skips (does not block) any other tool call once a submission is recorded, and adds a StopReason::Completed stop signal after the submission step
  • Evals\JudgeProtocolException (extends RuntimeException) -- thrown by AgentLoopJudge::judge() when the protocol was violated (no submission, a blocked second submission, or a failed run); always converted to a Gate failure by JudgeExpectation::resolve()
  • Evals\JudgePromptRenderer (readonly) -- renders the judge's fixed system prompt and per-request user prompt; wraps the target trace in <untrusted-target-trace> markers (a labeling reduction, not a security boundary)
  • Evals\UseJudgeInference (readonly, implements Builder\Contracts\CanProvideAgentCapability)
  • recommended driver capability for judge builders passed to AgentLoopJudge::fromBuilder(): installs ToolCallingDriver with temperature: 0.0 by default (caller-supplied options win)
  • note: AgentLoopJudge never installs this on the developer's behalf -- it is documented as the recommended judge driver, not injected

Repetition:

  • Evals\EvalRepetition (readonly)
  • the N trials of one repeated case; present only when a case ran more than once
  • factories: fromTrials(list<EvalResult>, passRate)
  • accessors: trials(), trialCount(), passCount(), requiredPasses(), satisfied(), allSkipped(), judgeScoreMean() (null when nothing was judged), judgeScoreStdDev() (POPULATION deviation -- divided by N, not N-1; 0.0 for a single score, never a division by zero), representative() (first non-Passed trial, else the first trial), toArray()

Verdict, running & config:

  • Evals\EvalVerdict -- Passed, Failed, Scored, Skipped
  • Evals\EvalVerdictResolver (readonly)
  • key API: resolve(AssertionResults, skipped, ?error): EvalVerdict (error or failed gate -> Failed; skipped -> Skipped; failed soft -> Scored; else Passed), resolveRepeated(EvalRepetition): EvalVerdict (all-skipped -> Skipped; satisfied k-of-N -> Passed; else Failed)
  • static: requiredPasses(trials, passRate): int -- ceil(passRate * trials - 1e-9) clamped to [1, trials], guarding against IEEE-754 near-integer error
  • Evals\EvalExitCode -- Success = 0, EvalFailure = 1, ConfigurationError = 2
  • Evals\EvalRunner (readonly)
  • key API: run(AgentEvals, ?EvalRunOptions): EvalRunResult
  • note: each repeated trial opens a FRESH EvalContext/session (never EvalContext::newSession(), which shares collectors); repeat=1 returns the trial's EvalResult unchanged, not wrapped; the cooperative timeout is a per-trial budget, not per-case
  • Evals\EvalRunOptions (readonly)
  • factories: default()
  • key API: withFilter(), withTags(), withExcludedTags(), withStrict(), withSkipReport(), withVerbose(), withTimeout(), withRepeat(), withPassRate()
  • accessors: filter(), tags(), excludedTags(), strict(), skipReport(), verbose(), timeout(), repeat(), passRate()
  • note: constructor validates repeat >= 1 and passRate in (0, 1]; repetition only measures TARGET variance when the judge is separately pinned to a fixed temperature (e.g. via UseJudgeInference) -- AgentLoopJudge never installs that for you
  • Evals\EvalConfig (readonly)
  • factories: default()
  • key API: withTarget(), withJudge(), withReporters(), withReporter()
  • accessors: target(), judge(), reporters()
  • Evals\EvalApplication (readonly)
  • CLI entry point
  • key API: run(argv, ?callable $stdout, ?callable $stderr): int
  • flags: --filter=<glob>, --tag=<tag> (repeatable), --exclude-tag=<tag> (repeatable), --strict, --timeout=<seconds>, --repeat=<n>, --pass-rate=<r>, --junit=<path>, --list, --verbose, --json, --skip-report, -h/--help
  • note: loads <root>/evals.config.php when present (must return EvalConfig); --repeat rejects a non-whole-number or fractional value outright rather than silently truncating or casting it

Logging:

  • Evals\EvalLog (readonly) -- one log entry: message(), context(), toArray()
  • Evals\EvalLogs (readonly, Countable, IteratorAggregate) -- immutable EvalLog collection; factories: none(); key API: with()
  • Evals\EvalLogCollector -- mutable collector behind EvalContext::log(); key API: record(), logs(): EvalLogs

Reporting:

  • Evals\CanReportAgentEvals -- contract: id(), onRunStarted(caseCount), onEvalCompleted(EvalResult), onRunCompleted(EvalRunResult)
  • Evals\CanFailAgentEvalTestSuite (extends CanReportAgentEvals) -- marker for reporters that propagate the final assertion into the host test runner
  • Evals\ConsoleEvalReporter (readonly)
  • factories: fromWriter(Closure(string): void, verbose = false)
  • key API: withVerbose()
  • note: a repeated case prints a rate line (PASS 4/5 ... judge=0.88+/-0.06) instead of a single verdict; the judge= field is omitted -- not printed as a fabricated 0.00 -- when nothing in the case was judged
  • Evals\ArtifactEvalReporter
  • writes a full run's artifacts to disk under .instructor/evals/<run>/: per-eval details.json, events.ndjson, target-trace.json, target-steps.jsonl, per-judged-assertion judges/NNN.json (+ -steps.jsonl), per-trial trials/NNN/ for repeated cases, and run-level summary.json/results.jsonl
  • constructor: root, ?ClockInterface $clock, ?Closure(): ?string $gitShaResolver, ?Closure(): ?string $packageVersionResolver
  • accessors: runDirectory()
  • note: never writes a raw target-messages.json conversation snapshot -- that would bypass EvalTracePolicy::safe()'s digesting and reintroduce the exact leak class the trace hardening closed
  • Evals\JUnitEvalReporter -- writes JUnit XML to the path given at construction
  • Evals\PHPUnitEvalReporter (implements CanFailAgentEvalTestSuite) -- factories: default(); asserts EvalExitCode::Success via PHPUnit\Framework\Assert
  • Evals\PestEvalReporter (implements CanFailAgentEvalTestSuite) -- factories: default(); asserts EvalExitCode::Success via a Pest Expectation
  • Evals\EvalReporters (readonly, IteratorAggregate)
  • immutable, id-deduplicated reporter collection
  • factories: none()
  • key API: with(), withVerboseConsole() (upgrades any ConsoleEvalReporter in the collection in place)
  • Evals\EvalTestFailureMessage -- static fromResult(EvalRunResult): string; renders a CI-friendly multi-line failure summary (counts, per-eval failures, repetition rate, judge evidence)

Result:

  • Evals\EvalResult (readonly)
  • outcome of one eval case, possibly a repeated case's aggregate
  • accessors: id(), description(), verdict(), assertions(), run(), duration(), error(), skipReason(), logs(), repetition(): ?EvalRepetition, trials(), trialCount(), passCount(), judgeScoreMean(), judgeScoreStdDev(), provenance(), tokens(): array{target, judge, total}, toArray(?envelope)
  • note: provenance()['judge']['temperature'] is always null -- AgentLoopJudge's built loop exposes no temperature accessor, and reporting an assumed default would fabricate a value the judge may not have used; guardsWarningObserved is derived only from the presence of a JudgeGuardsNotConfigured event on the judge's own run, never from its absence
  • Evals\EvalRunResult (readonly, Countable, IteratorAggregate)
  • full result of one EvalRunner::run() call
  • key API: exitCode(?bool $strict = null): EvalExitCode (EvalFailure if any result Failed, or effective-strict with any result Scored; else EvalFailure if reporterErrors() !== []; else Success), provenance(), tokens(), toArray(?envelope)
  • accessors: all(), reporterErrors(), strict()

Events:

  • Evals\Events\JudgeGuardsNotConfigured (extends Events\AgentEvent) -- dispatched at most once per AgentLoopJudge instance when its built judge loop has no UseGuards capability; capability, suggestedFix