RunResult, analyzers inspect trajectories, metrics, errors, latency, model output, and tool calls. Their output is attached to each attempt under analysis_result.<AnalyzerId> and aggregated into analysis_summary.json / analysis_summary.md.
Badcase detection
Flag failures such as runtime exceptions, truncation, JSON errors, repetition, latency spikes, or terminal misuse.
Statistics
Compute distributions such as step counts, tool-call counts, duration, token length, and value-count summaries.
Qualitative analysis
Use LLM-backed analyzers to annotate trajectory phases, summarize behavior, and render segment-graph reports.
Component registry
agentcompass list analyzer prints registered analyzers, and agentcompass list dump exports them to agentcompass_components.md.Run with Evaluation
Re-run on Existing Results
analysis copies the input run into a new timestamped sibling. Use --output to choose the copy destination, or --override to mutate the input run in place.
Selection Rules
Supported Families
Currentagentcompass list analyzer output reports 25 registered analyzers:
- Basic statistics:
BasicMetricAnalyzer,TrajectoryTimeCostAnalyzer,CompletionLengthAnalyzer. - Error detection:
ExceptionAnalyzer,TerminalBench2ExceptionAnalyzer,TruncationAnalyzer,JSONErrorAnalyzer,EmptyContentAnalyzer. - Efficiency:
LLMInferLatencyAnalyzer,ToolExecutionLatencyAnalyzer. - Behavior patterns: repetition, keyword, mixed-language, network-operation, command-running, and reward-hacking analyzers.
- Qualitative:
QualitativeAnalyzer,MultiQualitativeAnalyzer.
Output Shape
Per-task details keep every analyzer result underanalysis_result. Aggregated summaries group analyzers by category and render badcase ratios, average scores, numeric stats, and value counts declared by each analyzer.