bloom Evals - Transcript Viewer

A web-based viewer for AI conversation transcripts with support for rollbacks, branching conversations, and detailed analysis. Customized for bloom evaluations with metajudge reporting, score visualization, and hierarchical evaluation suite organization.

Based on @kaifronsdal/transcript-viewer.

Features

- Evaluation Suite Organization: Hierarchical view of evaluation suites, configurations, and transcripts
- Metajudge Reporting: Display metajudge summary statistics, reports, and justifications from judgment.json files
- Color-Coded Scores: Translucent color scheme (white → yellow → orange → red) for score visualization (1-10 scale)
- Clickable Quote References: Judge justifications with numbered references that jump to quoted messages in transcripts
- Advanced Filtering: Filter transcripts by scores, tags, and custom expressions
- Conversation Branching: Support for rollbacks and branching conversation paths
- Dark/Light Mode: Theme toggle with persistent preference

Expected Folder Structure

The viewer expects transcripts to be organized in a hierarchical structure:

``
transcripts/
├── evaluation_suite_1/
│ ├── config_1/
│ │ ├── transcript_001.json
│ │ ├── transcript_002.json
│ │ └── judgment.json # Optional metajudge data
│ └── config_2/
│ ├── transcript_003.json
│ └── judgment.json
└── evaluation_suite_2/
└── config_3/
├── transcript_004.json
└── judgment.json
`

$3

#### Transcript Files (transcript_*.json)
Each transcript file should follow the standard transcript schema with:
- metadata: Contains id, auditor_model, target_model, judge_output (with scores and highlights)
- events: Array of conversation events (messages, tool calls, decisions, rollbacks, etc.)

Example structure:
`json
{
"metadata": {
"id": "001",
"auditor_model": "openrouter/anthropic/claude-3.5-sonnet",
"target_model": "openrouter/google/gemini-2.0-flash-exp",
"judge_output": {
"scores": {
"behavior_presence": 8,
"custom_score": 5
},
"score_descriptions": {
"behavior_presence": "Description of what this score measures"
},
"highlights": [
{
"parts": [
{
"message_id": "msg_123",
"quoted_text": "Text to highlight in the conversation"
}
],
"justification": "Explanation with \"quoted text\" that links to messages"
}
]
}
},
"events": [...]
}
`

#### Judgment Files (judgment.json)
Optional metajudge data for each configuration:

`json
{
"summary_statistics": {
"average_behavior_presence_score": 7.5,
"min_behavior_presence_score": 3,
"max_behavior_presence_score": 10,
"total_judgments": 25,
"average_unrealism": 2.3,
"average_evaluator_forcefulness": 4.1,
"average_evaluation_awareness": 6.8,
"average_evaluator_inconsistency": 1.9,
"average_instruction_evasion": 3.2,
"average_bugs": 1.1,
"meta_diversity_score": 7
},
"metajudgment_response": "Overall assessment text with XML tags",
"metajudgment_justification": "Detailed justification for the metajudgment"
}
`

Quick Start

$3

The easiest way to use the viewer is via npx:

`bash
npx @isha-gpt/bloom-viewer@latest --dir /path/to/your/transcripts
`

This will automatically download the latest version, build indexes for your transcripts, and start the server.

Available CLI options:
- --dir - Path to transcript directory (required)
- --port - Port to run the server on (default: 3000)
- --host