{
  "interval": {
    "intervalStart": "2025-08-20T00:00:00.000Z",
    "intervalEnd": "2025-08-21T00:00:00.000Z",
    "intervalType": "day"
  },
  "repository": "elizaos/eliza",
  "overview": "From 2025-08-20 to 2025-08-21, elizaos/eliza had 2 new PRs (3 merged), 0 new issues, and 4 active contributors.",
  "topIssues": [
    {
      "id": "I_kwDOMT5cIs7GSrbB",
      "title": "Epic: Scenario Matrix Runner and Reporting System",
      "author": "linear",
      "number": 5781,
      "repository": "elizaos/eliza",
      "body": "---\n\n### **Epic 1: Scenario Matrix Runner**\n\n**Goal:** Create a robust CLI tool to execute a scenario multiple times across a matrix of different configurations.\n\n* **Ticket 1.1:** [ELIZA-651](https://linear.app/eliza-labs/issue/ELIZA-651/design-and-implement-scenario-matrix-configuration)\n  * **Task:** Define a clear and extensible YAML schema for the matrix configuration file (e.g., `matrix.scenario.yaml`).\n  * **Acceptance Criteria:**\n    * The schema must support defining a `base_scenario` path.\n    * It must support a `runs_per_combination` integer.\n    * It must include a `matrix` array where each item specifies a `parameter` (using dot-notation like `character.llm.model`) and a list of `values`.\n    * Create TypeScript interfaces for this schema.\n    * Implement a validation function using a library like Zod to provide clear errors on misconfiguration.\n* **Ticket 1.2:** [ELIZA-652](https://linear.app/eliza-labs/issue/ELIZA-652/implement-elizaos-scenario-matrix-cli-command)\n  * **Task:** Create the new top-level CLI command `elizaos scenario matrix <config_path>`.\n  * **Acceptance Criteria:**\n    * The command is registered in `packages/cli` using `yargs`.\n    * It correctly parses the path to the matrix config file.\n    * The command's main function should read, parse, and validate the specified config file.\n* **Ticket 1.3:** [ELIZA-653](https://linear.app/eliza-labs/issue/ELIZA-653/develop-parameter-override-system)\n  * **Task:** Implement the core logic that dynamically overrides parameters in the base scenario for each run.\n  * **Acceptance Criteria:**\n    * Create a utility that takes a loaded scenario object, a dot-notation path (e.g., `run[0].input`), and a value.\n    * The utility must correctly set the value in the object, handling nested objects and array indices.\n    * This function will be the engine that creates the unique scenario configurations for each matrix run.\n* **Ticket 1.4:** [ELIZA-655](https://linear.app/eliza-labs/issue/ELIZA-655/implement-run-orchestration-and-isolation)\n  * **Task:** Build the main execution loop that runs all generated scenario combinations.\n  * **Acceptance Criteria:**\n    * The command must generate a Cartesian product of all matrix parameters.\n    * It must loop through each unique combination the specified number of times (`runs_per_combination`).\n    * For each run, it must programmatically invoke the existing scenario runner logic with the overridden in-memory scenario object.\n    * **Crucially**, ensure each run is completely isolated by cleaning up any generated artifacts (e.g., test databases, temporary files) between executions.\n\n---\n\n### **Epic 2: Advanced Evaluation & Data Collection**\n\n**Goal:** Enhance the scenario runner to capture rich, structured data about agent performance, moving beyond a simple pass/fail.\n\n* **Ticket 2.1:** [ELIZA-656](https://linear.app/eliza-labs/issue/ELIZA-656/implement-structured-json-output-from-evaluators)\n  * **Task:** Rework the `EvaluationEngine` and evaluators, especially the `LLMJudge`.\n  * **Acceptance Criteria:**\n    * Evaluators must return a structured JSON object, not a boolean.\n    * The `LLMJudge` must be updated with a new prompt that instructs it to generate the `qualitative_summary` and `capability_checklist`.\n    * The output schema must be strongly typed within the codebase.\n* **Ticket 2.2:** [ELIZA-657](https://linear.app/eliza-labs/issue/ELIZA-657/implement-scenario-capability-definitions)\n  * **Task:** Extend the `.scenario.yaml` schema to allow developers to define the capabilities to be tested.\n  * **Acceptance Criteria:**\n    * A new `capabilities` array can be added under an `llm_judge` evaluation in the scenario file.\n    * The `LLMJudge` will use this list to dynamically generate the `capability_checklist` in its output for each run.\n* **Ticket 2.3:** [ELIZA-658](https://linear.app/eliza-labs/issue/ELIZA-658/implement-agent-trajectory-logging)\n  * **Task:** Instrument the `AgentRuntime` to log the detailed sequence of cognitive steps the agent takes.\n  * **Acceptance Criteria:**\n    * The runtime must capture every `THINK` (the LLM's reasoning), `ACTION` (the tool call with parameters), and `OBSERVATION` (the result from the tool).\n    * This trajectory must be stored as an ordered array in the final JSON result for each run.\n* **Ticket 2.4:** [ELIZA-659](https://linear.app/eliza-labs/issue/ELIZA-659/centralize-and-serialize-run-data)\n  * **Task:** After each scenario run, collect all generated data into the final JSON structure and save it to disk.\n  * **Acceptance Criteria:**\n    * The matrix runner will create a unique output directory for each execution (e.g., `output/matrix-run-<timestamp>`).\n    * Each individual scenario run within the matrix will be saved as a separate file (e.g., `run-001.json`, `run-002.json`).\n    * The saved file must conform to the rich data structure we designed, including parameters, metrics, summary, trajectory, and checklist.\n\n---\n\n### **Epic 3: Performance Reporting Dashboard**\n\n**Goal:** Create a tool to generate a clear, insightful, and shareable HTML report from the aggregated data of a matrix run.\n\n* **Ticket 3.1:** [ELIZA-660](https://linear.app/eliza-labs/issue/ELIZA-660/implement-elizaos-report-generate-command)\n  * **Task:** Create a new CLI command to generate the report.\n  * **Acceptance Criteria:**\n    * The command is `elizaos report generate <input_dir>`.\n    * It reads all `run-*.json` files from the specified directory.\n    * It aggregates the data, calculating statistics like success rates per capability, average execution times, etc., grouped by the matrix parameters.\n* **Ticket 3.2:** [ELIZA-661](https://linear.app/eliza-labs/issue/ELIZA-661/design-and-build-html-report-template)\n  * **Task:** Create a self-contained, professional HTML file to serve as the report template.\n  * **Acceptance Criteria:**\n    * The design should be clean and easy to navigate.\n    * Use a lightweight CSS framework for styling.\n    * Integrate a charting library like [Chart.js](https://www.chartjs.org/) for data visualization.\n* **Ticket 3.3:** [ELIZA-662](https://linear.app/eliza-labs/issue/ELIZA-662/implement-dynamic-report-rendering)\n  * **Task:** Populate the HTML template with the aggregated data and render the final report.\n  * **Acceptance Criteria:**\n    * Use a templating engine to inject the aggregated statistics into the report.\n    * Generate interactive charts (e.g., bar charts for capability success rates, histograms for performance metrics).\n    * Implement a visual representation (e.g., a Sankey diagram or flowchart) for the most common action trajectories to analyze action chaining.",
      "createdAt": "2025-08-16T03:00:36Z",
      "closedAt": "2025-08-20T02:31:10Z",
      "state": "CLOSED",
      "commentCount": 1
    }
  ],
  "topPRs": [
    {
      "id": "PR_kwDOMT5cIs6kWSnk",
      "title": "feat: Sessions API ++",
      "author": "ChristopherTrimboli",
      "number": 5799,
      "body": "## Enhanced Session Management with Advanced Timeout Configuration and Lifecycle Control\r\n\r\n### Overview\r\nThis PR significantly enhances the sessions API with comprehensive timeout management, auto-renewal capabilities, and robust error handling. The changes transform the basic session management into an enterprise-ready system with configurable lifecycles, warning states, and graceful cleanup mechanisms.\r\n\r\n### Key Improvements\r\n\r\n#### 🔧 **Advanced Timeout Configuration System**\r\n**Before:** Simple fixed timeout (`SESSION_TIMEOUT_MS`)\r\n**After:** Multi-layered timeout configuration with:\r\n- Configurable timeout minutes (5-1440 min range)\r\n- Auto-renewal capabilities\r\n- Maximum session duration limits\r\n- Warning threshold notifications\r\n- Agent-specific timeout settings\r\n- Session-specific overrides\r\n\r\n#### 🛡️ **Robust Error Handling**\r\n**Before:** Basic `errorResponse` function with generic error messages\r\n**After:** Comprehensive error system with:\r\n- 11 custom error classes (`SessionNotFoundError`, `SessionExpiredError`, etc.)\r\n- Detailed error context and metadata\r\n- Centralized error middleware\r\n- Type-safe error handling throughout\r\n\r\n#### ♻️ **Session Lifecycle Management**\r\n**New Features:**\r\n- **Auto-renewal**: Sessions automatically extend on activity\r\n- **Manual renewal**: `/sessions/:id/renew` endpoint\r\n- **Heartbeat support**: `/sessions/:id/heartbeat` for keep-alive\r\n- **Warning states**: Proactive expiration notifications\r\n- **Renewal tracking**: Monitor renewal counts and patterns\r\n- **Graceful expiration**: Proper cleanup of expired sessions\r\n\r\n#### 🎯 **Type Safety Enhancements**\r\n**Added:**\r\n- Type guards for all request/response objects\r\n- `isValidSession()`, `isCreateSessionRequest()`, `isSendMessageRequest()`\r\n- `isValidTimeoutConfig()` for configuration validation\r\n- Proper TypeScript interfaces for all data structures\r\n\r\n#### 🔌 **New API Endpoints**\r\n```\r\nPOST   /sessions/:id/renew      - Manually renew a session\r\nPOST   /sessions/:id/heartbeat  - Keep session alive\r\nPATCH  /sessions/:id/timeout    - Update timeout configuration\r\n```\r\n\r\n#### 📊 **Enhanced Health Monitoring**\r\n**Before:** Basic session count\r\n**After:** Detailed health metrics including:\r\n- Active vs expired sessions\r\n- Sessions expiring soon\r\n- Invalid session detection\r\n- Server uptime tracking\r\n\r\n#### 🧹 **Memory Leak Prevention**\r\n**New:**\r\n- `SessionRouter` interface with cleanup method\r\n- Proper interval management with `activeCleanupIntervals` Set\r\n- Process handler registration tracking\r\n- Resource cleanup on router destruction\r\n\r\n#### 📈 **Improved Pagination**\r\n**Enhanced:**\r\n- Better cursor-based pagination with proper `before`/`after` handling\r\n- Cursor information in response for easier client implementation\r\n- Proper handling of edge cases in date range queries\r\n\r\n### Breaking Changes\r\nNone - All existing endpoints maintain backward compatibility while adding optional new features.\r\n\r\n### Migration Guide\r\nExisting clients will continue to work without changes. To leverage new features:\r\n\r\n1. **Enable auto-renewal**: Include `timeoutConfig: { autoRenew: true }` in session creation\r\n2. **Set custom timeouts**: Add `timeoutConfig: { timeoutMinutes: 60 }` \r\n3. **Monitor expiration**: Check `isNearExpiration` in responses\r\n4. **Use heartbeats**: Send periodic POST to `/sessions/:id/heartbeat`\r\n\r\n### Configuration\r\nNew environment variables (all optional):\r\n```env\r\nSESSION_DEFAULT_TIMEOUT_MINUTES=30      # Default: 30\r\nSESSION_MIN_TIMEOUT_MINUTES=5           # Default: 5  \r\nSESSION_MAX_TIMEOUT_MINUTES=1440        # Default: 1440 (24h)\r\nSESSION_MAX_DURATION_MINUTES=720        # Default: 720 (12h)\r\nSESSION_WARNING_THRESHOLD_MINUTES=5     # Default: 5\r\nSESSION_CLEANUP_INTERVAL_MINUTES=5      # Default: 5\r\nCLEAR_SESSIONS_ON_SHUTDOWN=false        # Default: false\r\n```\r\n\r\n### Technical Details\r\n\r\n#### Session State Structure\r\n```typescript\r\ninterface Session {\r\n  // ... existing fields ...\r\n  expiresAt: Date;              // NEW: Calculated expiration time\r\n  timeoutConfig: {              // NEW: Comprehensive timeout settings\r\n    timeoutMinutes?: number;\r\n    autoRenew?: boolean;\r\n    maxDurationMinutes?: number;\r\n    warningThresholdMinutes?: number;\r\n  };\r\n  renewalCount: number;         // NEW: Track renewal history\r\n  warningState?: {              // NEW: Warning notification state\r\n    sent: boolean;\r\n    sentAt: Date;\r\n  };\r\n}\r\n```\r\n\r\n#### Error Classes Hierarchy\r\n- `SessionError` (base class)\r\n  - `SessionNotFoundError`\r\n  - `SessionExpiredError`\r\n  - `SessionCreationError`\r\n  - `SessionRenewalError`\r\n  - `MessageSendError`\r\n  - `InvalidUuidError`\r\n  - `MissingFieldsError`\r\n  - `InvalidContentError`\r\n  - `InvalidMetadataError`\r\n  - `InvalidPaginationError`\r\n  - `InvalidTimeoutConfigError`\r\n  - `AgentNotFoundError`\r\n\r\n### Testing\r\n- ✅ All error scenarios covered with proper error classes\r\n- ✅ Timeout and renewal logic validated\r\n- ✅ Memory leak prevention verified\r\n- ✅ Backward compatibility maintained\r\n\r\n### Performance Impact\r\n- Minimal overhead from timeout calculations\r\n- Efficient cleanup with configurable intervals\r\n- Optimized validation with early returns\r\n- No impact on existing session operations\r\n\r\n### Security Considerations\r\n- Session expiration enforced at API level\r\n- Automatic cleanup of stale sessions\r\n- Validation of all input parameters\r\n- No sensitive data in error responses (except in dev mode)",
      "repository": "elizaos/eliza",
      "createdAt": "2025-08-19T18:39:03Z",
      "mergedAt": "2025-08-20T14:10:28Z",
      "additions": 2377,
      "deletions": 559
    },
    {
      "id": "PR_kwDOMT5cIs6j_jmP",
      "title": "feat(bootstrap): async embedding generation via queue service",
      "author": "0xbbjoker",
      "number": 5793,
      "body": "# Relates to\r\n\r\nPerformance improvement for message processing latency - embeddings were blocking runtime for 500ms+ per message\r\n\r\n# Risks\r\n\r\n**Low risk** - The change is backward compatible and includes fallback behavior. Main risks:\r\n- Memory usage could increase if embedding queue grows large (mitigated by priority queue and processing limits)\r\n- Embeddings may be generated slightly later than before (acceptable trade-off for faster message responses)\r\n\r\n# Background\r\n\r\n## What does this PR do?\r\n\r\nThis PR implements asynchronous embedding generation to prevent blocking the message processing pipeline. Previously, generating embeddings for messages would block the runtime for 500ms+ per API call, delaying message responses. Now embeddings are queued and processed in the background by a dedicated service.\r\n\r\nKey changes:\r\n- Added `queueEmbeddingGeneration` method to `AgentRuntime` that emits events instead of blocking\r\n- Created `EmbeddingGenerationService` that listens for `EMBEDDING_QUEUED` events and processes them asynchronously\r\n- Implemented a priority queue system (high/normal/low) for embedding generation\r\n- Added comprehensive test coverage for both the runtime method and service\r\n\r\n## What kind of change is this?\r\n\r\n**Features** (non-breaking change which adds functionality) + **Improvements** (performance enhancement to existing features)\r\n\r\n# Documentation changes needed?\r\n\r\nMy changes require a change to the project documentation.\r\n- [ ] Need to document the new `queueEmbeddingGeneration` method in runtime API docs\r\n- [ ] Need to document the `EmbeddingGenerationService` in plugin documentation\r\n\r\n# Testing\r\n\r\n## Where should a reviewer start?\r\n\r\n1. Review `packages/core/src/runtime.ts` - see the new `queueEmbeddingGeneration` method (lines 2024-2055)\r\n2. Review `packages/plugin-bootstrap/src/services/embedding.ts` - the new service implementation\r\n3. Check the integration in `packages/plugin-bootstrap/src/index.ts` where we replaced `addEmbeddingToMemory` calls with `queueEmbeddingGeneration`\r\n4. Review test coverage in:\r\n   - `packages/plugin-bootstrap/src/__tests__/embedding-service.test.ts`\r\n   - `packages/core/src/__tests__/runtime-embedding.test.ts`\r\n\r\n## Detailed testing steps\r\n\r\n### Automated Tests\r\n```bash\r\n# Run core tests\r\ncd packages/core && bun run test\r\n\r\n# Run bootstrap plugin tests  \r\ncd packages/plugin-bootstrap && bun run test\r\n```\r\n\r\n### Manual Testing\r\n1. Start an agent with the bootstrap plugin\r\n2. Send a message to the agent\r\n3. Observe that:\r\n   - The agent responds quickly (no 500ms delay)\r\n   - Check logs for \"Queueing embedding generation\" and \"Processing embedding for memory\"\r\n   - Verify embeddings are eventually generated (check database or memory state)\r\n\r\n### Performance Verification\r\nThe tests include a specific test case demonstrating non-blocking behavior:\r\n- `\"should not block runtime while generating embeddings\"` test verifies that embedding generation happens asynchronously\r\n- Response times should be significantly faster (500ms+ improvement per message)\r\n\r\n## Before/After Performance\r\n\r\n**Before:** Message → Create Memory → Generate Embedding (500ms+) → Process Actions → Respond  \r\n**After:** Message → Create Memory → Queue Embedding (instant) → Process Actions → Respond | (Embedding generated in background)\r\n\r\n# Deploy Notes\r\n\r\nNo deployment changes required. The service is automatically registered when the bootstrap plugin is loaded.\r\n\r\n## Database changes\r\n\r\nNone - uses existing memory table structure",
      "repository": "elizaos/eliza",
      "createdAt": "2025-08-17T15:38:09Z",
      "mergedAt": "2025-08-20T23:23:49Z",
      "additions": 1736,
      "deletions": 34
    },
    {
      "id": "PR_kwDOMT5cIs6kiDQa",
      "title": "add bench plugin locally",
      "author": "0xbbjoker",
      "number": 5800,
      "body": "",
      "repository": "elizaos/eliza",
      "createdAt": "2025-08-20T16:32:11Z",
      "mergedAt": null,
      "additions": 341,
      "deletions": 138
    },
    {
      "id": "PR_kwDOMT5cIs6kkzE3",
      "title": "feat: getServiceLoadPromise",
      "author": "odilitime",
      "number": 5801,
      "body": "# Risks\r\n\r\nLow\r\n\r\n# Background\r\n\r\n## What does this PR do?\r\n\r\n- add getServiceLoadPromise interface to runtime\r\n- fix component queries in plugin-sql (was too easy for dates to get invalid, this is a more flexible set up, allowing the intended effect)\r\n- remove bootstrap capabilities\r\n- fixes ensureConnections large addPartipants, now does it in batches\r\n- lower logging\r\n\r\n## What kind of change is this?\r\n\r\nImprovements (misc. changes to existing features)\r\n\r\n## Why are we doing this? Any context or related work?\r\n\r\nremoves the need for polling and timers when waiting for a service to be available in initialization\r\n\r\n# Documentation changes needed?\r\n\r\nMy changes do not require a change to the project documentation.",
      "repository": "elizaos/eliza",
      "createdAt": "2025-08-20T21:26:25Z",
      "mergedAt": "2025-08-20T22:44:58Z",
      "additions": 66,
      "deletions": 12
    }
  ],
  "codeChanges": {
    "additions": 4179,
    "deletions": 605,
    "files": 23,
    "commitCount": 19
  },
  "completedItems": [
    {
      "title": "feat(bootstrap): async embedding generation via queue service",
      "prNumber": 5793,
      "type": "feature",
      "body": "# Relates to\r\n\r\nPerformance improvement for message processing latency - embeddings were blocking runtime for 500ms+ per message\r\n\r\n# Risks\r\n\r\n**Low risk** - The change is backward compatible and includes fallback behavior. Main risks:\r\n- M",
      "files": [
        "packages/core/src/__tests__/runtime-embedding.test.ts",
        "packages/core/src/runtime.ts",
        "packages/core/src/types/events.ts",
        "packages/core/src/types/runtime.ts",
        "packages/plugin-bootstrap/src/__tests__/embedding-service.test.ts",
        "packages/plugin-bootstrap/src/__tests__/evaluators.test.ts",
        "packages/plugin-bootstrap/src/__tests__/test-utils.ts",
        "packages/plugin-bootstrap/src/evaluators/reflection.ts",
        "packages/plugin-bootstrap/src/index.ts",
        "packages/plugin-bootstrap/src/services/embedding.ts",
        "packages/plugin-quick-starter/src/__tests__/test-utils.ts",
        "packages/plugin-starter/src/__tests__/test-utils.ts",
        "packages/server/src/__tests__/test-utils/mocks.ts",
        "packages/test-utils/src/mocks/runtime.ts",
        "packages/plugin-bootstrap/src/__tests__/embedding-queue-management.test.ts"
      ]
    },
    {
      "title": "feat: Sessions API ++",
      "prNumber": 5799,
      "type": "feature",
      "body": "## Enhanced Session Management with Advanced Timeout Configuration and Lifecycle Control\r\n\r\n### Overview\r\nThis PR significantly enhances the sessions API with comprehensive timeout management, auto-renewal capabilities, and robust error han",
      "files": [
        "packages/server/src/api/messaging/__tests__/sessions.test.ts",
        "packages/server/src/api/messaging/channels.ts",
        "packages/server/src/api/messaging/errors/SessionErrors.ts",
        "packages/server/src/api/messaging/sessions.ts",
        "packages/server/src/index.ts",
        "packages/server/src/types/sessions.ts"
      ]
    },
    {
      "title": "feat: getServiceLoadPromise",
      "prNumber": 5801,
      "type": "feature",
      "body": "# Risks\r\n\r\nLow\r\n\r\n# Background\r\n\r\n## What does this PR do?\r\n\r\n- add getServiceLoadPromise interface to runtime\r\n- fix component queries in plugin-sql (was too easy for dates to get invalid, this is a more flexible set up, allowing the inten",
      "files": [
        "packages/core/src/runtime.ts",
        "packages/core/src/settings.ts",
        "packages/core/src/types/runtime.ts",
        "packages/plugin-bootstrap/src/index.ts",
        "packages/plugin-sql/src/base.ts",
        "packages/test-utils/src/mocks/runtime.ts"
      ]
    }
  ],
  "topContributors": [
    {
      "username": "odilitime",
      "avatarUrl": "https://avatars.githubusercontent.com/u/16395496?u=c9bac48e632aae594a0d85aaf9e9c9c69b674d8b&v=4",
      "totalScore": 35.48334355740107,
      "prScore": 35.28334355740107,
      "issueScore": 0,
      "reviewScore": 0,
      "commentScore": 0.2,
      "summary": null
    },
    {
      "username": "0xbbjoker",
      "avatarUrl": "https://avatars.githubusercontent.com/u/54844437?u=90fe1762420de6ad493a1c1582f1f70c0d87d8e2&v=4",
      "totalScore": 30.608251363656777,
      "prScore": 30.608251363656777,
      "issueScore": 0,
      "reviewScore": 0,
      "commentScore": 0,
      "summary": null
    },
    {
      "username": "wtfsayo",
      "avatarUrl": "https://avatars.githubusercontent.com/u/82053242?u=98209a1f10456f42d4d2fa71db4d5bf4a672cbc3&v=4",
      "totalScore": 15,
      "prScore": 0,
      "issueScore": 0,
      "reviewScore": 15,
      "commentScore": 0,
      "summary": null
    },
    {
      "username": "ChristopherTrimboli",
      "avatarUrl": "https://avatars.githubusercontent.com/u/27584221?u=0d816ce1dcdea8f925aba18bb710153d4a87a719&v=4",
      "totalScore": 5,
      "prScore": 0,
      "issueScore": 0,
      "reviewScore": 5,
      "commentScore": 0,
      "summary": null
    }
  ],
  "newPRs": 2,
  "mergedPRs": 3,
  "newIssues": 0,
  "closedIssues": 1,
  "activeContributors": 4
}