{ "interval": { "intervalStart": "2025-08-25T00:00:00.000Z", "intervalEnd": "2025-08-26T00:00:00.000Z", "intervalType": "day" }, "repository": "elizaos/eliza", "overview": "From 2025-08-25 to 2025-08-26, elizaos/eliza had 5 new PRs (5 merged), 3 new issues, and 9 active contributors.", "topIssues": [ { "id": "I_kwDOMT5cIs7H3Obz", "title": "Implement Dynamic Prompting (Multi-Turn Conversations) in ElizaOS Scenarios", "author": "linear", "number": 5819, "repository": "elizaos/eliza", "body": "# Dynamic Prompting Implementation for ElizaOS Scenarios\n\n## Overview\n\nImplement **Dynamic Prompting** (multi-turn conversations) in ElizaOS scenarios to enable sophisticated testing of agent behavior through extended conversations where an LLM simulates realistic user responses.\n\n## Problem Statement\n\nCurrent ElizaOS scenarios are limited to single-turn interactions, making it impossible to test:\n\n* Multi-step problem solving\n* Context retention across conversation turns\n* Clarification and follow-up question handling\n* Natural conversation flow assessment\n* Error recovery and correction flows\n\n## Solution\n\nExtend the existing scenario framework to support multi-turn conversations where:\n\n1. **Agent** receives initial user input\n2. **Agent** responds with thoughts, actions, and replies\n3. **LLM Simulator** generates realistic user follow-up based on agent response\n4. **Agent** continues the conversation based on simulated user input\n5. Process repeats for specified number of turns or until conditions are met\n\n## Key Requirements\n\n### 1\\. Backward Compatibility\n\n* **100% backward compatible** - all existing scenarios must work unchanged\n* Gradual adoption path for teams to enhance existing scenarios\n* No breaking changes to existing APIs or CLI commands\n\n### 2\\. Core Components\n\n#### Schema Extensions\n\n* Extend `RunStepSchema` with optional `conversation` field\n* Add new evaluation types: `conversation_length`, `conversation_flow`, `user_satisfaction`, `context_retention`\n* Support conversation configuration with user simulator settings\n\n#### User Simulator\n\n* LLM-based response generation with persona-driven prompts\n* Configurable personality, objectives, constraints, and knowledge level\n* Realistic conversation progression based on agent responses\n\n#### Conversation Manager\n\n* Multi-turn execution orchestration\n* Termination condition checking (satisfaction, solution provided, escalation needed)\n* Turn-level and final evaluation support\n* Conversation transcript generation\n\n#### New Evaluators\n\n* **Conversation Length**: Validate optimal conversation duration\n* **Conversation Flow**: Detect required conversation patterns\n* **User Satisfaction**: Measure user satisfaction through sentiment analysis\n* **Context Retention**: Verify agent memory across conversation turns\n\n### 3\\. Configuration Examples\n\n#### Basic Multi-Turn Conversation\n\n```yaml\nrun:\n - input: \"I need help with something\"\n conversation:\n max_turns: 4\n user_simulator:\n persona: \"polite customer with a billing question\"\n objective: \"find out why charged twice this month\"\n temperature: 0.6\n final_evaluations:\n - type: \"llm_judge\"\n prompt: \"Did the agent successfully help resolve the billing issue?\"\n expected: \"yes\"\n```\n\n#### Advanced Persona-Driven Conversation\n\n```yaml\nrun:\n - input: \"This is ridiculous! Your product doesn't work!\"\n conversation:\n max_turns: 6\n user_simulator:\n persona: \"angry customer who had bad experience\"\n objective: \"vent frustration but eventually want help\"\n style: \"initially hostile, gradually becomes cooperative if handled well\"\n constraints:\n - \"Start with complaints and criticism\"\n - \"Don't accept first solution immediately\"\n - \"Become more cooperative if agent shows empathy\"\n termination_conditions:\n - type: \"user_expresses_satisfaction\"\n - type: \"agent_escalates_to_human\"\n```\n\n## Implementation Plan\n\n### Phase 1: Core Infrastructure (Weeks 1-2)\n\n- [ ] Schema extensions and type definitions\n- [ ] User Simulator implementation\n- [ ] Basic conversation flow testing\n\n### Phase 2: Conversation Management (Weeks 3-4)\n\n- [ ] ConversationManager class implementation\n- [ ] Provider integration (LocalEnvironmentProvider, E2BEnvironmentProvider)\n- [ ] Termination condition logic\n\n### Phase 3: Evaluation System (Weeks 5-6)\n\n- [ ] New conversation evaluators implementation\n- [ ] EvaluationEngine integration\n- [ ] End-to-end testing\n\n### Phase 4: Polish and Documentation (Week 7)\n\n- [ ] Example scenarios creation\n- [ ] Documentation updates\n- [ ] Performance optimizations\n\n## Technical Specifications\n\n### File Structure\n\n```\npackages/cli/src/commands/scenario/src/\n├── schema.ts (extend)\n├── conversation-types.ts (new)\n├── UserSimulator.ts (new)\n├── ConversationManager.ts (new)\n├── ConversationEvaluators.ts (new)\n├── LocalEnvironmentProvider.ts (modify)\n├── E2BEnvironmentProvider.ts (modify)\n└── __tests__/\n ├── UserSimulator.test.ts (new)\n ├── ConversationManager.test.ts (new)\n ├── ConversationEvaluators.test.ts (new)\n └── integration/ (new tests)\n```\n\n### Key Interfaces\n\n```typescript\ninterface ConversationConfig {\n max_turns: number;\n user_simulator: UserSimulatorConfig;\n termination_conditions: TerminationCondition[];\n turn_evaluations: EvaluationSchema[];\n final_evaluations: EvaluationSchema[];\n}\n\ninterface UserSimulatorConfig {\n persona: string;\n objective: string;\n style?: string;\n constraints: string[];\n knowledge_level: 'beginner' | 'intermediate' | 'expert';\n}\n```\n\n## Success Criteria\n\n### Functional Requirements\n\n- [ ] Single-turn scenarios continue to work unchanged\n- [ ] Multi-turn conversations execute successfully\n- [ ] User simulator generates realistic, persona-consistent responses\n- [ ] Termination conditions work correctly\n- [ ] All new evaluation types function properly\n- [ ] Matrix testing supports conversation parameters\n\n### Performance Requirements\n\n- [ ] Conversation scenarios complete within reasonable time limits\n- [ ] Memory usage remains within acceptable bounds\n- [ ] LLM API usage is optimized and rate-limited appropriately\n\n### Quality Requirements\n\n- [ ] Comprehensive test coverage (unit, integration, e2e)\n- [ ] Clear error handling and debugging capabilities\n- [ ] Well-documented examples and migration guide\n\n## Risk Mitigation\n\n### Technical Risks\n\n* **LLM API failures**: Implement retry logic and graceful degradation\n* **Infinite loops**: Hard max_turns limit and timeout mechanisms\n* **Memory leaks**: Turn-based cleanup and conversation archiving\n\n### Integration Risks\n\n* **Breaking existing scenarios**: Comprehensive backward compatibility testing\n* **Performance impact**: Resource monitoring and optimization\n\n## Dependencies\n\n* Existing `askAgentViaApi` infrastructure\n* Current evaluation engine and trajectory reconstruction\n* LLM provider integration for user simulation\n* Database schema (no changes required)\n\n## Acceptance Criteria\n\n1. **Backward Compatibility**: All existing scenarios pass without modification\n2. **New Functionality**: Multi-turn conversation scenarios execute successfully\n3. **Evaluation Quality**: New evaluators provide meaningful insights\n4. **Performance**: No significant impact on existing scenario execution time\n5. **Documentation**: Clear examples and migration path provided", "createdAt": "2025-08-25T20:36:17Z", "closedAt": null, "state": "OPEN", "commentCount": 1 }, { "id": "I_kwDOMT5cIs7HyidA", "title": "Analyze options for MCP Gateway, Add some MCP servers, add x402 layer", "author": "borisudovicic", "number": 5814, "repository": "elizaos/eliza", "body": "[https://hackmd.io/aM2pd2NbSmOMzD0X9GPYiQ](https://hackmd.io/aM2pd2NbSmOMzD0X9GPYiQ)", "createdAt": "2025-08-25T13:40:42Z", "closedAt": null, "state": "OPEN", "commentCount": 0 }, { "id": "I_kwDOMT5cIs7Hx_mQ", "title": "i Successfully Publish a Plugin in 23AUG but it's not visible in PR", "author": "1BDO", "number": 5813, "repository": "elizaos/eliza", "body": "plugin-delta-0.1.0", "createdAt": "2025-08-25T13:01:43Z", "closedAt": null, "state": "OPEN", "commentCount": 0 } ], "topPRs": [ { "id": "PR_kwDOMT5cIs6lPozi", "title": "improve summary", "author": "tcm390", "number": 5818, "body": "", "repository": "elizaos/eliza", "createdAt": "2025-08-25T16:12:51Z", "mergedAt": "2025-08-25T16:12:57Z", "additions": 40, "deletions": 5 }, { "id": "PR_kwDOMT5cIs6lN-r7", "title": "Revert processActions change: use cacheState to retrieve action result instead; minor auth prompt fix", "author": "tcm390", "number": 5815, "body": "", "repository": "elizaos/eliza", "createdAt": "2025-08-25T13:45:13Z", "mergedAt": "2025-08-25T13:45:19Z", "additions": 19, "deletions": 21 }, { "id": "PR_kwDOMT5cIs6lOOXW", "title": "Refine prompt logic to enforce user ID requirement for actions needing authentication", "author": "tcm390", "number": 5816, "body": "", "repository": "elizaos/eliza", "createdAt": "2025-08-25T14:05:18Z", "mergedAt": "2025-08-25T14:05:25Z", "additions": 9, "deletions": 0 }, { "id": "PR_kwDOMT5cIs6lMYJc", "title": "Fix typo in runtime.ts comment: \"initalized\" → \"initialized\"", "author": "prestoalvarez", "number": 5812, "body": "\r\n\r\nFixes a typo in the comment on line 1615 of `packages/core/src/runtime.ts`.\r\n\r\n### Changes\r\n- **Before:** `// not initalized or registered yet, registerPlugin is already smart enough to`\r\n- **After:** `// not initialized or registered yet, registerPlugin is already smart enough to`\r\n\r\n", "repository": "elizaos/eliza", "createdAt": "2025-08-25T11:25:13Z", "mergedAt": "2025-08-25T13:52:17Z", "additions": 1, "deletions": 1 }, { "id": "PR_kwDOMT5cIs6lO2kQ", "title": "fix: Remove duplicate actionNames block from message handler template", "author": "tcm390", "number": 5817, "body": "The actionNames block was appearing twice in the messageHandlerTemplate: once correctly inside the section and once redundantly after it. This PR removes the duplicated block after the tag to streamline the template and prevent confusion.\r\n\r\n\"Screenshot\r\n\r\n\r\n\"Screenshot\r\n", "repository": "elizaos/eliza", "createdAt": "2025-08-25T14:57:11Z", "mergedAt": "2025-08-25T21:13:58Z", "additions": 0, "deletions": 6 } ], "codeChanges": { "additions": 69, "deletions": 33, "files": 6, "commitCount": 28 }, "completedItems": [ { "title": "improve summary", "prNumber": 5818, "type": "other", "body": "", "files": [ "packages/plugin-action-bench/src/actions/retail/exchangeDeliveredOrderItems.ts" ] }, { "title": "fix: Remove duplicate actionNames block from message handler template", "prNumber": 5817, "type": "bugfix", "body": "The actionNames block was appearing twice in the messageHandlerTemplate: once correctly inside the section and once redundantly after it. This PR removes the duplicated block after the tag to streamline the template", "files": [ "packages/core/src/__tests__/prompts.test.ts", "packages/core/src/prompts.ts" ] }, { "title": "Refine prompt logic to enforce user ID requirement for actions needing authentication", "prNumber": 5816, "type": "other", "body": "", "files": [ "packages/plugin-bootstrap/src/index.ts" ] }, { "title": "Revert processActions change: use cacheState to retrieve action result instead; minor auth prompt fix", "prNumber": 5815, "type": "bugfix", "body": "", "files": [ "packages/core/src/runtime.ts", "packages/core/src/types/runtime.ts", "packages/plugin-bootstrap/src/index.ts" ] }, { "title": "Fix typo in runtime.ts comment: \"initalized\" → \"initialized\"", "prNumber": 5812, "type": "bugfix", "body": "\r\n\r\nFixes a typo in the comment on line 1615 of `packages/core/src/runtime.ts`.\r\n\r\n### Changes\r\n- **Before:** `// not initalized or registered yet, registerPlugin is already smart enough to`\r\n- **After:** `// not initialized or registered y", "files": [ "packages/core/src/runtime.ts" ] } ], "topContributors": [ { "username": "tcm390", "avatarUrl": "https://avatars.githubusercontent.com/u/60634884?u=c6c41679b8322eaa0c81f72e0b4ed95e80f0ac16&v=4", "totalScore": 92.37288149385333, "prScore": 92.37288149385333, "issueScore": 0, "reviewScore": 0, "commentScore": 0, "summary": null }, { "username": "prestoalvarez", "avatarUrl": "https://avatars.githubusercontent.com/u/140459501?u=b843478cdfec2bca070ff99ebb65f8f6a8161aba&v=4", "totalScore": 21.419306144334055, "prScore": 21.419306144334055, "issueScore": 0, "reviewScore": 0, "commentScore": 0, "summary": null }, { "username": "linear", "avatarUrl": "https://avatars.githubusercontent.com/in/20150?v=4", "totalScore": 2, "prScore": 0, "issueScore": 2, "reviewScore": 0, "commentScore": 0, "summary": null }, { "username": "borisudovicic", "avatarUrl": "https://avatars.githubusercontent.com/u/31806472?u=27713fbe603baae91ef519990facbacd6c23e93d&v=4", "totalScore": 2, "prScore": 0, "issueScore": 2, "reviewScore": 0, "commentScore": 0, "summary": null }, { "username": "1BDO", "avatarUrl": "https://avatars.githubusercontent.com/u/210645034?v=4", "totalScore": 2, "prScore": 0, "issueScore": 2, "reviewScore": 0, "commentScore": 0, "summary": null }, { "username": "ChristopherTrimboli", "avatarUrl": "https://avatars.githubusercontent.com/u/27584221?u=0d816ce1dcdea8f925aba18bb710153d4a87a719&v=4", "totalScore": 0.33999999999999997, "prScore": 0, "issueScore": 0, "reviewScore": 0, "commentScore": 0.33999999999999997, "summary": null }, { "username": "monilpat", "avatarUrl": "https://avatars.githubusercontent.com/u/15067321?v=4", "totalScore": 0.2, "prScore": 0, "issueScore": 0, "reviewScore": 0, "commentScore": 0.2, "summary": null } ], "newPRs": 5, "mergedPRs": 5, "newIssues": 3, "closedIssues": 0, "activeContributors": 9 }