🔄 refactor: Sequential Event Ordering in Redis Streaming Mode (#11650)

* chore: linting image context file

* refactor: Event Emission with Async Handling for Redis Ordering

- Updated emitEvent and related functions to be async, ensuring proper event ordering in Redis mode.
- Refactored multiple handlers to await emitEvent calls, improving reliability for streaming deltas.
- Enhanced GenerationJobManager to await chunk emissions, critical for maintaining sequential event delivery.
- Added tests to verify that events are delivered in strict order when using Redis, addressing previous issues with out-of-order messages.

* refactor: Clear Pending Buffers and Timeouts in RedisEventTransport

- Enhanced the cleanup process in RedisEventTransport by ensuring that pending messages and flush timeouts are cleared when the last subscriber unsubscribes.
- Updated the destroy method to also clear pending messages and flush timeouts for all streams, improving resource management and preventing memory leaks.

* refactor: Update Event Emission to Async for Improved Ordering

- Refactored GenerationJobManager and RedisEventTransport to make emitDone and emitError methods async, ensuring proper event ordering in Redis mode.
- Updated all relevant calls to await these methods, enhancing reliability in event delivery.
- Adjusted tests to verify that events are processed in the correct sequence, addressing previous issues with out-of-order messages.

* refactor: Adjust RedisEventTransport for 0-Indexed Sequence Handling

- Updated sequence handling in RedisEventTransport to be 0-indexed, ensuring consistency across event emissions and buffer management.
- Modified integration tests to reflect the new sequence logic, improving the accuracy of event processing and delivery order.
- Enhanced comments for clarity on sequence management and terminal event handling.

* chore: Add Redis dump file to .gitignore

- Included dump.rdb in .gitignore to prevent accidental commits of Redis database dumps, enhancing repository cleanliness and security.

* test: Increase wait times in RedisEventTransport integration tests for CI stability

- Adjusted wait times for subscription establishment and event propagation from 100ms and 200ms to 500ms to improve reliability in CI environments.
- Enhanced code readability by formatting promise resolution lines for better clarity.

api/server/controllers/agents/request.js

@@ -324,7 +324,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
             conversationId: conversation?.conversationId,
           });
 
-          GenerationJobManager.emitDone(streamId, finalEvent);
+          await GenerationJobManager.emitDone(streamId, finalEvent);
           GenerationJobManager.completeJob(streamId);
           await decrementPendingRequest(userId);
         } else {
@@ -344,7 +344,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
             conversationId: conversation?.conversationId,
           });
 
-          GenerationJobManager.emitDone(streamId, finalEvent);
+          await GenerationJobManager.emitDone(streamId, finalEvent);
           GenerationJobManager.completeJob(streamId, 'Request aborted');
           await decrementPendingRequest(userId);
         }
@@ -377,7 +377,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
           // abortJob already handled emitDone and completeJob
         } else {
           logger.error(`[ResumableAgentController] Generation error for ${streamId}:`, error);
-          GenerationJobManager.emitError(streamId, error.message || 'Generation failed');
+          await GenerationJobManager.emitError(streamId, error.message || 'Generation failed');
           GenerationJobManager.completeJob(streamId, error.message);
         }
 
@@ -406,7 +406,7 @@ const ResumableAgentController = async (req, res, next, initializeClient, addTit
       res.status(500).json({ error: error.message || 'Failed to start generation' });
     } else {
       // JSON already sent, emit error to stream so client can receive it
-      GenerationJobManager.emitError(streamId, error.message || 'Failed to start generation');
+      await GenerationJobManager.emitError(streamId, error.message || 'Failed to start generation');
     }
     GenerationJobManager.completeJob(streamId, error.message);
     await decrementPendingRequest(userId);