🚀 feat: Assistants Streaming (#2159)

* chore: bump openai to 4.29.0 and npm audit fix

* chore: remove unnecessary stream field from ContentData

* feat: new enum and types for AssistantStreamEvent

* refactor(AssistantService): remove stream field and add conversationId to text ContentData
> - return `finalMessage` and `text` on run completion
> - move `processMessages` to services/Threads to avoid circular dependencies with new stream handling
> - refactor(processMessages/retrieveAndProcessFile): add new `client` field to differentiate new RunClient type

* WIP: new assistants stream handling

* chore: stores messages to StreamRunManager

* chore: add additional typedefs

* fix: pass req and openai to StreamRunManager

* fix(AssistantService): pass openai as client to `retrieveAndProcessFile`

* WIP: streaming tool i/o, handle in_progress and completed run steps

* feat(assistants): process required actions with streaming enabled

* chore: condense early return check for useSSE useEffect

* chore: remove unnecessary comments and only handle completed tool calls when not function

* feat: add TTL for assistants run abort cacheKey

* feat: abort stream runs

* fix(assistants): render streaming cursor

* fix(assistants): hide edit icon as functionality is not supported

* fix(textArea): handle pasting edge cases; first, when onChange events wouldn't fire; second, when textarea wouldn't resize

* chore: memoize Conversations

* chore(useTextarea): reverse args order

* fix: load default capabilities when an azure is configured to support assistants, but `assistants` endpoint is not configured

* fix(AssistantSelect): update form assistant model on assistant form select

* fix(actions): handle azure strict validation for function names to fix crud for actions

* chore: remove content data debug log as it fires in rapid succession

* feat: improve UX for assistant errors mid-request

* feat: add tool call localizations and replace any domain separators from azure action names

* refactor(chat): error out tool calls without outputs during handleError

* fix(ToolService): handle domain separators allowing Azure use of actions

* refactor(StreamRunManager): types and throw Error if tool submission fails

api/typedefs.js

@@ -8,6 +8,180 @@
  * @memberof typedefs
  */
 
+/**
+ * @exports AssistantStreamEvent
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent} AssistantStreamEvent
+ * @memberof typedefs
+ */
+
+/**
+ * @exports AssistantStream
+ * @typedef {AsyncIterable<AssistantStreamEvent>} AssistantStream
+ * @memberof typedefs
+ */
+
+/**
+ * @exports RunCreateAndStreamParams
+ * @typedef {import('openai').OpenAI.Beta.Threads.RunCreateAndStreamParams} RunCreateAndStreamParams
+ * @memberof typedefs
+ */
+
+/**
+ * @exports OpenAIRequestOptions
+ * @typedef {import('openai').OpenAI.RequestOptions} OpenAIRequestOptions
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadCreated
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadCreated} ThreadCreated
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunCreated
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunCreated} ThreadRunCreated
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunQueued
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunQueued} ThreadRunQueued
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunInProgress
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunInProgress} ThreadRunInProgress
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunRequiresAction
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunRequiresAction} ThreadRunRequiresAction
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunCompleted
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunCompleted} ThreadRunCompleted
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunFailed
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunFailed} ThreadRunFailed
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunCancelling
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunCancelling} ThreadRunCancelling
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunCancelled
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunCancelled} ThreadRunCancelled
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunExpired
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunExpired} ThreadRunExpired
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepCreated
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepCreated} ThreadRunStepCreated
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepInProgress
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepInProgress} ThreadRunStepInProgress
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepDelta
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepDelta} ThreadRunStepDelta
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepCompleted
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepCompleted} ThreadRunStepCompleted
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepFailed
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepFailed} ThreadRunStepFailed
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepCancelled
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepCancelled} ThreadRunStepCancelled
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadRunStepExpired
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadRunStepExpired} ThreadRunStepExpired
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadMessageCreated
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadMessageCreated} ThreadMessageCreated
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadMessageInProgress
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadMessageInProgress} ThreadMessageInProgress
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadMessageDelta
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadMessageDelta} ThreadMessageDelta
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadMessageCompleted
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadMessageCompleted} ThreadMessageCompleted
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ThreadMessageIncomplete
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ThreadMessageIncomplete} ThreadMessageIncomplete
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ErrorEvent
+ * @typedef {import('openai').default.Beta.AssistantStreamEvent.ErrorEvent} ErrorEvent
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ToolCallDeltaObject
+ * @typedef {import('openai').default.Beta.Threads.Runs.Steps.ToolCallDeltaObject} ToolCallDeltaObject
+ * @memberof typedefs
+ */
+
+/**
+ * @exports ToolCallDelta
+ * @typedef {import('openai').default.Beta.Threads.Runs.Steps.ToolCallDelta} ToolCallDelta
+ * @memberof typedefs
+ */
+
 /**
  * @exports Assistant
  * @typedef {import('librechat-data-provider').Assistant} Assistant
@@ -109,6 +283,18 @@
  * @memberof typedefs
  */
 
+/**
+ * @exports TMessageContentParts
+ * @typedef {import('librechat-data-provider').TMessageContentParts} TMessageContentParts
+ * @memberof typedefs
+ */
+
+/**
+ * @exports StreamContentData
+ * @typedef {import('librechat-data-provider').StreamContentData} StreamContentData
+ * @memberof typedefs
+ */
+
 /**
  * @exports ActionRequest
  * @typedef {import('librechat-data-provider').ActionRequest} ActionRequest
@@ -698,6 +884,8 @@
  * @property {Run} run - The detailed information about the run.
  * @property {RunStep[]} steps - An array of steps taken during the run.
  * @property {StepMessage[]} messages - An array of messages related to the run.
+ * @property {ResponseMessage} finalMessage - The final response message, with all content parts.
+ * @property {string} text - The final response text, accumulated from message parts
  * @memberof typedefs
  */
 
@@ -726,7 +914,7 @@
 // */
 
 /**
- * @typedef {Object} OpenAIClientType
+ * @typedef {Object} RunClient
  *
  * @property {Express.Request} req - The Express request object.
  * @property {Express.Response} res - The Express response object.
@@ -754,7 +942,7 @@
  *
  * @property {ResponseMessage} responseMessage - A message object for responses.
  *
- * @typedef {OpenAI & OpenAIClientType} OpenAIClient
+ * @typedef {OpenAI & RunClient} OpenAIClient
  */
 
 /**
@@ -773,3 +961,50 @@
  * @property {Object} [metadata] - Optional. Metadata for the run.
  * @memberof typedefs
  */
+
+/**
+ * @typedef {Object} StreamRunManager
+ * Manages streaming and processing of run steps, messages, and tool calls within a thread.
+ *
+ * @property {number} index - Tracks the current index for step or message processing.
+ * @property {Map<string, any>} steps - Stores run steps by their IDs.
+ * @property {Map<string, number>} mappedOrder - Maps step or message IDs to their processing order index.
+ * @property {Map<number, any>} orderedRunSteps - Stores run steps in order of processing.
+ * @property {Set<string>} processedFileIds - Keeps track of file IDs that have been processed.
+ * @property {Map<string, Function>} progressCallbacks - Stores callbacks for reporting progress on step or message processing.
+ * @property {boolean} submittedToolOutputs - Indicates whether tool outputs have been submitted.
+ * @property {Object|null} run - Holds the current run object.
+ * @property {Object} req - The HTTP request object associated with the run.
+ * @property {Object} res - The HTTP response object for sending back data.
+ * @property {Object} openai - The OpenAI client instance.
+ * @property {string} apiKey - The API key used for OpenAI requests.
+ * @property {string} thread_id - The ID of the thread associated with the run.
+ * @property {Object} initialRunBody - The initial body of the run request.
+ * @property {Object.<string, Function>} clientHandlers - Custom handlers provided by the client.
+ * @property {Object} streamOptions - Options for streaming the run.
+ * @property {Object} finalMessage - The final message object to be constructed and sent.
+ * @property {Array} messages - An array of messages processed during the run.
+ * @property {string} text - Accumulated text from text content data.
+ * @property {Object.<string, Function>} handlers - Internal event handlers for different types of streaming events.
+ *
+ * @method addContentData Adds content data to the final message or sends it immediately depending on type.
+ * @method runAssistant Initializes and manages the streaming of a thread run.
+ * @method handleEvent Dispatches streaming events to the appropriate handlers.
+ * @method handleThreadCreated Handles the event when a thread is created.
+ * @method handleRunEvent Handles various run state events.
+ * @method handleRunStepEvent Handles events related to individual run steps.
+ * @method handleCodeImageOutput Processes and handles code-generated image outputs.
+ * @method createToolCallStream Initializes streaming for tool call outputs.
+ * @method handleNewToolCall Handles the creation of a new tool call within a run step.
+ * @method handleCompletedToolCall Handles the completion of tool call processing.
+ * @method handleRunStepDeltaEvent Handles updates (deltas) for run steps.
+ * @method handleMessageDeltaEvent Handles updates (deltas) for messages.
+ * @method handleErrorEvent Handles error events during streaming.
+ * @method getStepIndex Retrieves or assigns an index for a given step or message key.
+ * @method generateToolCallKey Generates a unique key for a tool call within a step.
+ * @method onRunRequiresAction Handles actions required by a run to proceed.
+ * @method onRunStepCreated Handles the creation of a new run step.
+ * @method onRunStepCompleted Handles the completion of a run step.
+ * @method handleMessageEvent Handles events related to messages within the run.
+ * @method messageCompleted Handles the completion of a message processing.
+ */