🎞️ feat: OpenRouter Audio/Video File Upload Support (#11070)

* Added video upload support for OpenRouter

- Added VIDEO_URL content type to support video_url message format
- Implemented OpenRouter video encoding using base64 data URLs
- Extended encodeAndFormatVideos() to handle OpenRouter provider
- Updated UI to accept video uploads for OpenRouter (mp4, webm, mpeg, mov)
- Fixed case-sensitivity in provider detection for agents
- Made isDocumentSupportedProvider() and isOpenAILikeProvider() case-insensitive

Videos are now converted to data:video/mp4;base64,... format compatible
with OpenRouter's API requirements per their documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: change multimodal and google_multimodal to more transparent variable names of image_document and image_document_video_audio

(also google_multimodal doesn't apply as much since we are adding support for video and audio uploads for open router)

* fix: revert .toLowerCase change to isOpenAILikeProvider and isDocumentSupportedProvider which broke upload to provider detection for openAI endpoints

* wip: add audio support to openrouter

* fix: filetypes now properly parsed and sent rather than destructured mimetypes for openrouter

* refactor: Omit to Exclude for ESLint

* feat: update DragDropModal for new openrouter support

* fix: special case openrouter for lower case provider

(currently getting issues with the provider coming in as 'OpenRouter' and our enum being 'openrouter') This will probably require a larger refactor later to handle case insensitivity for all providers, but that will have to be thoroughly tested in its own isolated PR

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Dustin Healy <54083382+dustinhealy@users.noreply.github.com>

packages/api/src/files/encode/audio.ts

@@ -79,6 +79,21 @@ export async function encodeAndFormatAudios(
         mimeType: file.type,
         data: content,
       });
+    } else if (provider === Providers.OPENROUTER) {
+      // Extract format from filename extension (e.g., 'audio.mp3' -> 'mp3')
+      // OpenRouter expects format values like: wav, mp3, aiff, aac, ogg, flac, m4a, pcm16, pcm24
+      // Note: MIME types don't always match (e.g., 'audio/mpeg' is mp3, not mpeg), so that is why we are using the file extension instead
+      const format = file.filename.split('.').pop()?.toLowerCase();
+      if (!format) {
+        throw new Error(`Could not extract audio format from filename: ${file.filename}`);
+      }
+      result.audios.push({
+        type: 'input_audio',
+        input_audio: {
+          data: content,
+          format,
+        },
+      });
     }
 
     result.files.push(metadata);

packages/api/src/files/encode/video.ts

@@ -79,6 +79,13 @@ export async function encodeAndFormatVideos(
         mimeType: file.type,
         data: content,
       });
+    } else if (provider === Providers.OPENROUTER) {
+      result.videos.push({
+        type: 'video_url',
+        video_url: {
+          url: `data:${file.type};base64,${content}`,
+        },
+      });
     }
 
     result.files.push(metadata);

packages/api/src/types/files.ts

@@ -29,12 +29,25 @@ export interface AudioProcessingResult {
   bytes: number;
 }
 
+/** Google video block format */
+export interface GoogleVideoBlock {
+  type: 'media';
+  mimeType: string;
+  data: string;
+}
+
+/** OpenRouter video block format */
+export interface OpenRouterVideoBlock {
+  type: 'video_url';
+  video_url: {
+    url: string;
+  };
+}
+
+export type VideoBlock = GoogleVideoBlock | OpenRouterVideoBlock;
+
 export interface VideoResult {
-  videos: Array<{
-    type: string;
-    mimeType: string;
-    data: string;
-  }>;
+  videos: VideoBlock[];
   files: Array<{
     file_id?: string;
     temp_file_id?: string;
@@ -100,12 +113,26 @@ export interface DocumentResult {
   }>;
 }
 
-export interface AudioResult {
-  audios: Array<{
-    type: string;
-    mimeType: string;
+/** Google audio block format */
+export interface GoogleAudioBlock {
+  type: 'media';
+  mimeType: string;
+  data: string;
+}
+
+/** OpenRouter audio block format */
+export interface OpenRouterAudioBlock {
+  type: 'input_audio';
+  input_audio: {
     data: string;
-  }>;
+    format: string;
+  };
+}
+
+export type AudioBlock = GoogleAudioBlock | OpenRouterAudioBlock;
+
+export interface AudioResult {
+  audios: AudioBlock[];
   files: Array<{
     file_id?: string;
     temp_file_id?: string;