💫 feat: Config File & Custom Endpoints (#1474)

* WIP(backend/api): custom endpoint * WIP(frontend/client): custom endpoint * chore: adjust typedefs for configs * refactor: use data-provider for cache keys and rename enums and custom endpoint for better clarity and compatibility * feat: loadYaml utility * refactor: rename back to from and proof-of-concept for creating schemas from user-defined defaults * refactor: remove custom endpoint from default endpointsConfig as it will be exclusively managed by yaml config * refactor(EndpointController): rename variables for clarity * feat: initial load custom config * feat(server/utils): add simple `isUserProvided` helper * chore(types): update TConfig type * refactor: remove custom endpoint handling from model services as will be handled by config, modularize fetching of models * feat: loadCustomConfig, loadConfigEndpoints, loadConfigModels * chore: reorganize server init imports, invoke loadCustomConfig * refactor(loadConfigEndpoints/Models): return each custom endpoint as standalone endpoint * refactor(Endpoint/ModelController): spread config values after default (temporary) * chore(client): fix type issues * WIP: first pass for multiple custom endpoints - add endpointType to Conversation schema - add update zod schemas for both convo/presets to allow non-EModelEndpoint value as endpoint (also using type assertion) - use `endpointType` value as `endpoint` where mapping to type is necessary using this field - use custom defined `endpoint` value and not type for mapping to modelsConfig - misc: add return type to `getDefaultEndpoint` - in `useNewConvo`, add the endpointType if it wasn't already added to conversation - EndpointsMenu: use user-defined endpoint name as Title in menu - TODO: custom icon via custom config, change unknown to robot icon * refactor(parseConvo): pass args as an object and change where used accordingly; chore: comment out 'create schema' code * chore: remove unused availableModels field in TConfig type * refactor(parseCompactConvo): pass args as an object and change where used accordingly * feat: chat through custom endpoint * chore(message/convoSchemas): avoid saving empty arrays * fix(BaseClient/saveMessageToDatabase): save endpointType * refactor(ChatRoute): show Spinner if endpointsQuery or modelsQuery are still loading, which is apparent with slow fetching of models/remote config on first serve * fix(useConversation): assign endpointType if it's missing * fix(SaveAsPreset): pass real endpoint and endpointType when saving Preset) * chore: recorganize types order for TConfig, add `iconURL` * feat: custom endpoint icon support: - use UnknownIcon in all icon contexts - add mistral and openrouter as known endpoints, and add their icons - iconURL support * fix(presetSchema): move endpointType to default schema definitions shared between convoSchema and defaults * refactor(Settings/OpenAI): remove legacy `isOpenAI` flag * fix(OpenAIClient): do not invoke abortCompletion on completion error * feat: add responseSender/label support for custom endpoints: - use defaultModelLabel field in endpointOption - add model defaults for custom endpoints in `getResponseSender` - add `useGetSender` hook which uses EndpointsQuery to determine `defaultModelLabel` - include defaultModelLabel from endpointConfig in custom endpoint client options - pass `endpointType` to `getResponseSender` * feat(OpenAIClient): use custom options from config file * refactor: rename `defaultModelLabel` to `modelDisplayLabel` * refactor(data-provider): separate concerns from `schemas` into `parsers`, `config`, and fix imports elsewhere * feat: `iconURL` and extract environment variables from custom endpoint config values * feat: custom config validation via zod schema, rename and move to `./projectRoot/librechat.yaml` * docs: custom config docs and examples * fix(OpenAIClient/mistral): mistral does not allow singular system message, also add `useChatCompletion` flag to use openai-node for title completions * fix(custom/initializeClient): extract env var and use `isUserProvided` function * Update librechat.example.yaml * feat(InputWithLabel): add className props, and forwardRef * fix(streamResponse): handle error edge case where either messages or convos query throws an error * fix(useSSE): handle errorHandler edge cases where error response is and is not properly formatted from API, especially when a conversationId is not yet provided, which ensures stream is properly closed on error * feat: user_provided keys for custom endpoints * fix(config/endpointSchema): do not allow default endpoint values in custom endpoint `name` * feat(loadConfigModels): extract env variables and optimize fetching models * feat: support custom endpoint iconURL for messages and Nav * feat(OpenAIClient): add/dropParams support * docs: update docs with default params, add/dropParams, and notes to use config file instead of `OPENAI_REVERSE_PROXY` * docs: update docs with additional notes * feat(maxTokensMap): add mistral models (32k context) * docs: update openrouter notes * Update ai_setup.md * docs(custom_config): add table of contents and fix note about custom name * docs(custom_config): reorder ToC * Update custom_config.md * Add note about `max_tokens` field in custom_config.md
2026-01-08 03:28:51 +01:00 · 2024-01-03 09:22:48 -05:00 · 2024-01-03 09:22:48 -05:00 · 29473a72db
commit 29473a72db
parent 3f98f92d4c
100 changed files with 2146 additions and 627 deletions
--- a/client/src/components/Chat/Input/ChatForm.tsx
+++ b/client/src/components/Chat/Input/ChatForm.tsx
@ -30,6 +30,8 @@ export default function ChatForm({ index = 0 }) {
  };

  const { requiresKey } = useRequiresKey();
+  const { endpoint: _endpoint, endpointType } = conversation ?? { endpoint: null };
+  const endpoint = endpointType ?? _endpoint;

  return (
    <form
@ -49,9 +51,9 @@ export default function ChatForm({ index = 0 }) {
              onChange={(e: ChangeEvent<HTMLTextAreaElement>) => setText(e.target.value)}
              setText={setText}
              submitMessage={submitMessage}
-              endpoint={conversation?.endpoint}
+              endpoint={endpoint}
            />
-            <AttachFile endpoint={conversation?.endpoint ?? ''} disabled={requiresKey} />
+            <AttachFile endpoint={endpoint ?? ''} disabled={requiresKey} />
            {isSubmitting && showStopButton ? (
              <StopButton stop={handleStopGenerating} setShowStopButton={setShowStopButton} />
            ) : (
--- a/client/src/components/Chat/Input/HeaderOptions.tsx
+++ b/client/src/components/Chat/Input/HeaderOptions.tsx
@ -2,7 +2,8 @@ import { useRecoilState } from 'recoil';
 import { Settings2 } from 'lucide-react';
 import { Root, Anchor } from '@radix-ui/react-popover';
 import { useState, useEffect, useMemo } from 'react';
-import { tPresetSchema, EModelEndpoint } from 'librechat-data-provider';
+import { tPresetUpdateSchema, EModelEndpoint } from 'librechat-data-provider';
+import type { TPreset } from 'librechat-data-provider';
 import { EndpointSettings, SaveAsPresetDialog } from '~/components/Endpoints';
 import { ModelSelect } from '~/components/Input/ModelSelect';
 import { PluginStoreDialog } from '~/components';
@ -106,7 +107,11 @@ export default function OptionsBar() {
            <SaveAsPresetDialog
              open={saveAsDialogShow}
              onOpenChange={setSaveAsDialogShow}
-              preset={tPresetSchema.parse({ ...conversation })}
+              preset={
+                tPresetUpdateSchema.parse({
+                  ...conversation,
+                }) as TPreset
+              }
            />
            <PluginStoreDialog
              isOpen={showPluginStoreDialog}
--- a/client/src/components/Chat/Input/OptionsBar.tsx
+++ b/client/src/components/Chat/Input/OptionsBar.tsx
@ -1,7 +1,8 @@
 import { useRecoilState } from 'recoil';
 import { Settings2 } from 'lucide-react';
 import { useState, useEffect, useMemo } from 'react';
-import { tPresetSchema, EModelEndpoint } from 'librechat-data-provider';
+import { tPresetUpdateSchema, EModelEndpoint } from 'librechat-data-provider';
+import type { TPreset } from 'librechat-data-provider';
 import { PluginStoreDialog } from '~/components';
 import {
  EndpointSettings,
@ -24,14 +25,8 @@ export default function OptionsBar({ messagesTree }) {
    store.showPluginStoreDialog,
  );

-  const {
-    showPopover,
-    conversation,
-    latestMessage,
-    setShowPopover,
-    setShowBingToneSetting,
-    textareaHeight,
-  } = useChatContext();
+  const { showPopover, conversation, latestMessage, setShowPopover, setShowBingToneSetting } =
+    useChatContext();
  const { setOption } = useSetIndexOptions();

  const { endpoint, conversationId, jailbreak } = conversation ?? {};
@ -81,14 +76,7 @@ export default function OptionsBar({ messagesTree }) {
    ? altSettings[endpoint]
    : () => setShowPopover((prev) => !prev);
  return (
-    <div
-      className="absolute left-0 right-0 mx-auto mb-2 last:mb-2 md:mx-4 md:last:mb-6 lg:mx-auto lg:max-w-2xl xl:max-w-3xl"
-      style={{
-        // TODO: option to hide footer and handle this
-        // bottom: `${80 + (textareaHeight - 56)}px`, // without footer
-        bottom: `${85 + (textareaHeight - 56)}px`, // with footer
-      }}
-    >
+    <div className="absolute left-0 right-0 mx-auto mb-2 last:mb-2 md:mx-4 md:last:mb-6 lg:mx-auto lg:max-w-2xl xl:max-w-3xl">
      <GenerationButtons
        endpoint={endpoint}
        showPopover={showPopover}
@ -151,7 +139,7 @@ export default function OptionsBar({ messagesTree }) {
          visible={showPopover}
          saveAsPreset={saveAsPreset}
          closePopover={() => setShowPopover(false)}
-          PopoverButtons={<PopoverButtons endpoint={endpoint} />}
+          PopoverButtons={<PopoverButtons />}
        >
          <div className="px-4 py-4">
            <EndpointSettings
@ -164,7 +152,11 @@ export default function OptionsBar({ messagesTree }) {
        <SaveAsPresetDialog
          open={saveAsDialogShow}
          onOpenChange={setSaveAsDialogShow}
-          preset={tPresetSchema.parse({ ...conversation })}
+          preset={
+            tPresetUpdateSchema.parse({
+              ...conversation,
+            }) as TPreset
+          }
        />
        <PluginStoreDialog isOpen={showPluginStoreDialog} setIsOpen={setShowPluginStoreDialog} />
      </span>
--- a/client/src/components/Chat/Input/PopoverButtons.tsx
+++ b/client/src/components/Chat/Input/PopoverButtons.tsx
@ -27,7 +27,8 @@ export default function PopoverButtons({
    setShowAgentSettings,
  } = useChatContext();

-  const { model, endpoint } = conversation ?? {};
+  const { model, endpoint: _endpoint, endpointType } = conversation ?? {};
+  const endpoint = endpointType ?? _endpoint;
  const isGenerativeModel = model?.toLowerCase()?.includes('gemini');
  const isChatModel = !isGenerativeModel && model?.toLowerCase()?.includes('chat');
  const isTextModel = !isGenerativeModel && !isChatModel && /code|text/.test(model ?? '');
--- a/client/src/components/Chat/Landing.tsx
+++ b/client/src/components/Chat/Landing.tsx
@ -1,10 +1,12 @@
 import type { ReactNode } from 'react';
+import { useGetEndpointsQuery } from 'librechat-data-provider/react-query';
 import { EModelEndpoint } from 'librechat-data-provider';
 import { icons } from './Menus/Endpoints/Icons';
 import { useChatContext } from '~/Providers';
 import { useLocalize } from '~/hooks';

 export default function Landing({ Header }: { Header?: ReactNode }) {
+  const { data: endpointsConfig } = useGetEndpointsQuery();
  const { conversation } = useChatContext();
  const localize = useLocalize();
  let { endpoint } = conversation ?? {};
@ -16,13 +18,22 @@ export default function Landing({ Header }: { Header?: ReactNode }) {
  ) {
    endpoint = EModelEndpoint.openAI;
  }
+
+  const iconKey = endpointsConfig?.[endpoint ?? '']?.type ? 'unknown' : endpoint ?? 'unknown';
+
  return (
    <div className="relative h-full">
      <div className="absolute left-0 right-0">{Header && Header}</div>
      <div className="flex h-full flex-col items-center justify-center">
        <div className="mb-3 h-[72px] w-[72px]">
          <div className="gizmo-shadow-stroke relative flex h-full items-center justify-center rounded-full bg-white text-black">
-            {icons[endpoint ?? 'unknown']({ size: 41, className: 'h-2/3 w-2/3' })}
+            {icons[iconKey]({
+              size: 41,
+              context: 'landing',
+              className: 'h-2/3 w-2/3',
+              endpoint: endpoint as EModelEndpoint | string,
+              iconURL: endpointsConfig?.[endpoint ?? ''].iconURL,
+            })}
          </div>
        </div>
        <div className="mb-5 text-2xl font-medium dark:text-white">
--- a/client/src/components/Chat/Menus/Endpoints/Icons.tsx
+++ b/client/src/components/Chat/Menus/Endpoints/Icons.tsx
@ -6,8 +6,10 @@ import {
  AzureMinimalIcon,
  BingAIMinimalIcon,
  GoogleMinimalIcon,
+  CustomMinimalIcon,
  LightningIcon,
 } from '~/components/svg';
+import UnknownIcon from './UnknownIcon';
 import { cn } from '~/utils';

 export const icons = {
@ -18,6 +20,7 @@ export const icons = {
  [EModelEndpoint.chatGPTBrowser]: LightningIcon,
  [EModelEndpoint.google]: GoogleMinimalIcon,
  [EModelEndpoint.bingAI]: BingAIMinimalIcon,
+  [EModelEndpoint.custom]: CustomMinimalIcon,
  [EModelEndpoint.assistant]: ({ className = '' }) => (
    <svg
      width="24"
@ -39,5 +42,5 @@ export const icons = {
      ></path>
    </svg>
  ),
-  unknown: GPTIcon,
+  unknown: UnknownIcon,
 };
--- a/client/src/components/Chat/Menus/Endpoints/MenuItem.tsx
+++ b/client/src/components/Chat/Menus/Endpoints/MenuItem.tsx
@ -1,6 +1,7 @@
 import { useState } from 'react';
 import { Settings } from 'lucide-react';
 import { EModelEndpoint } from 'librechat-data-provider';
+import { useGetEndpointsQuery } from 'librechat-data-provider/react-query';
 import type { FC } from 'react';
 import { useLocalize, useUserKey } from '~/hooks';
 import { SetKeyDialog } from '~/components/Input/SetKeyDialog';
@ -26,7 +27,8 @@ const MenuItem: FC<MenuItemProps> = ({
  userProvidesKey,
  ...rest
 }) => {
-  const Icon = icons[endpoint] ?? icons.unknown;
+  const { data: endpointsConfig } = useGetEndpointsQuery();
+
  const [isDialogOpen, setDialogOpen] = useState(false);
  const { newConversation } = useChatContext();
  const { getExpiry } = useUserKey(endpoint);
@ -44,6 +46,10 @@ const MenuItem: FC<MenuItemProps> = ({
    }
  };

+  const endpointType = endpointsConfig?.[endpoint ?? '']?.type;
+  const iconKey = endpointType ? 'unknown' : endpoint ?? 'unknown';
+  const Icon = icons[iconKey];
+
  return (
    <>
      <div
@ -56,7 +62,15 @@ const MenuItem: FC<MenuItemProps> = ({
        <div className="flex grow items-center justify-between gap-2">
          <div>
            <div className="flex items-center gap-2">
-              {<Icon size={18} className="icon-md shrink-0 dark:text-white" />}
+              {
+                <Icon
+                  size={18}
+                  endpoint={endpoint}
+                  context={'menu-item'}
+                  className="icon-md shrink-0 dark:text-white"
+                  iconURL={endpointsConfig?.[endpoint ?? '']?.iconURL}
+                />
+              }
              <div>
                {title}
                <div className="text-token-text-tertiary">{description}</div>
@ -128,7 +142,13 @@ const MenuItem: FC<MenuItemProps> = ({
        </div>
      </div>
      {userProvidesKey && (
-        <SetKeyDialog open={isDialogOpen} onOpenChange={setDialogOpen} endpoint={endpoint} />
+        <SetKeyDialog
+          open={isDialogOpen}
+          endpoint={endpoint}
+          endpointType={endpointType}
+          onOpenChange={setDialogOpen}
+          userProvideURL={endpointsConfig?.[endpoint ?? '']?.userProvideURL}
+        />
      )}
    </>
  );
--- a/client/src/components/Chat/Menus/Endpoints/UnknownIcon.tsx
+++ b/client/src/components/Chat/Menus/Endpoints/UnknownIcon.tsx
@ -0,0 +1,36 @@
+import { EModelEndpoint, KnownEndpoints } from 'librechat-data-provider';
+import { CustomMinimalIcon } from '~/components/svg';
+
+export default function UnknownIcon({
+  className = '',
+  endpoint,
+  iconURL,
+  context,
+}: {
+  iconURL?: string;
+  className?: string;
+  endpoint: EModelEndpoint | string | null;
+  context?: 'landing' | 'menu-item' | 'nav' | 'message';
+}) {
+  if (!endpoint) {
+    return <CustomMinimalIcon className={className} />;
+  }
+
+  const currentEndpoint = endpoint.toLowerCase();
+
+  if (iconURL) {
+    return <img className={className} src={iconURL} alt={`${endpoint} Icon`} />;
+  } else if (currentEndpoint === KnownEndpoints.mistral) {
+    return (
+      <img
+        className={context === 'landing' ? '' : className}
+        src="/assets/mistral.png"
+        alt="Mistral AI Icon"
+      />
+    );
+  } else if (currentEndpoint === KnownEndpoints.openrouter) {
+    return <img className={className} src="/assets/openrouter.png" alt="OpenRouter Icon" />;
+  }
+
+  return <CustomMinimalIcon className={className} />;
+}
--- a/client/src/components/Chat/Menus/EndpointsMenu.tsx
+++ b/client/src/components/Chat/Menus/EndpointsMenu.tsx
@ -21,7 +21,7 @@ const EndpointsMenu: FC = () => {
  }
  return (
    <Root>
-      <TitleButton primaryText={(alternateName[selected] ?? '') + ' '} />
+      <TitleButton primaryText={(alternateName[selected] ?? selected ?? '') + ' '} />
      <Portal>
        <div
          style={{
--- a/client/src/components/Chat/Menus/Presets/EditPresetDialog.tsx
+++ b/client/src/components/Chat/Menus/Presets/EditPresetDialog.tsx
@ -77,7 +77,6 @@ const EditPresetDialog = ({
                    {'ㅤ'}
                  </Label>
                  <PopoverButtons
-                    endpoint={endpoint}
                    buttonClass="ml-0 w-full dark:bg-gray-700 dark:hover:bg-gray-800 p-2 h-[40px] justify-center mt-0"
                    iconClass="hidden lg:block w-4"
                  />
--- a/client/src/components/Chat/Menus/Presets/PresetItems.tsx
+++ b/client/src/components/Chat/Menus/Presets/PresetItems.tsx
@ -2,6 +2,7 @@ import { Trash2 } from 'lucide-react';
 import { useRecoilValue } from 'recoil';
 import { Close } from '@radix-ui/react-popover';
 import { Flipper, Flipped } from 'react-flip-toolkit';
+import { useGetEndpointsQuery } from 'librechat-data-provider/react-query';
 import type { FC } from 'react';
 import type { TPreset } from 'librechat-data-provider';
 import FileUpload from '~/components/Input/EndpointMenu/FileUpload';
@ -31,6 +32,7 @@ const PresetItems: FC<{
  clearAllPresets,
  onFileSelected,
 }) => {
+  const { data: endpointsConfig } = useGetEndpointsQuery();
  const defaultPreset = useRecoilValue(store.defaultPreset);
  const localize = useLocalize();
  return (
@ -93,6 +95,10 @@ const PresetItems: FC<{
              return null;
            }

+            const iconKey = endpointsConfig?.[preset.endpoint ?? '']?.type
+              ? 'unknown'
+              : preset.endpoint ?? 'unknown';
+
            return (
              <Close asChild key={`preset-${preset.presetId}`}>
                <div key={`preset-${preset.presetId}`}>
@ -103,8 +109,11 @@ const PresetItems: FC<{
                      title={getPresetTitle(preset)}
                      disableHover={true}
                      onClick={() => onSelectPreset(preset)}
-                      icon={icons[preset.endpoint ?? 'unknown']({
+                      icon={icons[iconKey]({
+                        context: 'menu-item',
+                        iconURL: endpointsConfig?.[preset.endpoint ?? ''].iconURL,
                        className: 'icon-md mr-1 dark:text-white',
+                        endpoint: preset.endpoint,
                      })}
                      selected={false}
                      data-testid={`preset-item-${preset}`}
--- a/client/src/components/Chat/Menus/PresetsMenu.tsx
+++ b/client/src/components/Chat/Menus/PresetsMenu.tsx
@ -3,6 +3,7 @@ import { BookCopy } from 'lucide-react';
 import { Content, Portal, Root, Trigger } from '@radix-ui/react-popover';
 import { EditPresetDialog, PresetItems } from './Presets';
 import { useLocalize, usePresets } from '~/hooks';
+import { useChatContext } from '~/Providers';
 import { cn } from '~/utils';

 const PresetsMenu: FC = () => {
@ -18,6 +19,7 @@ const PresetsMenu: FC = () => {
    submitPreset,
    exportPreset,
  } = usePresets();
+  const { preset } = useChatContext();

  const presets = presetsQuery.data || [];
  return (
@ -64,7 +66,7 @@ const PresetsMenu: FC = () => {
          </Content>
        </div>
      </Portal>
-      <EditPresetDialog submitPreset={submitPreset} exportPreset={exportPreset} />
+      {preset && <EditPresetDialog submitPreset={submitPreset} exportPreset={exportPreset} />}
    </Root>
  );
 };
--- a/client/src/components/Chat/Messages/Content/EditMessage.tsx
+++ b/client/src/components/Chat/Messages/Content/EditMessage.tsx
@ -19,7 +19,8 @@ const EditMessage = ({

  const textEditor = useRef<HTMLDivElement | null>(null);
  const { conversationId, parentMessageId, messageId } = message;
-  const { endpoint } = conversation ?? { endpoint: null };
+  const { endpoint: _endpoint, endpointType } = conversation ?? { endpoint: null };
+  const endpoint = endpointType ?? _endpoint;
  const updateMessageMutation = useUpdateMessageMutation(conversationId ?? '');
  const localize = useLocalize();

--- a/client/src/components/Chat/Messages/HoverButtons.tsx
+++ b/client/src/components/Chat/Messages/HoverButtons.tsx
@ -1,7 +1,7 @@
 import { useState } from 'react';
 import type { TConversation, TMessage } from 'librechat-data-provider';
 import { Clipboard, CheckMark, EditIcon, RegenerateIcon, ContinueIcon } from '~/components/svg';
-import { useGenerations, useLocalize } from '~/hooks';
+import { useGenerationsByLatest, useLocalize } from '~/hooks';
 import { cn } from '~/utils';

 type THoverButtons = {
@ -28,9 +28,10 @@ export default function HoverButtons({
  latestMessage,
 }: THoverButtons) {
  const localize = useLocalize();
-  const { endpoint } = conversation ?? {};
+  const { endpoint: _endpoint, endpointType } = conversation ?? {};
+  const endpoint = endpointType ?? _endpoint;
  const [isCopied, setIsCopied] = useState(false);
-  const { hideEditButton, regenerateEnabled, continueSupported } = useGenerations({
+  const { hideEditButton, regenerateEnabled, continueSupported } = useGenerationsByLatest({
    isEditing,
    isSubmitting,
    message,