🤖 feat: OpenAI Assistants v2 (initial support) (#2781)

* 🤖 Assistants V2 Support: Part 1

- Separated Azure Assistants to its own endpoint
- File Search / Vector Store integration is incomplete, but can toggle and use storage from playground
- Code Interpreter resource files can be added but not deleted
- GPT-4o is supported
- Many improvements to the Assistants Endpoint overall

data-provider v2 changes

copy existing route as v1

chore: rename new endpoint to reduce comparison operations and add new azure filesource

api: add azureAssistants part 1

force use of version for assistants/assistantsAzure

chore: switch name back to azureAssistants

refactor type version: string | number

Ensure assistants endpoints have version set

fix: isArchived type issue in ConversationListParams

refactor: update assistants mutations/queries with endpoint/version definitions, update Assistants Map structure

chore:  FilePreview component ExtendedFile type assertion

feat: isAssistantsEndpoint helper

chore: remove unused useGenerations

chore(buildTree): type issue

chore(Advanced): type issue (unused component, maybe in future)

first pass for multi-assistant endpoint rewrite

fix(listAssistants): pass params correctly

feat: list separate assistants by endpoint

fix(useTextarea): access assistantMap correctly

fix: assistant endpoint switching, resetting ID

fix: broken during rewrite, selecting assistant mention

fix: set/invalidate assistants endpoint query data correctly

feat: Fix issue with assistant ID not being reset correctly

getOpenAIClient helper function

feat: add toast for assistant deletion

fix: assistants delete right after create issue for azure

fix: assistant patching

refactor: actions to use getOpenAIClient

refactor: consolidate logic into helpers file

fix: issue where conversation data was not initially available

v1 chat support

refactor(spendTokens): only early return if completionTokens isNaN

fix(OpenAIClient): ensure spendTokens has all necessary params

refactor: route/controller logic

fix(assistants/initializeClient): use defaultHeaders field

fix: sanitize default operation id

chore: bump openai package

first pass v2 action service

feat: retroactive domain parsing for actions added via v1

feat: delete db records of actions/assistants on openai assistant deletion

chore: remove vision tools from v2 assistants

feat: v2 upload and delete assistant vision images

WIP first pass, thread attachments

fix: show assistant vision files (save local/firebase copy)

v2 image continue

fix: annotations

fix: refine annotations

show analyze as error if is no longer submitting before progress reaches 1 and show file_search as retrieval tool

fix: abort run, undefined endpoint issue

refactor: consolidate capabilities logic and anticipate versioning

frontend version 2 changes

fix: query selection and filter

add endpoint to unknown filepath

add file ids to resource, deleting in progress

enable/disable file search

remove version log

* 🤖 Assistants V2 Support: Part 2

🎹 fix: Autocompletion Chrome Bug on Action API Key Input

chore: remove `useOriginNavigate`

chore: set correct OpenAI Storage Source

fix: azure file deletions, instantiate clients by source for deletion

update code interpret files info

feat: deleteResourceFileId

chore: increase poll interval as azure easily rate limits

fix: openai file deletions, TODO: evaluate rejected deletion settled promises to determine which to delete from db records

file source icons

update table file filters

chore: file search info and versioning

fix: retrieval update with necessary tool_resources if specified

fix(useMentions): add optional chaining in case listMap value is undefined

fix: force assistant avatar roundedness

fix: azure assistants, check correct flag

chore: bump data-provider

* fix: merge conflict

* ci: fix backend tests due to new updates

* chore: update .env.example

* meilisearch improvements

* localization updates

* chore: update comparisons

* feat: add additional metadata: endpoint, author ID

* chore: azureAssistants ENDPOINTS exclusion warning
This commit is contained in:
Danny Avila 2024-05-19 12:56:55 -04:00 committed by GitHub
parent af8bcb08d6
commit 1a452121fa
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
158 changed files with 4184 additions and 1204 deletions

View file

@ -3,8 +3,8 @@ import { useForm } from 'react-hook-form';
import { memo, useCallback, useRef, useMemo } from 'react';
import {
supportsFiles,
EModelEndpoint,
mergeFileConfig,
isAssistantsEndpoint,
fileConfig as defaultFileConfig,
} from 'librechat-data-provider';
import { useChatContext, useAssistantsMapContext } from '~/Providers';
@ -74,8 +74,9 @@ const ChatForm = ({ index = 0 }) => {
const endpointFileConfig = fileConfig.endpoints[endpoint ?? ''];
const invalidAssistant = useMemo(
() =>
conversation?.endpoint === EModelEndpoint.assistants &&
(!conversation?.assistant_id || !assistantMap?.[conversation?.assistant_id ?? '']),
isAssistantsEndpoint(conversation?.endpoint) &&
(!conversation?.assistant_id ||
!assistantMap?.[conversation?.endpoint ?? '']?.[conversation?.assistant_id ?? '']),
[conversation?.assistant_id, conversation?.endpoint, assistantMap],
);
const disableInputs = useMemo(

View file

@ -2,6 +2,7 @@ import type { TFile } from 'librechat-data-provider';
import type { ExtendedFile } from '~/common';
import FileIcon from '~/components/svg/Files/FileIcon';
import ProgressCircle from './ProgressCircle';
import SourceIcon from './SourceIcon';
import { useProgress } from '~/hooks';
import { cn } from '~/utils';
@ -20,8 +21,7 @@ const FilePreview = ({
}) => {
const radius = 55; // Radius of the SVG circle
const circumference = 2 * Math.PI * radius;
const progress = useProgress(file?.['progress'] ?? 1, 0.001, file?.size ?? 1);
console.log(progress);
const progress = useProgress(file?.['progress'] ?? 1, 0.001, (file as ExtendedFile)?.size ?? 1);
// Calculate the offset based on the loading progress
const offset = circumference - progress * circumference;
@ -32,6 +32,7 @@ const FilePreview = ({
return (
<div className={cn('h-10 w-10 shrink-0 overflow-hidden rounded-md', className)}>
<FileIcon file={file} fileType={fileType} />
<SourceIcon source={file?.source} />
{progress < 1 && (
<ProgressCircle
circumference={circumference}

View file

@ -1,4 +1,5 @@
import { useEffect } from 'react';
import { EToolResources } from 'librechat-data-provider';
import type { ExtendedFile } from '~/common';
import { useDeleteFilesMutation } from '~/data-provider';
import { useFileDeletion } from '~/hooks/Files';
@ -10,6 +11,7 @@ export default function FileRow({
setFiles,
setFilesLoading,
assistant_id,
tool_resource,
fileFilter,
Wrapper,
}: {
@ -18,6 +20,7 @@ export default function FileRow({
setFilesLoading: React.Dispatch<React.SetStateAction<boolean>>;
fileFilter?: (file: ExtendedFile) => boolean;
assistant_id?: string;
tool_resource?: EToolResources;
Wrapper?: React.FC<{ children: React.ReactNode }>;
}) {
const files = Array.from(_files.values()).filter((file) =>
@ -25,7 +28,8 @@ export default function FileRow({
);
const { mutateAsync } = useDeleteFilesMutation({
onMutate: async () => console.log('Deleting files: assistant_id', assistant_id),
onMutate: async () =>
console.log('Deleting files: assistant_id, tool_resource', assistant_id, tool_resource),
onSuccess: () => {
console.log('Files deleted');
},
@ -34,7 +38,7 @@ export default function FileRow({
},
});
const { deleteFile } = useFileDeletion({ mutateAsync, assistant_id });
const { deleteFile } = useFileDeletion({ mutateAsync, assistant_id, tool_resource });
useEffect(() => {
if (!files) {
@ -82,6 +86,7 @@ export default function FileRow({
url={file.preview}
onDelete={handleDelete}
progress={file.progress}
source={file.source}
/>
);
}

View file

@ -12,16 +12,9 @@ export default function Files({ open, onOpenChange }) {
const { data: files = [] } = useGetFiles<TFile[]>({
select: (files) =>
files.map((file) => {
if (file.source === FileSources.local || file.source === FileSources.openai) {
file.context = file.context ?? FileContext.unknown;
return file;
} else {
return {
...file,
context: file.context ?? FileContext.unknown,
source: FileSources.local,
};
}
file.context = file.context ?? FileContext.unknown;
file.filterSource = file.source === FileSources.firebase ? FileSources.local : file.source;
return file;
}),
});

View file

@ -1,3 +1,4 @@
import { FileSources } from 'librechat-data-provider';
import ImagePreview from './ImagePreview';
import RemoveFile from './RemoveFile';
@ -6,16 +7,18 @@ const Image = ({
url,
onDelete,
progress = 1,
source = FileSources.local,
}: {
imageBase64?: string;
url?: string;
onDelete: () => void;
progress: number; // between 0 and 1
source?: FileSources;
}) => {
return (
<div className="group relative inline-block text-sm text-black/70 dark:text-white/90">
<div className="relative overflow-hidden rounded-xl border border-gray-200 dark:border-gray-600">
<ImagePreview imageBase64={imageBase64} url={url} progress={progress} />
<ImagePreview source={source} imageBase64={imageBase64} url={url} progress={progress} />
</div>
<RemoveFile onRemove={onDelete} />
</div>

View file

@ -1,4 +1,6 @@
import { FileSources } from 'librechat-data-provider';
import ProgressCircle from './ProgressCircle';
import SourceIcon from './SourceIcon';
import { cn } from '~/utils';
type styleProps = {
@ -13,11 +15,13 @@ const ImagePreview = ({
url,
progress = 1,
className = '',
source,
}: {
imageBase64?: string;
url?: string;
progress?: number; // between 0 and 1
className?: string;
source?: FileSources;
}) => {
let style: styleProps = {
backgroundSize: 'cover',
@ -65,6 +69,7 @@ const ImagePreview = ({
circleCSSProperties={circleCSSProperties}
/>
)}
<SourceIcon source={source} />
</div>
);
};

View file

@ -0,0 +1,45 @@
import { EModelEndpoint, FileSources } from 'librechat-data-provider';
import { MinimalIcon } from '~/components/Endpoints';
import { cn } from '~/utils';
const sourceToEndpoint = {
[FileSources.openai]: EModelEndpoint.openAI,
[FileSources.azure]: EModelEndpoint.azureOpenAI,
};
const sourceToClassname = {
[FileSources.openai]: 'bg-black/65',
[FileSources.azure]: 'azure-bg-color opacity-85',
};
const defaultClassName =
'absolute right-0 bottom-0 rounded-full p-[0.15rem] text-gray-600 transition-colors';
export default function SourceIcon({
source,
className = defaultClassName,
}: {
source?: FileSources;
className?: string;
}) {
if (source === FileSources.local || source === FileSources.firebase) {
return null;
}
const endpoint = sourceToEndpoint[source ?? ''];
if (!endpoint) {
return null;
}
return (
<button type="button" className={cn(className, sourceToClassname[source ?? ''] ?? '')}>
<span className="flex items-center justify-center">
<MinimalIcon
endpoint={endpoint}
size={14}
isCreatedByUser={false}
iconClassName="h-3 w-3"
/>
</span>
</button>
);
}

View file

@ -7,6 +7,7 @@ import ImagePreview from '~/components/Chat/Input/Files/ImagePreview';
import FilePreview from '~/components/Chat/Input/Files/FilePreview';
import { SortFilterHeader } from './SortFilterHeader';
import { OpenAIMinimalIcon } from '~/components/svg';
import { AzureMinimalIcon } from '~/components/svg';
import { Button, Checkbox } from '~/components/ui';
import { formatDate, getFileType } from '~/utils';
import useLocalize from '~/hooks/useLocalize';
@ -71,10 +72,11 @@ export const columns: ColumnDef<TFile>[] = [
const file = row.original;
if (file.type?.startsWith('image')) {
return (
<div className="flex gap-2 ">
<div className="flex gap-2">
<ImagePreview
url={file.filepath}
className="h-10 w-10 shrink-0 overflow-hidden rounded-md"
className="relative h-10 w-10 shrink-0 overflow-hidden rounded-md"
source={file?.source}
/>
<span className="self-center truncate ">{file.filename}</span>
</div>
@ -84,7 +86,7 @@ export const columns: ColumnDef<TFile>[] = [
const fileType = getFileType(file.type);
return (
<div className="flex gap-2">
{fileType && <FilePreview fileType={fileType} />}
{fileType && <FilePreview fileType={fileType} className="relative" file={file} />}
<span className="self-center truncate">{file.filename}</span>
</div>
);
@ -108,7 +110,7 @@ export const columns: ColumnDef<TFile>[] = [
cell: ({ row }) => formatDate(row.original.updatedAt),
},
{
accessorKey: 'source',
accessorKey: 'filterSource',
header: ({ column }) => {
const localize = useLocalize();
return (
@ -117,10 +119,14 @@ export const columns: ColumnDef<TFile>[] = [
title={localize('com_ui_storage')}
filters={{
Storage: Object.values(FileSources).filter(
(value) => value === FileSources.local || value === FileSources.openai,
(value) =>
value === FileSources.local ||
value === FileSources.openai ||
value === FileSources.azure,
),
}}
valueMap={{
[FileSources.azure]: 'Azure',
[FileSources.openai]: 'OpenAI',
[FileSources.local]: 'com_ui_host',
}}
@ -137,6 +143,13 @@ export const columns: ColumnDef<TFile>[] = [
{'OpenAI'}
</div>
);
} else if (source === FileSources.azure) {
return (
<div className="flex flex-wrap items-center gap-2">
<AzureMinimalIcon className="icon-sm text-cyan-700" />
{'Azure'}
</div>
);
}
return (
<div className="flex flex-wrap items-center gap-2">

View file

@ -48,7 +48,12 @@ const contextMap = {
[FileContext.bytes]: 'com_ui_size',
};
type Style = { width?: number | string; maxWidth?: number | string; minWidth?: number | string };
type Style = {
width?: number | string;
maxWidth?: number | string;
minWidth?: number | string;
zIndex?: number;
};
export default function DataTable<TData, TValue>({ columns, data }: DataTableProps<TData, TValue>) {
const localize = useLocalize();
@ -142,7 +147,7 @@ export default function DataTable<TData, TValue>({ columns, data }: DataTablePro
{table.getHeaderGroups().map((headerGroup) => (
<TableRow key={headerGroup.id}>
{headerGroup.headers.map((header, index) => {
const style: Style = { maxWidth: '32px', minWidth: '125px' };
const style: Style = { maxWidth: '32px', minWidth: '125px', zIndex: 50 };
if (header.id === 'filename') {
style.maxWidth = '50%';
style.width = '50%';

View file

@ -17,7 +17,9 @@ export default function Mention({
}) {
const localize = useLocalize();
const assistantMap = useAssistantsMapContext();
const { options, modelsConfig, assistants, onSelectMention } = useMentions({ assistantMap });
const { options, modelsConfig, assistantListMap, onSelectMention } = useMentions({
assistantMap,
});
const [activeIndex, setActiveIndex] = useState(0);
const timeoutRef = useRef<NodeJS.Timeout | null>(null);
@ -47,7 +49,12 @@ export default function Mention({
if (mention.type === 'endpoint' && mention.value === EModelEndpoint.assistants) {
setSearchValue('');
setInputOptions(assistants);
setInputOptions(assistantListMap[EModelEndpoint.assistants]);
setActiveIndex(0);
inputRef.current?.focus();
} else if (mention.type === 'endpoint' && mention.value === EModelEndpoint.azureAssistants) {
setSearchValue('');
setInputOptions(assistantListMap[EModelEndpoint.azureAssistants]);
setActiveIndex(0);
inputRef.current?.focus();
} else if (mention.type === 'endpoint') {

View file

@ -1,4 +1,4 @@
import { EModelEndpoint } from 'librechat-data-provider';
import { EModelEndpoint, isAssistantsEndpoint } from 'librechat-data-provider';
import { useGetEndpointsQuery, useGetStartupConfig } from 'librechat-data-provider/react-query';
import type { ReactNode } from 'react';
import { TooltipProvider, Tooltip, TooltipTrigger, TooltipContent } from '~/components/ui';
@ -30,7 +30,8 @@ export default function Landing({ Header }: { Header?: ReactNode }) {
const iconURL = conversation?.iconURL;
endpoint = getIconEndpoint({ endpointsConfig, iconURL, endpoint });
const assistant = endpoint === EModelEndpoint.assistants && assistantMap?.[assistant_id ?? ''];
const isAssistant = isAssistantsEndpoint(endpoint);
const assistant = isAssistant && assistantMap?.[endpoint]?.[assistant_id ?? ''];
const assistantName = (assistant && assistant?.name) || '';
const assistantDesc = (assistant && assistant?.description) || '';
const avatar = (assistant && (assistant?.metadata?.avatar as string)) || '';
@ -77,7 +78,7 @@ export default function Landing({ Header }: { Header?: ReactNode }) {
</div>
) : (
<div className="mb-5 max-w-[75vh] px-12 text-center text-lg font-medium dark:text-white md:px-0 md:text-2xl">
{endpoint === EModelEndpoint.assistants
{isAssistant
? conversation?.greeting ?? localize('com_nav_welcome_assistant')
: conversation?.greeting ?? localize('com_nav_welcome_message')}
</div>

View file

@ -15,6 +15,24 @@ import {
import UnknownIcon from './UnknownIcon';
import { cn } from '~/utils';
const AssistantAvatar = ({ className = '', assistantName, avatar, size }: IconMapProps) => {
if (assistantName && avatar) {
return (
<img
src={avatar}
className="bg-token-surface-secondary dark:bg-token-surface-tertiary h-full w-full rounded-full object-cover"
alt={assistantName}
width="80"
height="80"
/>
);
} else if (assistantName) {
return <AssistantIcon className={cn('text-token-secondary', className)} size={size} />;
}
return <Sparkles className={cn(assistantName === '' ? 'icon-2xl' : '', className)} />;
};
export const icons = {
[EModelEndpoint.azureOpenAI]: AzureMinimalIcon,
[EModelEndpoint.openAI]: GPTIcon,
@ -24,22 +42,7 @@ export const icons = {
[EModelEndpoint.google]: GoogleMinimalIcon,
[EModelEndpoint.bingAI]: BingAIMinimalIcon,
[EModelEndpoint.custom]: CustomMinimalIcon,
[EModelEndpoint.assistants]: ({ className = '', assistantName, avatar, size }: IconMapProps) => {
if (assistantName && avatar) {
return (
<img
src={avatar}
className="bg-token-surface-secondary dark:bg-token-surface-tertiary h-full w-full rounded-full object-cover"
alt={assistantName}
width="80"
height="80"
/>
);
} else if (assistantName) {
return <AssistantIcon className={cn('text-token-secondary', className)} size={size} />;
}
return <Sparkles className={cn(assistantName === '' ? 'icon-2xl' : '', className)} />;
},
[EModelEndpoint.assistants]: AssistantAvatar,
[EModelEndpoint.azureAssistants]: AssistantAvatar,
unknown: UnknownIcon,
};

View file

@ -1,5 +1,5 @@
import { Content, Portal, Root } from '@radix-ui/react-popover';
import { alternateName, EModelEndpoint } from 'librechat-data-provider';
import { alternateName, isAssistantsEndpoint } from 'librechat-data-provider';
import { useGetEndpointsQuery } from 'librechat-data-provider/react-query';
import type { FC } from 'react';
import { useChatContext, useAssistantsMapContext } from '~/Providers';
@ -16,7 +16,8 @@ const EndpointsMenu: FC = () => {
const { endpoint = '', assistant_id = null } = conversation ?? {};
const assistantMap = useAssistantsMapContext();
const assistant = endpoint === EModelEndpoint.assistants && assistantMap?.[assistant_id ?? ''];
const assistant =
isAssistantsEndpoint(endpoint) && assistantMap?.[endpoint ?? '']?.[assistant_id ?? ''];
const assistantName = (assistant && assistant?.name) || 'Assistant';
if (!endpoint) {

View file

@ -1,6 +1,7 @@
import { useState } from 'react';
import { useRecoilValue } from 'recoil';
import ProgressCircle from './ProgressCircle';
import CancelledIcon from './CancelledIcon';
import ProgressText from './ProgressText';
import FinishedIcon from './FinishedIcon';
import MarkdownLite from './MarkdownLite';
@ -11,10 +12,12 @@ export default function CodeAnalyze({
initialProgress = 0.1,
code,
outputs = [],
isSubmitting,
}: {
initialProgress: number;
code: string;
outputs: Record<string, unknown>[];
isSubmitting: boolean;
}) {
const showCodeDefault = useRecoilValue(store.showCode);
const [showCode, setShowCode] = useState(showCodeDefault);
@ -35,7 +38,13 @@ export default function CodeAnalyze({
<div className="my-2.5 flex items-center gap-2.5">
<div className="relative h-5 w-5 shrink-0">
{progress < 1 ? (
<CodeInProgress offset={offset} circumference={circumference} radius={radius} />
<CodeInProgress
offset={offset}
radius={radius}
progress={progress}
isSubmitting={isSubmitting}
circumference={circumference}
/>
) : (
<FinishedIcon />
)}
@ -74,18 +83,25 @@ const CodeInProgress = ({
offset,
circumference,
radius,
isSubmitting,
progress,
}: {
progress: number;
offset: number;
circumference: number;
radius: number;
isSubmitting: boolean;
}) => {
if (progress < 1 && !isSubmitting) {
return <CancelledIcon />;
}
return (
<div
className="absolute left-0 top-0 flex h-full w-full items-center justify-center rounded-full bg-transparent text-white"
style={{ opacity: 1, transform: 'none' }}
data-projection-id="77"
>
<div className='absolute right-[1.5px] bottom-[1.5px]'>
<div className="absolute bottom-[1.5px] right-[1.5px]">
<svg
xmlns="http://www.w3.org/2000/svg"
xmlnsXlink="http://www.w3.org/1999/xlink"

View file

@ -79,11 +79,13 @@ export default function Part({
initialProgress={toolCall.progress ?? 0.1}
code={code_interpreter.input}
outputs={code_interpreter.outputs ?? []}
isSubmitting={isSubmitting}
/>
);
} else if (
part.type === ContentTypes.TOOL_CALL &&
part[ContentTypes.TOOL_CALL].type === ToolCallTypes.RETRIEVAL
(part[ContentTypes.TOOL_CALL].type === ToolCallTypes.RETRIEVAL ||
part[ContentTypes.TOOL_CALL].type === ToolCallTypes.FILE_SEARCH)
) {
const toolCall = part[ContentTypes.TOOL_CALL];
return <RetrievalCall initialProgress={toolCall.progress ?? 0.1} isSubmitting={isSubmitting} />;

View file

@ -1,5 +1,4 @@
import { useState } from 'react';
import { EModelEndpoint } from 'librechat-data-provider';
import type { TConversation, TMessage } from 'librechat-data-provider';
import { Clipboard, CheckMark, EditIcon, RegenerateIcon, ContinueIcon } from '~/components/svg';
import { useGenerationsByLatest, useLocalize } from '~/hooks';
@ -35,14 +34,19 @@ export default function HoverButtons({
const { endpoint: _endpoint, endpointType } = conversation ?? {};
const endpoint = endpointType ?? _endpoint;
const [isCopied, setIsCopied] = useState(false);
const { hideEditButton, regenerateEnabled, continueSupported, forkingSupported } =
useGenerationsByLatest({
isEditing,
isSubmitting,
message,
endpoint: endpoint ?? '',
latestMessage,
});
const {
hideEditButton,
regenerateEnabled,
continueSupported,
forkingSupported,
isEditableEndpoint,
} = useGenerationsByLatest({
isEditing,
isSubmitting,
message,
endpoint: endpoint ?? '',
latestMessage,
});
if (!conversation) {
return null;
}
@ -58,7 +62,7 @@ export default function HoverButtons({
return (
<div className="visible mt-0 flex justify-center gap-1 self-end text-gray-400 lg:justify-start">
{endpoint !== EModelEndpoint.assistants && (
{isEditableEndpoint && (
<button
className={cn(
'hover-button rounded-md p-1 text-gray-400 hover:text-gray-900 dark:text-gray-400/70 dark:hover:text-gray-200 disabled:dark:hover:text-gray-400 md:group-hover:visible md:group-[.final-completion]:visible',