<pclass="last">You are reading an old version of the Evennia documentation. <ahref="https://www.evennia.com/docs/latest/index.html">The latest version is here</a></p>.
<h1>Large Language Model (“Chat-bot AI”) integration<aclass="headerlink"href="#large-language-model-chat-bot-ai-integration"title="Permalink to this headline">¶</a></h1>
<p>Contribution by Griatch 2023</p>
<p>This adds an LLMClient that allows Evennia to send prompts to a LLM server (Large Language Model, along the lines of ChatGPT). Example uses a local OSS LLM install. Included is an NPC you can chat with using a new <codeclass="docutils literal notranslate"><spanclass="pre">talk</span></code> command. The NPC will respond using the AI responses from the LLM server. All calls are asynchronous, so if the LLM is slow, Evennia is not affected.</p>
> talk villager Hello there friend, what's up?
You say (to villager): Hello there friend, what's up?
villager says (to You): Hello! Not much going on, really.
> talk villager Do you know where we are?
You say (to villager): Do you know where we are?
villager says (to You): We are in this strange place called 'Limbo'. Not much to do here.
</pre></div>
</div>
<sectionid="installation">
<h2>Installation<aclass="headerlink"href="#installation"title="Permalink to this headline">¶</a></h2>
<p>You need two components for this contrib - Evennia, and an LLM webserver that operates and provides an API to an LLM AI model.</p>
<sectionid="llm-server">
<h3>LLM Server<aclass="headerlink"href="#llm-server"title="Permalink to this headline">¶</a></h3>
<p>There are many LLM servers, but they can be pretty technical to install and set up. This contrib was tested with <aclass="reference external"href="https://github.com/oobabooga/text-generation-webui">text-generation-webui</a>. It has a lot of features while also being easy to install. |</p>
<olclass="simple">
<li><p><aclass="reference external"href="https://github.com/oobabooga/text-generation-webui#installation">Go to the Installation section</a> and grab the ‘one-click installer’ for your OS.</p></li>
<li><p>Unzip the files in a folder somewhere on your hard drive (you don’t have to put it next to your evennia stuff if you don’t want to).</p></li>
<li><p>In a terminal/console, <codeclass="docutils literal notranslate"><spanclass="pre">cd</span></code> into the folder and execute the source file in whatever way it’s done for your OS (like <codeclass="docutils literal notranslate"><spanclass="pre">source</span><spanclass="pre">start_linux.sh</span></code> for Linux, or <codeclass="docutils literal notranslate"><spanclass="pre">.\start_windows</span></code> for Windows). This is an installer that will fetch and install everything in a conda virtual environment. When asked, make sure to select your GPU (NVIDIA/AMD etc) if you have one, otherwise use CPU.</p></li>
<li><p>Once all is loaded, stop the server with <codeclass="docutils literal notranslate"><spanclass="pre">Ctrl-C</span></code> (or <codeclass="docutils literal notranslate"><spanclass="pre">Cmd-C</span></code>) and open the file <codeclass="docutils literal notranslate"><spanclass="pre">webui.py</span></code> (it’s one of the top files in the archive you unzipped). Find the text string <codeclass="docutils literal notranslate"><spanclass="pre">CMD_FLAGS</span><spanclass="pre">=</span><spanclass="pre">''</span></code> near the top and change this to <codeclass="docutils literal notranslate"><spanclass="pre">CMD_FLAGS</span><spanclass="pre">=</span><spanclass="pre">'--api'</span></code>. Then save and close. This makes the server activate its api automatically.</p></li>
<li><p>Now just run that server starting script (<codeclass="docutils literal notranslate"><spanclass="pre">start_linux.sh</span></code> etc) again. This is what you’ll use to start the LLM server henceforth.</p></li>
<li><p>Once the server is running, point your browser to <aclass="reference external"href="http://127.0.0.1:7860">http://127.0.0.1:7860</a> to see the running Text generation web ui running. If you turned on the API, you’ll find it’s now active on port 5000. This should not collide with default Evennia ports unless you changed something.</p></li>
<li><p>At this point you have the server and API, but it’s not actually running any Large-Language-Model (LLM) yet. In the web ui, go to the <codeclass="docutils literal notranslate"><spanclass="pre">models</span></code> tab and enter a github-style path in the <codeclass="docutils literal notranslate"><spanclass="pre">Download</span><spanclass="pre">custom</span><spanclass="pre">model</span><spanclass="pre">or</span><spanclass="pre">LoRA</span></code> field. To test so things work, enter <codeclass="docutils literal notranslate"><spanclass="pre">DeepPavlov/bart-base-en-persona-chat</span></code> and download. This is a small model (350 million parameters) so should be possible to run on most machines using only CPU. Update the models in the drop-down on the left and select it, then load it with the <codeclass="docutils literal notranslate"><spanclass="pre">Transformers</span></code> loader. It should load pretty quickly. If you want to load this every time, you can select the <codeclass="docutils literal notranslate"><spanclass="pre">Autoload</span><spanclass="pre">the</span><spanclass="pre">model</span></code> checkbox; otherwise you’ll need to select and load the model every time you start the LLM server.</p></li>
<li><p>To experiment, you can find thousands of other open-source text-generation LLM models on <aclass="reference external"href="https://huggingface.co/models?pipeline_tag=text-generation&sort=trending">huggingface.co/models</a>. Beware to not download a too huge model; your machine may not be able to load it! If you try large models, <em>don’t</em> set the <codeclass="docutils literal notranslate"><spanclass="pre">Autoload</span><spanclass="pre">the</span><spanclass="pre">model</span></code> checkbox, in case the model crashes your server on startup.</p></li>
</ol>
<p>For troubleshooting, you can look at the terminal output of the <codeclass="docutils literal notranslate"><spanclass="pre">text-generation-webui</span></code> server; it will show you the requests you do to it and also list any errors. See the text-generation-webui homepage for more details.</p>
</section>
<sectionid="evennia-config">
<h3>Evennia config<aclass="headerlink"href="#evennia-config"title="Permalink to this headline">¶</a></h3>
<p>To be able to talk to NPCs, import and add the <codeclass="docutils literal notranslate"><spanclass="pre">evennia.contrib.rpg.llm.llm_npc.CmdLLMTalk</span></code> to your default cmdset in <codeclass="docutils literal notranslate"><spanclass="pre">mygame/commands/default_cmdsets.py</span></code>:</p>
<divclass="highlight-py notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># in mygame/commands/default_cmdsets.py</span>
<p>See this <aclass="reference internal"href="../Howtos/Beginner-Tutorial/Part1/Beginner-Tutorial-Adding-Commands.html"><spanclass="doc std std-doc">the tutorial on adding commands</span></a> for more info.</p>
<p>The default LLM api config should work with the <codeclass="docutils literal notranslate"><spanclass="pre">text-generation-webui</span></code> LLM server running its API on port 5000. You can also customize it via settings (if a setting is not added, the default below is used):</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># in mygame/server/conf/settings.py</span>
<spanclass="s2">"max_new_tokens"</span><spanclass="p">:</span><spanclass="mi">250</span><spanclass="p">,</span><spanclass="c1"># set how many tokens are part of a response</span>
<spanclass="s2">"You are roleplaying as </span><spanclass="si">{name}</span><spanclass="s2">, a </span><spanclass="si">{desc}</span><spanclass="s2"> existing in </span><spanclass="si">{location}</span><spanclass="s2">. "</span>
<spanclass="s2">"Answer with short sentences. Only respond as </span><spanclass="si">{name}</span><spanclass="s2"> would. "</span>
<spanclass="s2">"From here on, the conversation between </span><spanclass="si">{name}</span><spanclass="s2"> and </span><spanclass="si">{character}</span><spanclass="s2"> begins."</span>
<spanclass="p">)</span>
</pre></div>
</div>
<p>Don’t forget to reload Evennia (<codeclass="docutils literal notranslate"><spanclass="pre">reload</span></code> in game, or <codeclass="docutils literal notranslate"><spanclass="pre">evennia</span><spanclass="pre">reload</span></code> from the terminal) if you make any changes.</p>
<p>It’s also important to note that the <codeclass="docutils literal notranslate"><spanclass="pre">PROMPT_PREFIX</span></code> needed by each model depends on how they were trained. There are a bunch of different formats. So you need to look into what should be used for each model you try. Report your findings!</p>
</section>
</section>
<sectionid="usage">
<h2>Usage<aclass="headerlink"href="#usage"title="Permalink to this headline">¶</a></h2>
<p>With the LLM server running and the new <codeclass="docutils literal notranslate"><spanclass="pre">talk</span></code> command added, create a new LLM-connected NPC and talk to it in-game.</p>
<p>The conversation will be echoed to everyone in the room. The NPC will show a thinking/pondering message if the server responds slower than 2 seconds (by default).</p>
</section>
<sectionid="primer-on-open-source-llm-models">
<h2>Primer on open-source LLM models<aclass="headerlink"href="#primer-on-open-source-llm-models"title="Permalink to this headline">¶</a></h2>
<p><aclass="reference external"href="https://huggingface.co/models?pipeline_tag=text-generation&sort=trending">Hugging Face</a> is becoming a sort of standard for downloading OSS models. In the <codeclass="docutils literal notranslate"><spanclass="pre">text</span><spanclass="pre">generation</span></code> category (which is what we want for chat bots), there are some 20k models to choose from (2023). Just to get you started, check out models by <aclass="reference external"href="https://huggingface.co/models?pipeline_tag=text-generation&sort=trending&search=TheBloke">TheBloke</a>. TheBloke has taken on ‘quantizing’ (lowering their resolution) models released by others for them to fit on consumer hardware. Models from TheBloke follows roughly this naming standard:</p>
<p>Here, <codeclass="docutils literal notranslate"><spanclass="pre">Llama-2</span></code> is a ‘base model’ released open-source by Meta for free (also commercial) use. A base model takes millions of dollars and a supercomputer to train from scratch. Then others “fine tune” that base model. The <codeclass="docutils literal notranslate"><spanclass="pre">StableBeluga</span></code> model is created by someone partly retraining the <codeclass="docutils literal notranslate"><spanclass="pre">Llama-2</span></code> to make it more focused in some particular area, like chatting in a particular style.</p>
<p>Models come in sizes, given as number of parameters they have, sort of how many ‘neurons’ they have in their brain. In the two examples above, the top one has <codeclass="docutils literal notranslate"><spanclass="pre">7B</span></code> - 7 billion parameters and the second <codeclass="docutils literal notranslate"><spanclass="pre">13B</span></code> - 13 billion. The small model we suggested to try during install is only <codeclass="docutils literal notranslate"><spanclass="pre">0.35B</span></code> by comparson.</p>
<p>Running these models in their base form would still not be possible to do without people like TheBloke “quantizing” them, basically reducing their precision. Quantiziation are given in byte precision. So if the original supercomputer version uses 32bit precision, the model you can actually run on your machine often only uses 8bit or 4bit resolution. The common wisdom seems to be that being able to run a model with more parameters at low resolution is better than a smaller one with a higher resolution.</p>
<p>You will see GPTQ or GGML endings to TheBloke’s quantized models. Simplified, GPTQ are the main quantized models. To run this model, you need to have a beefy enough GPU to be able to fit the entire model in VRAM. GGML, in contrast, allows you to offload some of the model to normal RAM and use your CPU intead. Since you probably have more RAM than VRAM, this means you can run much bigger models this way, but they will run much slower.</p>
<p>Moreover, you need additional memory space for the <em>context</em> of the model. If you are chatting, this would be the chat history. While this sounds like it would just be some text, the length of the context determines how much the AI must ‘keep in mind’ in order to draw conclusions. This is measured in ‘tokens’ (roughly parts of words). Common context length is 2048 tokens, and a model must be specifically trained to be able to handle longer contexts.</p>
<p>Here are some rough estimates of hardware requirements for the most common model sizes and 2048 token context. Use GPTQ models if you have enough VRAM on your GPU, otherwise use GMML models to also be able to put some or all data in RAM.</p>
<thclass="head"><p>approx VRAM or RAM needed (4bit / 8bit)</p></th>
</tr>
</thead>
<tbody>
<trclass="row-even"><td><p>3B</p></td>
<td><p>1.5 GB / 3 GB</p></td>
</tr>
<trclass="row-odd"><td><p>7B</p></td>
<td><p>3.5 GB / 7 GB</p></td>
</tr>
<trclass="row-even"><td><p>13B</p></td>
<td><p>7 GB/13 GB</p></td>
</tr>
<trclass="row-odd"><td><p>33B</p></td>
<td><p>14 GB / 33 GB</p></td>
</tr>
<trclass="row-even"><td><p>70B</p></td>
<td><p>35 GB / 70 GB</p></td>
</tr>
</tbody>
</table>
<p>The results from a 7B or even a 3B model can be astounding! But set your expectations. Current (2023) top of the line consumer gaming GPUs have 24GB or VRAM and can at most fit a 33B 4bit quantized model at full speed (GPTQ).</p>
<p>By comparison, Chat-GPT 3.5 is a 175B model. We don’t know how large Chat-GPT 4 is, but it may be up to 1700B. For this reason you may also consider paying a commercial provider to run the model for you, over an API. This is discussed a little later, but try running locally with a small model first to see everything worls.</p>
</section>
<sectionid="using-an-ai-cloud-service">
<h2>Using an AI cloud service<aclass="headerlink"href="#using-an-ai-cloud-service"title="Permalink to this headline">¶</a></h2>
<p>You could also call out to an external API, like OpenAI (chat-GPT) or Google. Most cloud-hosted services are commercial and costs money. But since they have the hardware to run bigger models (or their own, proprietary models), they may give better and faster results.</p>
<divclass="admonition warning">
<pclass="admonition-title">Warning</p>
<p>Calling an external API is currently untested, so report any findings. Since the Evennia Server (not the Portal) is doing the calling, you are recommended to put a proxy between you and the internet if you call out like this.</p>
</div>
<p>Here is an untested example of the Evennia setting for calling <aclass="reference external"href="https://platform.openai.com/docs/api-reference/completions">OpenAI’s v1/completions API</a>:</p>
<div><p>TODO: OpenAI’s more modern <aclass="reference external"href="https://platform.openai.com/docs/api-reference/chat">v1/chat/completions</a> api does currently not work out of the gate since it’s a bit more complex.</p>
</div></blockquote>
</section>
<sectionid="the-llmnpc-class">
<h2>The LLMNPC class<aclass="headerlink"href="#the-llmnpc-class"title="Permalink to this headline">¶</a></h2>
<p>The LLM-able NPC class has a new method <codeclass="docutils literal notranslate"><spanclass="pre">at_talked_to</span></code> which does the connection to the LLM server and responds. This is called by the new <codeclass="docutils literal notranslate"><spanclass="pre">talk</span></code> command. Note that all these calls are asynchronous, meaning a slow response will not block Evennia.</p>
<p>The NPC’s AI is controlled with a few extra properties and Attributes, most of which can be customized directly in-game by a builder.</p>
<sectionid="prompt-prefix">
<h3><codeclass="docutils literal notranslate"><spanclass="pre">prompt_prefix</span></code><aclass="headerlink"href="#prompt-prefix"title="Permalink to this headline">¶</a></h3>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">prompt_prefix</span></code> is very important. This will be added in front of your prompt and helps the AI know how to respond. Remember that an LLM model is basically an auto-complete mechaniss, so by providing examples and instructions in the prefix, you can help it respond in a better way.</p>
<p>The prefix string to use for a given NPC is looked up from one of these locations, in order:</p>
<ol>
<li><p>An Attribute <codeclass="docutils literal notranslate"><spanclass="pre">npc.db.chat_prefix</span></code> stored on the NPC (not set by default)</p></li>
<li><p>A property <codeclass="docutils literal notranslate"><spanclass="pre">chat_prefix</span></code> on the the LLMNPC class (set to <codeclass="docutils literal notranslate"><spanclass="pre">None</span></code> by default).</p></li>
<li><p>The <codeclass="docutils literal notranslate"><spanclass="pre">LLM_PROMPT_PREFIX</span></code> setting (unset by default)</p></li>
<li><p>If none of the above locations are set, the following default is used:</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>"You are roleplaying as {name}, a {desc} existing in {location}.
Answer with short sentences. Only respond as {name} would.
From here on, the conversation between {name} and {character} begins."
</pre></div>
</div>
</li>
</ol>
<p>Here, the formatting tag <codeclass="docutils literal notranslate"><spanclass="pre">{name}</span></code> is replaced with the NPCs’s name, <codeclass="docutils literal notranslate"><spanclass="pre">desc</span></code> by it’s description, the <codeclass="docutils literal notranslate"><spanclass="pre">location</span></code> by its current location’s name and <codeclass="docutils literal notranslate"><spanclass="pre">character</span></code> by the one talking to it. All names of characters are given by the <codeclass="docutils literal notranslate"><spanclass="pre">get_display_name(looker)</span></code> call, so this may be different
from person to person.</p>
<p>Depending on the model, it can be very important to extend the prefix both with more information about the character as well as communication examples. A lot of tweaking may be necessary before producing something remniscent of human speech.</p>
</section>
<sectionid="response-template">
<h3>Response template<aclass="headerlink"href="#response-template"title="Permalink to this headline">¶</a></h3>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">response_template</span></code> AttributeProperty defaults to being</p>
<p>following common <codeclass="docutils literal notranslate"><spanclass="pre">msg_contents</span></code><aclass="reference internal"href="../Components/FuncParser.html"><spanclass="doc std std-doc">FuncParser</span></a> syntax. The <codeclass="docutils literal notranslate"><spanclass="pre">character</span></code> string will be mapped to the one talking to the NPC and the <codeclass="docutils literal notranslate"><spanclass="pre">response</span></code> will be what is said by the NPC.</p>
</section>
<sectionid="memory">
<h3>Memory<aclass="headerlink"href="#memory"title="Permalink to this headline">¶</a></h3>
<p>The NPC remembers what has been said to it by each player. This memory will be included with the prompt to the LLM and helps it understand the context of the conversation. The length of this memory is given by the <codeclass="docutils literal notranslate"><spanclass="pre">max_chat_memory_size</span></code> AttributeProperty. Default is 25 messages. Once the memory is maximum is reached, older messages are forgotten. Memory is stored separately for each player talking to the NPC.</p>
</section>
<sectionid="thinking">
<h3>Thinking<aclass="headerlink"href="#thinking"title="Permalink to this headline">¶</a></h3>
<p>If the LLM server is slow to respond, the NPC will echo a random ‘thinking message’ to show it has not forgotten about you (something like “The villager ponders your words …”).</p>
<p>They are controlled by two <codeclass="docutils literal notranslate"><spanclass="pre">AttributeProperties</span></code> on the LLMNPC class:</p>
<ulclass="simple">
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">thinking_timeout</span></code>: How long, in seconds to wait before showing the message. Default is 2 seconds.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">thinking_messages</span></code>: A list of messages to randomly pick between. Each message string can contain <codeclass="docutils literal notranslate"><spanclass="pre">{name}</span></code>, which will be replaced by the NPCs name.</p></li>
</ul>
</section>
</section>
<sectionid="todo">
<h2>TODO<aclass="headerlink"href="#todo"title="Permalink to this headline">¶</a></h2>
<p>There is a lot of expansion potential with this contrib. Some ideas:</p>
<ulclass="simple">
<li><p>Easier support for different cloud LLM provider API structures.</p></li>
<li><p>More examples of useful prompts and suitable models for MUD use.</p></li>
</ul>
<hrclass="docutils"/>
<p><small>This document page is generated from <codeclass="docutils literal notranslate"><spanclass="pre">evennia/contrib/rpg/llm/README.md</span></code>. Changes to this
file will be overwritten, so edit that file rather than this one.</small></p>
<liclass="nav-item nav-item-this"><ahref="">Large Language Model (“Chat-bot AI”) integration</a></li>
</ul>
</div>
<divclass="admonition important">
<pclass="first admonition-title">Note</p>
<pclass="last">You are reading an old version of the Evennia documentation. <ahref="https://www.evennia.com/docs/latest/index.html">The latest version is here</a></p>.
</div>
<divclass="footer"role="contentinfo">
© Copyright 2023, The Evennia developer community.
Created using <ahref="https://www.sphinx-doc.org/">Sphinx</a> 3.2.1.
