<h1><spanclass="section-number">12. </span>Advanced searching - Django Database queries<aclass="headerlink"href="#advanced-searching-django-database-queries"title="Permalink to this headline">¶</a></h1>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>Learning about Django's queryset language is very useful once you start doing more advanced things
in Evennia. But it's not strictly needed out the box and can be a little overwhelming for a first
<p>The search functions and methods we used in the previous lesson are enough for most cases.
But sometimes you need to be more specific:</p>
<ulclass="simple">
<li><p>You want to find all <codeclass="docutils literal notranslate"><spanclass="pre">Characters</span></code> …</p></li>
<li><p>… who are in Rooms tagged as <codeclass="docutils literal notranslate"><spanclass="pre">moonlit</span></code> …</p></li>
<li><p>… <em>and</em> who has the Attribute <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> with a level higher than 2 …</p></li>
<li><p>… because they’ll should immediately transform to werewolves!</p></li>
</ul>
<p>In principle you could achieve this with the existing search functions combined with a lot of loops
and if statements. But for something non-standard like this, querying the database directly will be
much more efficient.</p>
<p>Evennia uses <aclass="reference external"href="https://www.djangoproject.com/">Django</a> to handle its connection to the database.
A <aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/">django queryset</a> represents
a database query. One can add querysets together to build ever-more complicated queries. Only when
you are trying to use the results of the queryset will it actually call the database.</p>
<p>The normal way to build a queryset is to define what class of entity you want to search by getting its
<codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> resource, and then call various methods on that. We’ve seen this one before:</p>
<p>This is now a queryset representing all instances of <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>. If <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> had a subclass <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> and we
<p>Note that <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> are different typeclasses. You won’t find any <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> instances in
the <codeclass="docutils literal notranslate"><spanclass="pre">all_weapon</span></code> result above, confusing as that may sound. To get instances of a Typeclass <em>and</em> the
instances of all its children classes you need to use <codeclass="docutils literal notranslate"><spanclass="pre">_family</span></code>:</p>
<p>This result now contains both <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> instances.</p>
<p>To limit your search by other criteria than the Typeclass you need to use <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>
<p>This is a queryset representing all objects having a <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> equal to <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>.
Since this is a queryset you can keep adding to it; this will act as an <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> condition.</p>
<p>From now on, the queryset is <em>evaluated</em> and we can’t keep adding more queries to it - we’d need to
create a new queryset if we wanted to find some other result. Other ways to evaluate the queryset is to
print it, convert it to a list with <codeclass="docutils literal notranslate"><spanclass="pre">list()</span></code> and otherwise try to access its results.</p>
<p>Note how we use <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code>. This is the actual names of these database fields. By convention
Evennia uses <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> in front of every database field. When you use the normal Evennia search helpers and objects
you can skip the <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> but here we are calling the database directly and need to use the ‘real’ names.</p>
<p>Here are the most commonly used methods to use with the <codeclass="docutils literal notranslate"><spanclass="pre">objects</span></code> managers:</p>
<ulclass="simple">
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code> - query for a listing of objects based on search criteria. Gives empty queryset if none
were found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get</span></code> - query for a single match - raises exception if none were found, or more than one was
found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all</span></code> - get all instances of the particular type.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">get</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">all</span></code>, but return entities of all subclasses as well.</p></li>
</ul>
<blockquote>
<div><p>All of Evennia search functions use querysets under the hood. The <codeclass="docutils literal notranslate"><spanclass="pre">evennia.search_*</span></code> functions actually
return querysets, which means you could in principle keep adding queries to their results as well.</p>
<h2><spanclass="section-number">12.1. </span>Queryset field lookups<aclass="headerlink"href="#queryset-field-lookups"title="Permalink to this headline">¶</a></h2>
<p>Above we found roses with exactly the <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code><codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>. This is an <em>exact</em> match that is <em>case sensitive</em>,
so it would not find <codeclass="docutils literal notranslate"><spanclass="pre">"Rose"</span></code>.</p>
<p>The Django field query language uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> in the same way as Python uses <codeclass="docutils literal notranslate"><spanclass="pre">.</span></code> to access resources. This
is because <codeclass="docutils literal notranslate"><spanclass="pre">.</span></code> is not allowed in a function keyword.</p>
<p>This will find all flowers whose name contains the string <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>, like <codeclass="docutils literal notranslate"><spanclass="pre">"roses"</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">"wild</span><spanclass="pre">rose"</span></code> etc. The
<codeclass="docutils literal notranslate"><spanclass="pre">i</span></code> in the beginning makes the search case-insensitive. Other useful variations to use
are <codeclass="docutils literal notranslate"><spanclass="pre">__istartswith</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__iendswith</span></code>. You can also use <codeclass="docutils literal notranslate"><spanclass="pre">__gt</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">__ge</span></code> for “greater-than”/“greater-or-equal-than”
comparisons (same for <codeclass="docutils literal notranslate"><spanclass="pre">__lt</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__le</span></code>). There is also <codeclass="docutils literal notranslate"><spanclass="pre">__in</span></code>:</p>
<p>One also uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> to access foreign objects like Tags. Let’s for example assume this is how we identify mages:</p>
<p>This looks at the <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> field on the <codeclass="docutils literal notranslate"><spanclass="pre">Vampire</span></code> and filters on the values of each tag’s
<codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_category</span></code> together.</p>
<p>For more field lookups, see the
<aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/#field-lookups">django docs</a> on the subject.</p>
<h2><spanclass="section-number">12.2. </span>Get that werewolf …<aclass="headerlink"href="#get-that-werewolf"title="Permalink to this headline">¶</a></h2>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>Note the way of writing this code. It would have been very hard to read if we just wrote it in
one long line. But since we wrapped it in `(...)` we can spread it out over multiple lines
<li><p><strong>Line 3</strong> - We want to find <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code>s, so we access <codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> on the <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code> typeclass.</p></li>
<li><p><strong>Line 4</strong> - We start to filter …</p></li>
<li><p><strong>Line 5</strong></p>
<ul>
<li><p>… by accessing the <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> field (usually this is a Room)</p></li>
<li><p>… and on that location, we get the value of <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> (this is a <em>many-to-many</em> database field
that we can treat like an object for this purpose; it references all Tags on the location)</p></li>
<li><p>… and from those <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code>, we looking for <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is “monlit” (non-case sensitive).</p></li>
</ul>
</li>
<li><p><strong>Line 6</strong> - … We also want only Characters with <codeclass="docutils literal notranslate"><spanclass="pre">Attributes</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is exactly <codeclass="docutils literal notranslate"><spanclass="pre">"lycantrophy"</span></code></p></li>
<li><p><strong>Line 7</strong> - … at the same time as the <codeclass="docutils literal notranslate"><spanclass="pre">Attribute</span></code>’s <codeclass="docutils literal notranslate"><spanclass="pre">db_value</span></code> is greater-than 2.</p></li>
</ul>
<p>Running this query makes our newly lycantrrophic Character appear in <codeclass="docutils literal notranslate"><spanclass="pre">will_transform</span></code>. Success!</p>
<div><p>Don’t confuse database fields with <aclass="reference internal"href="../../../Components/Attributes.html"><spanclass="doc std std-doc">Attributes</span></a> you set via <codeclass="docutils literal notranslate"><spanclass="pre">obj.db.attr</span><spanclass="pre">=</span><spanclass="pre">'foo'</span></code> or
<codeclass="docutils literal notranslate"><spanclass="pre">obj.attributes.add()</span></code>. Attributes are custom database entities <em>linked</em> to an object. They are not
separate fields <em>on</em> that object like <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> are.</p>
<p>All examples so far used <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> relations. The arguments to <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code> are added together with <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>
<p>For queries using <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code> we need Django’s
<aclass="reference external"href="https://docs.djangoproject.com/en/1.11/topics/db/queries/#complex-lookups-with-q-objects">Q object</a>. It is
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is an object that is created with the same arguments as <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, for example</p>
<p>You can then use this <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> instance as argument in a <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>:</p>
<p>The useful thing about <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is that these objects can be chained together with special symbols (bit operators):
<codeclass="docutils literal notranslate"><spanclass="pre">|</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">&</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>. A tilde <codeclass="docutils literal notranslate"><spanclass="pre">~</span></code> in front negates the expression inside the <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> and thus
works like <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code>.</p>
<p>Would get all Characters that are either named “Dalton” <em>or</em> which is <em>not</em> in prison. The result is a mix
of Daltons and non-prisoners.</p>
<p>Let us expand our original werewolf query. Not only do we want to find all Characters in a moonlit room
with a certain level of <codeclass="docutils literal notranslate"><spanclass="pre">lycanthrophy</span></code>. Now we also want the full moon to immediately transform people who were
recently bitten, even if their <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> level is not yet high enough (more dramatic this way!). Let’s say there is
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>These Python structures are internally converted to SQL, the native language of the database.
If you are familiar with SQL, these are many-to-many tables joined with `LEFT OUTER JOIN`,
which may lead to multiple merged rows combining the same object with different relations.
<p>This reads as “Find all Characters in a moonlit room that either has the Attribute <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> higher
than two <em>or</em> which has the Tag <codeclass="docutils literal notranslate"><spanclass="pre">recently_bitten</span></code>”. With an OR-query like this it’s possible to find the
same Character via different paths, so we add <codeclass="docutils literal notranslate"><spanclass="pre">.distinct()</span></code> at the end. This makes sure that there is only
<p>Above we get all rooms and then use <codeclass="docutils literal notranslate"><spanclass="pre">list.append()</span></code> to keep adding the right rooms
to an ever-growing list. This is <em>not</em> a good idea, once your database grows this will
be unnecessarily computing-intensive. The database is much more suitable for this.</p>
<p><em>Annotations</em> allow you to set a ‘variable’ inside the query that you can
then access from other parts of the query. Let’s do the same example as before directly in the database:</p>
<p><codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code> is a Django class for counting the number of things in the database.</p>
<p>Here we first create an annotation <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> of type <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. It creates an in-database function
that will count the number of results inside the database.</p>
<blockquote>
<div><p>Note the use of <codeclass="docutils literal notranslate"><spanclass="pre">location_set</span></code> in that <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. The <codeclass="docutils literal notranslate"><spanclass="pre">*_set</span></code> is a back-reference automatically created by
Django. In this case it allows you to find all objects that <em>has the current object as location</em>.</p>
</div></blockquote>
<p>Next we filter on this annotation, using the name <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> as something we can filter for. We
use <codeclass="docutils literal notranslate"><spanclass="pre">num_objects__gte=5</span></code> which means that <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> should be greater than 5. This is a little
harder to get one’s head around but much more efficient than lopping over all objects in Python.</p>
<p>Here we used <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> to create two in-query ‘variables’<codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">num_tags</span></code>. We then
directly use these results in the filter. Using <codeclass="docutils literal notranslate"><spanclass="pre">F()</span></code> allows for also the right-hand-side of the filter
condition to be calculated on the fly, completely within the database.</p>
<h2><spanclass="section-number">12.6. </span>Grouping and returning only certain properties<aclass="headerlink"href="#grouping-and-returning-only-certain-properties"title="Permalink to this headline">¶</a></h2>
<p>Suppose you used tags to mark someone belonging to an organization. Now you want to make a list and
need to get the membership count of every organization all at once.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code>, and <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> queryset methods are useful for this. Normally when
you run a <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, what you get back is a bunch of full typeclass instances, like roses or swords.
Using <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code> you can instead choose to only get back certain properties on objects.
The <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> method finally allows for sorting the results according to some criterion:</p>
<li><p>… has a tag of category “organization” on them</p></li>
<li><p>… along the way we count how many different Characters (each <codeclass="docutils literal notranslate"><spanclass="pre">id</span></code> is unique) we find for each organization
and store it in a ‘variable’<codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> using <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code></p></li>
<li><p>… we use this count to sort the result in descending order of <codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> (descending because there is a minus sign,
default is increasing order but we want the most popular organization to be first).</p></li>
<li><p>… and finally we make sure to only return exactly the properties we want, namely the name of the organization tag
and how many matches we found for that organization.</p></li>
</ul>
<p>The result queryset will be a list of tuples ordered in descending order by the number of matches,
