<h1><spanclass="section-number">12. </span>Advanced searching - Django Database queries<aclass="headerlink"href="#advanced-searching-django-database-queries"title="Permalink to this headline">¶</a></h1>
<p>The search functions and methods we used in the previous lesson are enough for most cases.
But sometimes you need to be more specific:</p>
<ulclass="simple">
<li><p>You want to find all <codeclass="docutils literal notranslate"><spanclass="pre">Characters</span></code> …</p></li>
<li><p>… who are in Rooms tagged as <codeclass="docutils literal notranslate"><spanclass="pre">moonlit</span></code> …</p></li>
<li><p>… <em>and</em> who has the Attribute <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> with a level higher than 2 …</p></li>
<p>In principle you could achieve this with the existing search functions combined with a lot of loops
and if statements. But for something non-standard like this, querying the database directly will be
much more efficient.</p>
<p>Evennia uses <aclass="reference external"href="https://www.djangoproject.com/">Django</a> to handle its connection to the database.
A <aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/">django queryset</a> represents
a database query. One can add querysets together to build ever-more complicated queries. Only when
you are trying to use the results of the queryset will it actually call the database.</p>
<p>The normal way to build a queryset is to define what class of entity you want to search by getting its
<codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> resource, and then call various methods on that. We’ve seen this one before:</p>
<p>This is now a queryset representing all instances of <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>. If <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> had a subclass <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> and we
<p>Note that <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> are <em>different</em> typeclasses. This means that you
won’t find any <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>-typeclassed results in <codeclass="docutils literal notranslate"><spanclass="pre">all_cannons</span></code>. Vice-versa, you
won’t find any <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code>-typeclassed results in <codeclass="docutils literal notranslate"><spanclass="pre">all_weapons</span></code>. This may not be
what you expect.</p>
<p>If you want to get all entities with typeclass <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code><em>as well</em> as all the
subclasses of <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>, such as <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code>, you need to use the <codeclass="docutils literal notranslate"><spanclass="pre">_family</span></code> type of
<p>This result now contains both <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> instances (and any other
entities whose typeclasses inherit at any distance from <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>, like <codeclass="docutils literal notranslate"><spanclass="pre">Musket</span></code> or
<p>To limit your search by other criteria than the Typeclass you need to use <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>
<p>This is a queryset representing all flowers having a <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> equal to <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>.
Since this is a queryset you can keep adding to it; this will act as an <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> condition.</p>
<p>From now on, the queryset is <em>evaluated</em> and we can’t keep adding more queries to it - we’d need to
create a new queryset if we wanted to find some other result. Other ways to evaluate the queryset is to
print it, convert it to a list with <codeclass="docutils literal notranslate"><spanclass="pre">list()</span></code> and otherwise try to access its results.</p>
<p>Note how we use <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code>. This is the actual names of these
database fields. By convention Evennia uses <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> in front of every database
field. When you use the normal Evennia search helpers and objects you can skip
the <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> but here we are calling the database directly and need to use the
‘real’ names.</p>
<asideclass="sidebar">
<pclass="sidebar-title">database fields</p>
<p>Each database table have only a few fields. For <codeclass="docutils literal notranslate"><spanclass="pre">Objects</span></code>, the most common ones
are <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_destination</span></code>. When accessing them they are
normally accessed just as <codeclass="docutils literal notranslate"><spanclass="pre">obj.key</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">obj.location</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">obj.destination</span></code>. You
only need to remember the <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> when using them in database queries. The object
description, <codeclass="docutils literal notranslate"><spanclass="pre">obj.db.desc</span></code> is not such a hard-coded field, but one of many
<p>Here are the most commonly used methods to use with the <codeclass="docutils literal notranslate"><spanclass="pre">objects</span></code> managers:</p>
<ulclass="simple">
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code> - query for a listing of objects based on search criteria. Gives empty queryset if none
were found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get</span></code> - query for a single match - raises exception if none were found, or more than one was
found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all</span></code> - get all instances of the particular type.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">get</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">all</span></code>, but return entities of all subclasses as well.</p></li>
</ul>
<blockquote>
<div><p>All of Evennia search functions use querysets under the hood. The <codeclass="docutils literal notranslate"><spanclass="pre">evennia.search_*</span></code> functions actually
return querysets, which means you could in principle keep adding queries to their results as well.</p>
<h2><spanclass="section-number">12.1. </span>Queryset field lookups<aclass="headerlink"href="#queryset-field-lookups"title="Permalink to this headline">¶</a></h2>
<p>Above we found roses with exactly the <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code><codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>. This is an <em>exact</em> match that is <em>case sensitive</em>,
so it would not find <codeclass="docutils literal notranslate"><spanclass="pre">"Rose"</span></code>.</p>
<p>The Django field query language uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> similarly to how Python uses <codeclass="docutils literal notranslate"><spanclass="pre">.</span></code> to access resources. This
<p>This will find all flowers whose name contains the string <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>, like <codeclass="docutils literal notranslate"><spanclass="pre">"roses"</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">"wild</span><spanclass="pre">rose"</span></code> etc. The
<codeclass="docutils literal notranslate"><spanclass="pre">i</span></code> in the beginning makes the search case-insensitive. Other useful variations to use
are <codeclass="docutils literal notranslate"><spanclass="pre">__istartswith</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__iendswith</span></code>. You can also use <codeclass="docutils literal notranslate"><spanclass="pre">__gt</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">__ge</span></code> for “greater-than”/“greater-or-equal-than”
comparisons (same for <codeclass="docutils literal notranslate"><spanclass="pre">__lt</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__le</span></code>). There is also <codeclass="docutils literal notranslate"><spanclass="pre">__in</span></code>:</p>
<p>One also uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> to access foreign objects like Tags. Let’s for example assume
<p>This looks at the <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> field on the <codeclass="docutils literal notranslate"><spanclass="pre">Vampire</span></code> and filters on the values of each tag’s
<codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_category</span></code> together.</p>
<p>For more field lookups, see the
<aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/#field-lookups">django docs</a> on the subject.</p>
<h2><spanclass="section-number">12.2. </span>Get that werewolf …<aclass="headerlink"href="#get-that-werewolf"title="Permalink to this headline">¶</a></h2>
<p>Note the way of writing this code. It would have been very hard to read if we
just wrote it in one long line. But since we wrapped it in <codeclass="docutils literal notranslate"><spanclass="pre">(...)</span></code> we can spread
it out over multiple lines without worrying about line breaks!</p>
<li><p>We want to find <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code>s, so we access <codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> on the <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code> typeclass.</p></li>
<li><p>… by accessing the <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> field (usually this is a Room)</p></li>
<li><p>… and on that location, we get the value of <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> (this is a <em>many-to-many</em> database field
that we can treat like an object for this purpose; it references all Tags on the location)</p></li>
<li><p>… and from those <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code>, we looking for <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is “monlit” (non-case sensitive).</p></li>
<li><p>… We also want only Characters with <codeclass="docutils literal notranslate"><spanclass="pre">Attributes</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is exactly <codeclass="docutils literal notranslate"><spanclass="pre">"lycantrophy"</span></code></p></li>
<li><p>… at the same time as the <codeclass="docutils literal notranslate"><spanclass="pre">Attribute</span></code>’s <codeclass="docutils literal notranslate"><spanclass="pre">db_value</span></code> is greater-than 2.</p></li>
<p>Running this query makes our newly lycantrophic Character appear in <codeclass="docutils literal notranslate"><spanclass="pre">will_transform</span></code> so we
<div><p>Don’t confuse database fields with <aclass="reference internal"href="../../../Components/Attributes.html"><spanclass="doc std std-doc">Attributes</span></a> you set via <codeclass="docutils literal notranslate"><spanclass="pre">obj.db.attr</span><spanclass="pre">=</span><spanclass="pre">'foo'</span></code> or
<codeclass="docutils literal notranslate"><spanclass="pre">obj.attributes.add()</span></code>. Attributes are custom database entities <em>linked</em> to an object. They are not
separate fields <em>on</em> that object like <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> are.</p>
<p>All examples so far used <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> relations. The arguments to <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code> are added together with <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>
<p>For queries using <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code> we need Django’s
<aclass="reference external"href="https://docs.djangoproject.com/en/1.11/topics/db/queries/#complex-lookups-with-q-objects">Q object</a>. It is
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is an object that is created with the same arguments as <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, for example</p>
<p>You can then use this <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> instance as argument in a <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>:</p>
<p>The useful thing about <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is that these objects can be chained together with special symbols (bit operators):
<codeclass="docutils literal notranslate"><spanclass="pre">|</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">&</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>. A tilde <codeclass="docutils literal notranslate"><spanclass="pre">~</span></code> in front negates the expression inside the <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> and thus
works like <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code>.</p>
<p>Let us expand our original werewolf query. Not only do we want to find all
Characters in a moonlit room with a certain level of <codeclass="docutils literal notranslate"><spanclass="pre">lycanthrophy</span></code>. Now we also
want the full moon to immediately transform people who were recently bitten,
even if their <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> level is not yet high enough (more dramatic this
way!). When you get bitten, you’ll get a Tag <codeclass="docutils literal notranslate"><spanclass="pre">recently_bitten</span></code> put on you to
<p>These Python structures are internally converted to SQL, the native language of
the database. If you are familiar with SQL, these are many-to-many tables
joined with <codeclass="docutils literal notranslate"><spanclass="pre">LEFT</span><spanclass="pre">OUTER</span><spanclass="pre">JOIN</span></code>, which may lead to multiple merged rows combining
<p><codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code> is a Django class for counting the number of things in the database.</p>
<p>Here we first create an annotation <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> of type <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. It creates an in-database function
that will count the number of results inside the database.</p>
<blockquote>
<div><p>Note the use of <codeclass="docutils literal notranslate"><spanclass="pre">location_set</span></code> in that <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. The <codeclass="docutils literal notranslate"><spanclass="pre">*_set</span></code> is a back-reference automatically created by
Django. In this case it allows you to find all objects that <em>has the current object as location</em>.</p>
<p>Next we filter on this annotation, using the name <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> as something we
can filter for. We use <codeclass="docutils literal notranslate"><spanclass="pre">num_objects__gte=5</span></code> which means that <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code>
should be greater than or equal to 5. This is a little harder to get one’s head
around but much more efficient than lopping over all objects in Python.</p>
<p>What if we wanted to compare two dynamic parameters against one another in a
query? For example, what if instead of having 5 or more objects, we only wanted
objects that had a bigger inventory than they had tags (silly example, but …)?
This can be with Django’s <aclass="reference external"href="https://docs.djangoproject.com/en/1.11/ref/models/expressions/#f-expressions">F objects</a>.
So-called F expressions allow you to do a query that looks at a value of each
<p>Here we used <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> to create two in-query ‘variables’<codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">num_tags</span></code>. We then
directly use these results in the filter. Using <codeclass="docutils literal notranslate"><spanclass="pre">F()</span></code> allows for also the right-hand-side of the filter
condition to be calculated on the fly, completely within the database.</p>
<h2><spanclass="section-number">12.6. </span>Grouping and returning only certain properties<aclass="headerlink"href="#grouping-and-returning-only-certain-properties"title="Permalink to this headline">¶</a></h2>
<p>Suppose you used tags to mark someone belonging to an organization. Now you want to make a list and
need to get the membership count of every organization all at once.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code>, and <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> queryset methods are useful for this. Normally when
you run a <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, what you get back is a bunch of full typeclass instances, like roses or swords.
Using <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code> you can instead choose to only get back certain properties on objects.
The <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> method finally allows for sorting the results according to some criterion:</p>
<li><p>… has a tag of category “organization” on them</p></li>
<li><p>… along the way we count how many different Characters (each <codeclass="docutils literal notranslate"><spanclass="pre">id</span></code> is unique) we find for each organization
and store it in a ‘variable’<codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> using <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code></p></li>
<li><p>… we use this count to sort the result in descending order of <codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> (descending because there is a minus sign,
default is increasing order but we want the most popular organization to be first).</p></li>
<li><p>… and finally we make sure to only return exactly the properties we want, namely the name of the organization tag
and how many matches we found for that organization.</p></li>
</ul>
<p>The result queryset will be a list of tuples ordered in descending order by the number of matches,
