<h1><spanclass="section-number">12. </span>Advanced searching - Django Database queries<aclass="headerlink"href="#advanced-searching-django-database-queries"title="Permalink to this headline">¶</a></h1>
<divclass="admonition important">
<pclass="admonition-title">Important</p>
<p>More advanced lesson!</p>
<p>Learning about Django’s query language is very useful once you start doing more
advanced things in Evennia. But it’s not strictly needed out the box and can be
a little overwhelming for a first reading. So if you are new to Python and
Evennia, feel free to just skim this lesson and refer back to it later when
you’ve gained more experience.</p>
</div>
<p>The search functions and methods we used in the previous lesson are enough for most cases.
But sometimes you need to be more specific:</p>
<ulclass="simple">
<li><p>You want to find all <codeclass="docutils literal notranslate"><spanclass="pre">Characters</span></code> …</p></li>
<li><p>… who are in Rooms tagged as <codeclass="docutils literal notranslate"><spanclass="pre">moonlit</span></code> …</p></li>
<li><p>… <em>and</em> who has the Attribute <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> with a level higher than 2 …</p></li>
<li><p>… because they should immediately transform to werewolves!</p></li>
</ul>
<p>In principle you could achieve this with the existing search functions combined with a lot of loops
and if statements. But for something non-standard like this, querying the database directly will be
much more efficient.</p>
<p>Evennia uses <aclass="reference external"href="https://www.djangoproject.com/">Django</a> to handle its connection to the database.
A <aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/">django queryset</a> represents
a database query. One can add querysets together to build ever-more complicated queries. Only when
you are trying to use the results of the queryset will it actually call the database.</p>
<p>The normal way to build a queryset is to define what class of entity you want to search by getting its
<codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> resource, and then call various methods on that. We’ve seen this one before:</p>
<p>This is now a queryset representing all instances of <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>. If <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> had a subclass <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> and we
<p>Note that <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> are <em>different</em> typeclasses. This means that you
won’t find any <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>-typeclassed results in <codeclass="docutils literal notranslate"><spanclass="pre">all_cannons</span></code>. Vice-versa, you
won’t find any <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code>-typeclassed results in <codeclass="docutils literal notranslate"><spanclass="pre">all_weapons</span></code>. This may not be
what you expect.</p>
<p>If you want to get all entities with typeclass <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code><em>as well</em> as all the
subclasses of <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>, such as <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code>, you need to use the <codeclass="docutils literal notranslate"><spanclass="pre">_family</span></code> type of
query:</p>
<asideclass="sidebar">
<pclass="sidebar-title">_family</p>
<p>The all_family, filter_family etc is an Evennia-specific
<p>This result now contains both <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Cannon</span></code> instances (and any other
entities whose typeclasses inherit at any distance from <codeclass="docutils literal notranslate"><spanclass="pre">Weapon</span></code>, like <codeclass="docutils literal notranslate"><spanclass="pre">Musket</span></code> or
<p>To limit your search by other criteria than the Typeclass you need to use <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>
<p>This is a queryset representing all flowers having a <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> equal to <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>.
Since this is a queryset you can keep adding to it; this will act as an <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> condition.</p>
<p>It’s important to note that we haven’t called the database yet! Not until we
actually try to examine the result will the database be called. Here the
database is called when we try to loop over it (because now we need to actually
get results out of it to be able to loop):</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>for rose in local_non_red_roses:
print(rose)
</pre></div>
</div>
<p>From now on, the queryset is <em>evaluated</em> and we can’t keep adding more queries to it - we’d need to
create a new queryset if we wanted to find some other result. Other ways to evaluate the queryset is to
print it, convert it to a list with <codeclass="docutils literal notranslate"><spanclass="pre">list()</span></code> and otherwise try to access its results.</p>
<p>Note how we use <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code>. This is the actual names of these
database fields. By convention Evennia uses <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> in front of every database
field. When you use the normal Evennia search helpers and objects you can skip
the <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> but here we are calling the database directly and need to use the
‘real’ names.</p>
<asideclass="sidebar">
<pclass="sidebar-title">database fields</p>
<p>Each database table have only a few fields. For <codeclass="docutils literal notranslate"><spanclass="pre">Objects</span></code>, the most common ones
are <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_destination</span></code>. When accessing them they are
normally accessed just as <codeclass="docutils literal notranslate"><spanclass="pre">obj.key</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">obj.location</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">obj.destination</span></code>. You
only need to remember the <codeclass="docutils literal notranslate"><spanclass="pre">db_</span></code> when using them in database queries. The object
description, <codeclass="docutils literal notranslate"><spanclass="pre">obj.db.desc</span></code> is not such a hard-coded field, but one of many
arbitrary Attributes attached to the Object.</p>
</aside>
<p>Here are the most commonly used methods to use with the <codeclass="docutils literal notranslate"><spanclass="pre">objects</span></code> managers:</p>
<ulclass="simple">
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code> - query for a listing of objects based on search criteria. Gives empty queryset if none
were found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get</span></code> - query for a single match - raises exception if none were found, or more than one was
found.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all</span></code> - get all instances of the particular type.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">filter_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">get_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">get</span></code>, but search all sub classes as well.</p></li>
<li><p><codeclass="docutils literal notranslate"><spanclass="pre">all_family</span></code> - like <codeclass="docutils literal notranslate"><spanclass="pre">all</span></code>, but return entities of all subclasses as well.</p></li>
</ul>
<blockquote>
<div><p>All of Evennia search functions use querysets under the hood. The <codeclass="docutils literal notranslate"><spanclass="pre">evennia.search_*</span></code> functions actually
return querysets, which means you could in principle keep adding queries to their results as well.</p>
</div></blockquote>
<sectionid="queryset-field-lookups">
<h2><spanclass="section-number">12.1. </span>Queryset field lookups<aclass="headerlink"href="#queryset-field-lookups"title="Permalink to this headline">¶</a></h2>
<p>Above we found roses with exactly the <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code><codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>. This is an <em>exact</em> match that is <em>case sensitive</em>,
so it would not find <codeclass="docutils literal notranslate"><spanclass="pre">"Rose"</span></code>.</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span># this is case-sensitive and the same as =
<p>The Django field query language uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> similarly to how Python uses <codeclass="docutils literal notranslate"><spanclass="pre">.</span></code> to access resources. This
is because <codeclass="docutils literal notranslate"><spanclass="pre">.</span></code> is not allowed in a function keyword.</p>
<p>This will find all flowers whose name contains the string <codeclass="docutils literal notranslate"><spanclass="pre">"rose"</span></code>, like <codeclass="docutils literal notranslate"><spanclass="pre">"roses"</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">"wild</span><spanclass="pre">rose"</span></code> etc. The
<codeclass="docutils literal notranslate"><spanclass="pre">i</span></code> in the beginning makes the search case-insensitive. Other useful variations to use
are <codeclass="docutils literal notranslate"><spanclass="pre">__istartswith</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__iendswith</span></code>. You can also use <codeclass="docutils literal notranslate"><spanclass="pre">__gt</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">__ge</span></code> for “greater-than”/“greater-or-equal-than”
comparisons (same for <codeclass="docutils literal notranslate"><spanclass="pre">__lt</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">__le</span></code>). There is also <codeclass="docutils literal notranslate"><spanclass="pre">__in</span></code>:</p>
<p>One also uses <codeclass="docutils literal notranslate"><spanclass="pre">__</span></code> to access foreign objects like Tags. Let’s for example assume
<p>This looks at the <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> field on the <codeclass="docutils literal notranslate"><spanclass="pre">Vampire</span></code> and filters on the values of each tag’s
<codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">db_category</span></code> together.</p>
<p>For more field lookups, see the
<aclass="reference external"href="https://docs.djangoproject.com/en/3.0/ref/models/querysets/#field-lookups">django docs</a> on the subject.</p>
</section>
<sectionid="get-that-werewolf">
<h2><spanclass="section-number">12.2. </span>Get that werewolf …<aclass="headerlink"href="#get-that-werewolf"title="Permalink to this headline">¶</a></h2>
<p>Let’s see if we can make a query for the werewolves in the moonlight we mentioned at the beginning
of this lesson.</p>
<p>Firstly, we make ourselves and our current location match the criteria, so we can test:</p>
<p>This is an example of a more complex query. We’ll consider it an example of what is
possible.</p>
<asideclass="sidebar">
<pclass="sidebar-title">Line breaks</p>
<p>Note the way of writing this code. It would have been very hard to read if we
just wrote it in one long line. But since we wrapped it in <codeclass="docutils literal notranslate"><spanclass="pre">(...)</span></code> we can spread
it out over multiple lines without worrying about line breaks!</p>
<li><p>We want to find <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code>s, so we access <codeclass="docutils literal notranslate"><spanclass="pre">.objects</span></code> on the <codeclass="docutils literal notranslate"><spanclass="pre">Character</span></code> typeclass.</p></li>
<li><p>We start to filter …</p></li>
<li><ul>
<li><p>… by accessing the <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> field (usually this is a Room)</p></li>
<li><p>… and on that location, we get the value of <codeclass="docutils literal notranslate"><spanclass="pre">db_tags</span></code> (this is a <em>many-to-many</em> database field
that we can treat like an object for this purpose; it references all Tags on the location)</p></li>
<li><p>… and from those <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code>, we looking for <codeclass="docutils literal notranslate"><spanclass="pre">Tags</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is “monlit” (non-case sensitive).</p></li>
</ul>
</li>
<li><p>… We also want only Characters with <codeclass="docutils literal notranslate"><spanclass="pre">Attributes</span></code> whose <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> is exactly <codeclass="docutils literal notranslate"><spanclass="pre">"lycantrophy"</span></code></p></li>
<li><p>… at the same time as the <codeclass="docutils literal notranslate"><spanclass="pre">Attribute</span></code>’s <codeclass="docutils literal notranslate"><spanclass="pre">db_value</span></code> is greater-than 2.</p></li>
</ul>
<p>Running this query makes our newly lycantrophic Character appear in <codeclass="docutils literal notranslate"><spanclass="pre">will_transform</span></code> so we
know to transform it. Success!</p>
<blockquote>
<div><p>Don’t confuse database fields with <aclass="reference internal"href="../../../Components/Attributes.html"><spanclass="doc std std-doc">Attributes</span></a> you set via <codeclass="docutils literal notranslate"><spanclass="pre">obj.db.attr</span><spanclass="pre">=</span><spanclass="pre">'foo'</span></code> or
<codeclass="docutils literal notranslate"><spanclass="pre">obj.attributes.add()</span></code>. Attributes are custom database entities <em>linked</em> to an object. They are not
separate fields <em>on</em> that object like <codeclass="docutils literal notranslate"><spanclass="pre">db_key</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">db_location</span></code> are.</p>
</div></blockquote>
</section>
<sectionid="complex-queries">
<h2><spanclass="section-number">12.3. </span>Complex queries<aclass="headerlink"href="#complex-queries"title="Permalink to this headline">¶</a></h2>
<p>All examples so far used <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code> relations. The arguments to <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code> are added together with <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>
(“we want tag room to be “monlit” <em>and</em> lycantrhopy be > 2”).</p>
<p>For queries using <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code> we need Django’s
<aclass="reference external"href="https://docs.djangoproject.com/en/1.11/topics/db/queries/#complex-lookups-with-q-objects">Q object</a>. It is
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is an object that is created with the same arguments as <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, for example</p>
<p>You can then use this <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> instance as argument in a <codeclass="docutils literal notranslate"><spanclass="pre">filter</span></code>:</p>
<p>The useful thing about <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> is that these objects can be chained together with special symbols (bit operators):
<codeclass="docutils literal notranslate"><spanclass="pre">|</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">OR</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">&</span></code> for <codeclass="docutils literal notranslate"><spanclass="pre">AND</span></code>. A tilde <codeclass="docutils literal notranslate"><spanclass="pre">~</span></code> in front negates the expression inside the <codeclass="docutils literal notranslate"><spanclass="pre">Q</span></code> and thus
works like <codeclass="docutils literal notranslate"><spanclass="pre">NOT</span></code>.</p>
<p>Would get all Characters that are either named “Dalton” <em>or</em> which is <em>not</em> in prison. The result is a mix
of Daltons and non-prisoners.</p>
<p>Let us expand our original werewolf query. Not only do we want to find all
Characters in a moonlit room with a certain level of <codeclass="docutils literal notranslate"><spanclass="pre">lycanthrophy</span></code>. Now we also
want the full moon to immediately transform people who were recently bitten,
even if their <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> level is not yet high enough (more dramatic this
way!). When you get bitten, you’ll get a Tag <codeclass="docutils literal notranslate"><spanclass="pre">recently_bitten</span></code> put on you to
<p>These Python structures are internally converted to SQL, the native language of
the database. If you are familiar with SQL, these are many-to-many tables
joined with <codeclass="docutils literal notranslate"><spanclass="pre">LEFT</span><spanclass="pre">OUTER</span><spanclass="pre">JOIN</span></code>, which may lead to multiple merged rows combining
the same object with different relations.</p>
</aside>
<p>This reads as “Find all Characters in a moonlit room that either has the
Attribute <codeclass="docutils literal notranslate"><spanclass="pre">lycantrophy</span></code> higher than two, <em>or</em> which has the Tag
<codeclass="docutils literal notranslate"><spanclass="pre">recently_bitten</span></code>”. With an OR-query like this it’s possible to find the same
Character via different paths, so we add <codeclass="docutils literal notranslate"><spanclass="pre">.distinct()</span></code> at the end. This makes
sure that there is only one instance of each Character in the result.</p>
</section>
<sectionid="annotations">
<h2><spanclass="section-number">12.4. </span>Annotations<aclass="headerlink"href="#annotations"title="Permalink to this headline">¶</a></h2>
<p>What if we wanted to filter on some condition that isn’t represented easily by a
field on the object? Maybe we want to find rooms only containing five or more
objects?</p>
<p>We <em>could</em> do it like this (don’t actually do it this way!):</p>
<p><codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code> is a Django class for counting the number of things in the database.</p>
<p>Here we first create an annotation <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> of type <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. It creates an in-database function
that will count the number of results inside the database.</p>
<blockquote>
<div><p>Note the use of <codeclass="docutils literal notranslate"><spanclass="pre">location_set</span></code> in that <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code>. The <codeclass="docutils literal notranslate"><spanclass="pre">*_set</span></code> is a back-reference automatically created by
Django. In this case it allows you to find all objects that <em>has the current object as location</em>.</p>
</div></blockquote>
<p>Next we filter on this annotation, using the name <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> as something we
can filter for. We use <codeclass="docutils literal notranslate"><spanclass="pre">num_objects__gte=5</span></code> which means that <codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code>
should be greater than or equal to 5. This is a little harder to get one’s head
around but much more efficient than lopping over all objects in Python.</p>
</section>
<sectionid="f-objects">
<h2><spanclass="section-number">12.5. </span>F-objects<aclass="headerlink"href="#f-objects"title="Permalink to this headline">¶</a></h2>
<p>What if we wanted to compare two dynamic parameters against one another in a
query? For example, what if instead of having 5 or more objects, we only wanted
objects that had a bigger inventory than they had tags (silly example, but …)?
This can be with Django’s <aclass="reference external"href="https://docs.djangoproject.com/en/1.11/ref/models/expressions/#f-expressions">F objects</a>.
So-called F expressions allow you to do a query that looks at a value of each
<p>Here we used <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> to create two in-query ‘variables’<codeclass="docutils literal notranslate"><spanclass="pre">num_objects</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">num_tags</span></code>. We then
directly use these results in the filter. Using <codeclass="docutils literal notranslate"><spanclass="pre">F()</span></code> allows for also the right-hand-side of the filter
condition to be calculated on the fly, completely within the database.</p>
<h2><spanclass="section-number">12.6. </span>Grouping and returning only certain properties<aclass="headerlink"href="#grouping-and-returning-only-certain-properties"title="Permalink to this headline">¶</a></h2>
<p>Suppose you used tags to mark someone belonging to an organization. Now you want to make a list and
need to get the membership count of every organization all at once.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code>, and <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> queryset methods are useful for this. Normally when
you run a <codeclass="docutils literal notranslate"><spanclass="pre">.filter</span></code>, what you get back is a bunch of full typeclass instances, like roses or swords.
Using <codeclass="docutils literal notranslate"><spanclass="pre">.values_list</span></code> you can instead choose to only get back certain properties on objects.
The <codeclass="docutils literal notranslate"><spanclass="pre">.order_by</span></code> method finally allows for sorting the results according to some criterion:</p>
<li><p>… has a tag of category “organization” on them</p></li>
<li><p>… along the way we count how many different Characters (each <codeclass="docutils literal notranslate"><spanclass="pre">id</span></code> is unique) we find for each organization
and store it in a ‘variable’<codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> using <codeclass="docutils literal notranslate"><spanclass="pre">.annotate</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">Count</span></code></p></li>
<li><p>… we use this count to sort the result in descending order of <codeclass="docutils literal notranslate"><spanclass="pre">tagcount</span></code> (descending because there is a minus sign,
default is increasing order but we want the most popular organization to be first).</p></li>
<li><p>… and finally we make sure to only return exactly the properties we want, namely the name of the organization tag
and how many matches we found for that organization.</p></li>
</ul>
<p>The result queryset will be a list of tuples ordered in descending order by the number of matches,
