From 60207cd3473a36e94a4ac1aaba62e0ff8c5fde8a Mon Sep 17 00:00:00 2001
From: Griatch
As part of our new use of semantic versioning, Evennia
+2.0.0 was released today. Evennia is a Python MU* creation framework and
+server. As the change of the major version indicates, this is a backwards
+incompatible change ... well, maybe. It depends on your use-case. Read on.
In this post I'll go over some of the new things since the release of Evennia +1.0 some six months ago.
+The main thing that may be backwards income is the ExtendedRoom contrib.
+This contrib extends the standard room with the ability to look at 'details' and
+to have descriptions that change by season and by time-of-day in the game. I
+originally wrote the ExtendedRoom contrib more than a decade ago and it was
+in dire need to be recactored and cleaned up to use the modern tools available
+in Evennia. Not to mention make use of my additional years of experience since
+first creating it.
Here are the new features:
+season and time_of_day are now just
+treated as any other room-states except that they auto-change with in-game time.desc_<roomstate> to the room. This will be used when a given
+room-state is set on the room. If no room state matches, the traditional
+desc Attribute is used.<morning>...</morning>
+in your descriptions and have those parts of the texts only appear when it's
+morning in-game. This still works, but only ever supported time-of-day states.
+The new way is to make use Evennia's FuncParser to embed functions in the text
+directly. The ExtendedRoom understands the new state inline function. This
+can be used with any room state (including seasons and time-of-day). For
+example: $state(morning, The morning sun is shining in.) or
+$state(burning, This place is on fire!).details - the ability to look at things in the room
+without needing to create a new database object first. This hasn't changed.ExtendedRoom now also has simple support for echoing random messages now
+and then to the room.Overall, the code was completely re-written and cleaned up, making use of the +latest Evennia tooling. It also has much better unit-test coverage. All the unit +tests of the old implementation passes for the new one, but the changes are +so big that it's nevertheless possible people making heavy use of +this contrib may see side effects from upgrading.
+Now, a contrib is not part of the core, so it (potentially) being
+backwards-incompatible should normally not warrant a major version bump. But I
+feel the ExtendedRoom contrib is used by so many in the Evennia community that
+it's prudent to up the major version to let people know that they should
+keep an eye out when upgrading.
Most of my time has been spent continuing to work on the new Beginner Tutorial. Notably on part three, where +we are making a whole little Evennia MUD game from scratch. While still not +done, I have now added detailed lessons on
+TODOs are Monster/NPC AI, Dynamical generation of rooms, Questin, Shops and some +more bits and bobs. And of course a separate session on building the game world +and tying all these systems together into a little example game.
+As usual, the Evennia community ame through with a bunch of new useful stuff since +the release of Evennia 1.0.
+Containers - A new contrib with a typeclass and commands to make and manage
+all sorts of containers, from chests to crates and jars, along with commands
+to put things in them and take things out of them. A great place to start
+tweaking for your own game implementation (InspectorCaracal).logger.delete_log method for deleting log files from inside the server (aMiss-aWry).SessionHandler, it was cleaned up and
+refactored to make it less prone to cause circular import issues (Volund).create_channel(attr=...) keyword, for setting channel Attributes
+directly on creation, especially from channels defined in settings.DEFAULT_CHANNELS (me)deques with maxlen=... set (me).TagCategoryProperty - A new way to define tags with a particular category on a class at creation-time, without having to do so in the at_object_creation method (me).Over summer, my development usually slows down a bit, but I plan to keep pushing +on getting that Beginner tutorial done. We are also seeing more people joing +the evennia discord, which means more eyes on +the code and more bugs and edge cases being detected (and ironed out). So a lot +of maintenance work to be done in the short term.
+In the longer term, there are a lot of exciting plans for Evennia in the pipe, +but we'll get there when we get there. :)
+Have a nice summer!
... --- Evennia is a modern Python library and server for creating text-based @@ -49,6 +141,22 @@It's fun to see a growing level of activity in the Evennia community. The last few months have seen an increase in the number of people showing up in our IRC channel and mailing list. With this has come a slew of interesting ideas, projects and MUD-related discussion (as well as a few inevitable excursions into interesting but very non-mud-related territory - sorry to those reading our chat logs).
One sign of more people starting to actually use Evennia "for real" is how the number of bugs/feature requests have been increasing. These are sometimes issues related to new things being implemented but also features that have been there for some time and which people are now wanting to use in new and creative ways - or systems which noone has yet really used "seriously" before. This is very encouraging, especially since a lot of good alternative solutions, variations and edge cases can be ironed out this way. So keep submitting those Issues, people!
The budding Evennia community consists of people with a wide variety of interests, skillset and ambition.
-There are quite a few people who sees Evennia as a great stepping stone for learning Python, or for getting experience with creating a bigger programming project in general. Some are skilled programmers in other languages but we also have a few with only limited prior coding experience. From the experience in chat, it's really quite striking how fast members pick up the ropes. I'd like to think our documentation is at least partially helping here, but of course it helps that Python is inherently very easy a language to learn and use in the first place.
-Not all are participating with the goal of building a specific game. The general flow of patches and clone repository merges have also picked up. We have some users which are primarily interested in a coding challenge, to help with fixing bugs and features, or which uses Evennia as a starting point for exploring various web- and technical solutions that may or may not be a part of Evennia in the future.
+There are quite a few people who sees Evennia as a great stepping stone for learning Python, or for getting experience with creating a bigger programming project in general. Some are skilled programmers in other languages but we also have a few with only limited prior coding experience. From the experience in chat, it's really quite striking how fast members pick up the ropes. I'd like to think our documentation is at least partially helping here, but of course it helps that Python is inherently very easy a language to learn and use in the first place.
+Not all are participating with the goal of building a specific game. The general flow of patches and clone repository merges have also picked up. We have some users which are primarily interested in a coding challenge, to help with fixing bugs and features, or which uses Evennia as a starting point for exploring various web- and technical solutions that may or may not be a part of Evennia in the future.
The proposed Evennia game projects are just as varied as its users - and none are yet open to the public. As is common with these things, it's of course hard to determine who actually has the time and discipline to bring their plans to fruition. But I should really start to keep some sort of record of who works on what, I'm terrible with remembering this stuff ... so below is just some sort of summary of my impressions, not a comprehensive listing.
As can be expected, most proposed Evennia projects concern relatively standard MUD-style games. A few people are into building traditional hack-and-slash variety games, but most want to expand on the concept considerably. There was even one user exploring using Evennia for a RobotWars kind of experience (where you "program" robot programs in a custom language and battle them). Another project (Avaloria, also blogging on the MUD-dev rss feed) aims for a sort of base-building/strategy mechanic combined with more traditional MUD elements. There are at least two zombie-survival concepts floating around and a few large-scale procedural-content-driven science-fiction text games. One user has apparently a working Smaug->Evennia importer.
It seems that most Evennia users want to offer some sort of roleplaying environment, or at least a "roleplay-friendly" one. Currently we have at least two MUCK admins who aim to convert their existing, running games to Evennia. Whereas the initial idea was to implement parsers for MUCK's MUF language, it seems the conclusion has now shifted to it being faster and easier to just rewrite the MUF-coded functionality in Python (and maybe use something like Evlang for player scripting instead). Several people have announced their interest in creating "RPI"-style games (Armageddon seems to be a big inspiration here), but there was also a MOO admin and even a writer of Interactive Fiction who dropped into the mailing list to see if Evennia could be used for their style of game.
@@ -682,27 +790,17 @@Django implements a very nice abstract Python API for accessing a variety of SQL-like databases. It makes it very convenient to maintain the database schema (not to mention that django-South gives us easy database migrations). The fact that Django is really a web framework also makes it easy to offer various web features. There is for example an "admin site" that comes with Django. It allows to modify the database graphically (in Evennia's case the admin site is not quite as polished as we would like yet, but it's coming).
Here are some highlights of our architecture:
Portal - This is a stand-alone Twisted process talking to the outside world. It implements a range of communication protocols, such as telnet (traditional in MUD-world), ssh, ssl, a comet webclient and others. It is an auto-connecting client to Server (below).
-Server - This is the main MUD server. This twisted server handles everything related to the MUD world. It accesses and updates the database through Django models. It makes the world tick. Since all Players connect to the Server through the Portal's AMP connection, it means Server can be restarted without any players getting kicked off the game (they will re-sync from Portal as soon as Server is back up again).
-Webserver - Evennia optionally starts its own Twisted webserver. This serves the game's website (using the same database as the game for showing game statistics, for example). The website is of course a full Django project, with all the possibilities that entails. The Django admin site allows for modifying the database via a graphical interface.
-Webclient - There is a stand-alone MUD web client existing in a page on the default website. This uses Twisted to implement a long-polling ("comet") connection to a javascript client. As far as Evennia's concerned, this is just another outgoing protocol served by the Portal.
-Other protocols - Since it's easy to add new connectivity, Evennia also offers a bunch of other connectivity options, such as relaying in-game channels to IRC and IMC2 as well as RSS feeds and some other goodies.
-An important thing to note about Twisted's asynchronous model is that there is no magic at work here: Each little snippet of code Twisted loops over is blocking. It's just hopefully not blocking long enough for you to notice. So if you were to put sleep(10) in one of those snippets, then congratulations, you just froze the entire server for ten seconds.
Profiling becomes very important here. Evennia's main launcher takes command arguments to run either of its processes under Python's cProfile module. It also offers the ability to connect any number of dummy Players doing all sorts of automated random actions on the server. Such profile data is invaluable to know what is a bottleneck and what is not.
-I never found Twisted asynchronous paradigms much harder to understand than other code. But there are sure ways to write stupid blocking code that will come back and bite you. For example, much of Evennia's workload is spent in the Server, most notably in its command handler. This is not so strange; the command handler takes care of parsing and executing all input coming from Players, often modifying the game world in various ways (see my previous post for more info about the command handler).
-The command handler used to be a monolithic, single method. This meant that Twisted had to let it run its full course before letting anyone else do their turn. Using Twisted's inlineCallbacks instead allowed for yielding at many, many places in this method, giving Twisted ample possibilities to split execution. The effect on multi-user performance was quite impressive. Far from all code can be rewritten like this though.
+I never found Twisted asynchronous paradigms much harder to understand than other code. But there are sure ways to write stupid blocking code that will come back and bite you. For example, much of Evennia's workload is spent in the Server, most notably in its command handler. This is not so strange; the command handler takes care of parsing and executing all input coming from Players, often modifying the game world in various ways (see my previous post for more info about the command handler).
+The command handler used to be a monolithic, single method. This meant that Twisted had to let it run its full course before letting anyone else do their turn. Using Twisted's inlineCallbacks instead allowed for yielding at many, many places in this method, giving Twisted ample possibilities to split execution. The effect on multi-user performance was quite impressive. Far from all code can be rewritten like this though.
Another important bottleneck on asynchronous operations is database operations. Django, as opposed to Twisted, is not an asynchronous framework. Accessing the database is a blocking operation and can be potentially expensive. It was never extremely bad in testing, to be honest. But for large database operations (e.g. many Players) database access was a noticeable effect.
I have read of some people using Twisted's deferToThread to do database writes. The idea sounds reasonable - just offload the operation to another thread and go on your merry way. It did not help us at all though - rather it made things slower. I don't know if this is some sort of overhead (or error) in my test implementation - or an effect of Python just not being ideal with using threading for concurrency (due to the GIL). Either way, certain databases like SQlite3 doesn't support multiple threads very well anyway, and we prefer to keep giving plenty of options with that. So no deferToThread for database writes. I also did a little testing with parallel processes but found that even slower, at least once the number of writes started to pile up (we will offer easy process-pool offloading for other reasons though).
As many have found out before us, caching is king here. There is not so much to do about writes, but at least in our case the database is more often read than written to. Caching data and accessing the cache instead of accessing a field is doing much for performance, sometimes a lot. Database access is always going to cost, but it does not dominate the profile. We are now at a point where one of the most expensive single operations a Player (even a Builder) performs during an entire gaming session is the hashing of their password during login. I'd say that's good enough for our use case anyway.
@@ -735,40 +833,24 @@A common variant I've seen in various Python codebases is to implement commands as functions. A function maps intuitively to a command - it can take arguments and it does stuff in return. It is probably more than enough for some types of games.
Evennia chooses to let the command be defined as a class instead. There are a few reasons. Most predominantly, classes can inherit and require less boiler plate (there are a few more reasons that has to do with storing the results of a command between calls, but that's not as commonly useful). Each Evennia command class has two primary methods:
parse() - this is responsible for parsing and splitting up the options part of the command into easy-to use chunks. In the case of open door with red key, it could be as simple as splitting the options into a list of strings. But this may potentially be more complex. A mux-like command, for exampe, takes /switches to control its functionality. They also have a recurring syntax using the '=' character to set properties. These components could maybe be parsed into a list switches and two parameters lhs and rhs holding the left- and right hand side of the equation sign.
-func() - this takes the chunks of pre-parsed input and actually does stuff with it.
-One of of the good things with executing class instances is that neither of these methods need to have any arguments or returns. They just store the data on their object (self.switches) and the next method can just access them as it pleases. Same is true when the command system instantiates the command. It will set a few useful properties on the command for the programmer to make use of in their code (self.caller always references the one executing the command, for example). This shortcut may sound like a minor thing, but for developers using Evennia to create countless custom commands for their game, it's really very nice to not have to have all the input/output boilerplate to remember.
... And of course, class objects support inheritance. In Evennia's default command set the parse() function is only implemented once, all handling all possible permutations of the syntax. Other commands just inherit from it and only needs to implement func(). Some advanced build commands just use a parent with an overloaded and slightly expanded parse().
So we have individual commands. Just as important is how we now group and access them. The most common way to do this (also used in an older version of Evennia) is to use a simple global list. Whenever a player enters a command, the identifier looks the command up in the list. Every player has access to this list (admin commands check permissions before running). It seems this is what is used in a large amount of code bases and thus obviously works well for many types of games. Where it starts to crack is when it comes to game states.
A first example is an in-game menu. Selecting a menu item means an instruction from the player - i.e. a command. A menu could have numbered options but it might also have named options that vary from menu node to menu node. Each of these are a command name that must be identified by the parser. Should you make all those possible commands globally available to your players at all times? Or do you hide them somehow until the player actually is in a menu? Or do you bypass the command system entirely and write new code only for handling menus...?
-Second example: Picture this scenario: You are walking down a dark hallway, torch in hand. Suddenly your light goes out and you are thrown into darkness. You cannot see anything now, not even to look in your own backpack. How would you handle this in code? Trivially you can put if statements in your look and inventory commands. They check for the "dark" flag. Fair enough. Next you knock your head and goes 'dizzy'. Suddenly your "navigation" skill is gone and your movement commands may randomly be turned around. Dizziness combined with darkness means your inventory command now returns a strange confused mess. Next you get into a fight ... the number of if statements starts piling up.
-Last example: In the hypothetical FishingMUD,. you have lots of detailed skills for fishing. But different types of fishing rods makes different types of throws (commands) available. Also, they all work differently if you are on a shore as compared to being on a boat. Again, lots of if statements. It's all possible to do, but the problem is maintenance; your command body keep growing to handle edge cases. Especially in a MUD, where new features tend to be added gradually over the course of years, this gives lots of possibilities for regressions.
-All of these are examples of situation-dependent (or object-dependent) commands. Let's jointly call them state-dependent commands. You could picture handling the in-game menu by somehow dynamically changing the global list of commands available. But then the global bit becomes problematic - not all players are in the same menu at the same time. So you'll then have to start to track who has which list of commands available to them. And what happens when a state ends? How do you get back to the previous state - a state which may itself be different from the "default" state (like clearing your dizzy state while still being in darkness)? This means you have to track the previous few states and ...
A few iterations of such thinking lead to what Evennia now uses: a non-global command set system. A command set (cmdset) is a structure that looks pretty much like a mathematical set. It can contain any number of (unique) command objects, and a particular command can occur in any number of command sets.
A cmdset stored on an object makes all commands in that cmdset available to the object. So all player characters in the game has a "default cmdset" stored on them with all the common commands like look, get and so on.
-Optionally, an object can make its cmdset available to other objects in the same location instead. This allows for commands only applicable with a given object or location, such as wind up grandfather clock. Or the various commands of different types of fishing rods.
-Cmdsets can be non-destructively combined and merged like mathematical sets, using operations like "Union", "Intersect" and a few other cmdset-special operations. Each cmdset can have priorities and exceptions to the various operations applied to them. Removing a set from the mix will dynamically rebuild the remaining sets into a new mixed set.
-The last point is the most interesting aspect of cmdsets. The ability to merge cmdsets allows you to develop your game states in isolation. You then just merge them in dynamically whenever the game state changes. So to implement the dark example above, you would define two types of "look" (the dark version probably being a child of the normal version). Normally you use your "default cmdset" containing the normal look. But once you end up in a dark room the system (or more likely the room) "merges" the dark cmdset with the default one on the player, replacing same-named commands with new ones. The dark cmdset contains the commands that are different (or new) to the dark condition - such as the look command and the changed inventory command. Becoming dazed just means yet another merger - merging the dazed set on top of the other two. Since all merges are non-destructive, you can later remove either of the sets to rebuild a new "combined" set only involving the remaining ones in any combination.
Similarly, the menu becomes very simple to create in isolation (in Evennia it's actually an optional contrib). All it needs to do is define the required menu-commands in its own cmdset. Whenever someone triggers the menu, that cmdset is loaded onto the player. All relevant commands are then made available. Once the menu is exited, the menu-cmdset is simply removed and the player automatically returns to whichever state he or she was in before.
@@ -794,18 +876,10 @@("Contribs" are optional code snippets and systems that are not part of the actual codebase. They are intended for you to use or dissect as you like in your game development efforts).
The ExtendedRoom is a room typeclass meant to showcase some more advanced features than the default one. Its functionality is by all means nothing revolutionary in MUD-world, but it was fun and very simple to do using only Evennia's basic building blocks - the whole thing took me some two hours to code, document and clean up for a contrib push. The "ExtendedRoom" contribution has the following features:
Season-based descriptions. The new Room typeclass will change its overall description based on the time of year (the contrib supports the four seasons, you can hack this as you please). It's interesting from an implementation point of view since it doesn't require any Script or ticker at all - it just checks on-demand, whenever it is being looked at, only updating if the season has actually changed. There is also a general description used as a fallback in case of a missing seasonal one.
-Time-of-day-based descriptions. Within each Season-based description you can embed time-of-day based ones with special tags. The contrib supports four time slots out of the box (morning, afternoon, evening, night). In the description, you just embed time-dependent text within tags, like
Details. I took the inspiration of these from a MOO tutorial I read a long time ago. "Details" are "virtual" look-targets in the room. It allows you to add visual interest without having to add a bunch of actual objects for that purpose. Details are simply stored in a dictionary on the room. Details don't change with Season in this implementation, but they are parsed for time-of-day based tags!
-Custom commands. The room is supported by extending two of the custom commands. The Details require a slightly modified version of the look command. There is also a new @desc for setting/listing details and seasonal descriptions. The new time command, finally, simply shows the current game time and season in the room.
-Installing and testing the snippet is simple - just add the new commands to the default cmdset (they will dynamically replace the same-named default ones), dig a few rooms of the new typeclass and play around! Especially the details do make building interesting rooms a lot more fun (I got hung up playing with them way too long last night).
@@ -840,8 +914,8 @@Testing a few different recipes from the web, I eventually settled on an approach which (with some modifications compared to the original) uses a whitelist (and a blacklist for some other things) to only allow a given set of ast nodes and items in the execution environment. It walks the AST tree before execution and kills dangerous Python structures in one large swath. I expanded on this a fair bit, cutting away a lot of Python functionality for our particular use case. Stuff like attribute acces and assignments, while loops and many other Pythonesque things went out the window.
Around this highly stunted execution system I then built the Evennia in-game scripting system. This includes in-game commands as well as scriptable objects with suitable slots defining certain functionality the Player might want to change. Each Evennia developer can also supply any set of "safe" blackbox functions to offer more functionality to their Player-coders.
A drawback is the lack of a functional timeout watchdog in case of a script running too long. I'm using Twisted's deferToThread to make sure the code blocks as little as possible, but alas, whereas I can check timeouts just fine, the problem lies in reliably killing the errant sub-thread. Internet experts suggest this to be tricky to do safely at the best of times (with threads running arbitrary code at least), and not wanting to kill the Twisted server is not helping things. I pondered using Twisted's subprocess support, but haven't gotten further into that at this point. Fact is that most of the obvious DOS attack vectors (such as the while loop and huge powers etc) are completely disabled, so it should hopefully not be trivial to DOS the system (famous last words).
-I've tentatively dubbed the softcode system "Evlang" to differentiate it from our normal database-related "Scripts".
-So is Evlang "safe" to use by untrusted evil Players? Well, suffice to say I'm putting it up with a huge EXPERIMENTAL flag, with plenty of warnings and mentions of "on your own risk". Running Evennia in a chroot jail and with minimum permissions is probably to recommend for the security paranoid. Hopefully Evennia coders will try all sorts of nasty stuff with it in the future and report their finding in our Issue tracker!
+I've tentatively dubbed the softcode system "Evlang" to differentiate it from our normal database-related "Scripts".
+So is Evlang "safe" to use by untrusted evil Players? Well, suffice to say I'm putting it up with a huge EXPERIMENTAL flag, with plenty of warnings and mentions of "on your own risk". Running Evennia in a chroot jail and with minimum permissions is probably to recommend for the security paranoid. Hopefully Evennia coders will try all sorts of nasty stuff with it in the future and report their finding in our Issue tracker!
But implementation details aside, I must admit it's cool to be able to add custom code like this - the creative possibilities really do open up. And Python - even a stunted version of it - is really very nice to work with, also from inside the game.
This is to be my Evennia dev blog, but it will also cover various other musings concerning programming in general and mud designing in particular. Whereas the Evennia mailing list remains the main venue for discussion, I will probably use this blog for announcing features too.
+Some background:
+Evennia is a Python MUD/MUX/MU* server. More correct is probably to call it a "MUD-building system". The developer codes their entire game using normal Python modules. All development is done with the full power of Python - gritty stuff like database operations and network communication are hidden away behind custom classes that you can treat and modify mostly like any Python primitives.
Since the server is based on the Twisted and Django technologies we can offer many modern features out of the box. Evennia is for example its own web server and comes with both its own website and an "comet"-style browser mud client. But before this turns into even more of a sales pitch, I'll just just direct you to the evennia website if you want to know more. :)
+I, Griatch, took over the development of Evennia from the original author, Greg Taylor, in 2010.
+ + +This is to be my Evennia dev blog, but it will also cover various other musings concerning programming in general and mud designing in particular. Whereas the Evennia mailing list remains the main venue for discussion, I will probably use this blog for announcing features too.
-Some background:
-Evennia is a Python MUD/MUX/MU* server. More correct is probably to call it a "MUD-building system". The developer codes their entire game using normal Python modules. All development is done with the full power of Python - gritty stuff like database operations and network communication are hidden away behind custom classes that you can treat and modify mostly like any Python primitives.
-Since the server is based on the Twisted and Django technologies we can offer many modern features out of the box. Evennia is for example its own web server and comes with both its own website and an "comet"-style browser mud client. But before this turns into even more of a sales pitch, I'll just just direct you to the evennia website if you want to know more. :)
-I, Griatch, took over the development of Evennia from the original author, Greg Taylor, in 2010.
- - -