New year Devel branch updates (proxy-typeclasses and Evennia-library package)
177 views
Skip to first unread message
Griatch Art
Jan 9, 2015, 10:11:53 AM1/9/15
to eve...@googlegroups.com
Hi folks!
This is a little preview of what is coming to Evennia-world in the future.
Over Christmas I found some time to work on Evennia and the current state of affairs is currently visible in the "devel" branch of the Evennia repository. The "devel" update introduces two changes to Evennia, one behind the scenes (typeclasses to proxy models) and one very much visible to you as an end user (Evennia into a library). It should be remembered that this is not yet ready for production!
For those who want to see quick progress I encourage users to help me by pullng the devel branch and toying with it already. I need a lot more eyes on it before considering merging it into master. Whereas both features have some minor things which I know is still missing, what both are massively missing is more testing. Feedback and comments you can give should best come as soon as possible while this is still in devel branch.
Changing typeclasses to use proxy models
To understand this change you need to be a little familiar with how typeclasses work in master. A typeclass is a Python class whose __setattr__ and __getattributes__ methods are overridden in such a way that whenever you store something on it, it actually stores that data on its connected database model instance. That database model instance is in turn cached and held in memory by Evennia. It stores a text string holding the python path to the typeclass associated with it. For example, when you @create a "Rock" object of the typeclass "src.objects.objects.Object", what happens is that a new database object (model ObjectDB) is created to represent this new line in the database table. This model then initialize an instance of its typeclass (src.objects.objects.Object in this example) and ties to it using its .typeclass property. Vice-versa, this new typeclass ties back to the database model instance via its .dbobj property. By simply storing different typeclass-paths in the database one can this way represent any number of different game entities using very few database models.
A drawback with the typeclass system in the master branch is that it introduces some custom limitations on its classes. Notably you need to use the create_object, create_script etc functions to create typeclassed objects (you cannot overload __init__) since the database model and typeclass must be tied together and initialized correctly at creation. Furthermore there is a constant exchange between the two objects with lookups on the typeclass often leads to the lookup on its dbobj and also the other way around - this introduces a small but noticeable performance hit over time. Furthermore, since django knows nothing about this typeclass business querying objects in the database deals with database models and not typeclasses (this is a small limitation since we implement our own managers and there are also suggested other means of modifying queries to address this particular issue). When it comes to understanding the inheritance of typeclasses this is also a bit cumbersome since the typeclass inheritance tree is not actually stored in the database and can thus not be searched easily through it (such as when wanting to find all objects of a given typeclass and its children).
Enter proxy models. Proxy models has been around in django for a while but they were not there when I first started with Evennia and they had slipped under my radar until user Volund made me aware of them in chat. At the time I was working on another typeclass revamp using the existing system - I threw that aaway after looking into proxy models. A proxy model is basically described in the django docs as a way to expand a django model with alternative Python code without needing to change the database schema - in short a proxy class using the database model for storage. Sounds familiar? As it turns out django's proxy models suit our typeclass needs very well, without the need of our custom overlying implementation. So during Christmas I have converted Evennia's typeclass system to use proxy models. Below are the advantages and features of this new typeclass implementation:
There are two caveats of the new typeclass system:
So what changes does the new typeclass system require from you, the end user. Surprisingly little. The main thing is expected to be to remove your use of .dbobj and .typeclass and to change eventual imports of the default Object, Exit etc to instead be named DefaultObject, DefaultExit etc.
... But I'm not done yet. what WILL require some more changes is the next new change. Read on ...
Evennia becomes a library
Once I did the typeclass revamp I though I could just as well continue and add the other big change that has been discussed for a long time - converting Evennia to a proper package/library structure. The library change is also operational in devel now.
The package change means that Evennia itself takes on the role of a library with an "evennia" executable to do operations. The "game" directory is no longer shipped with the system but is created on-the-fly for each new game using the evennia launcher. This allows the evennia library to exist completely separately from the game implementation in the same way as django does. The freshly created game folder has empty starting modules for the common game entities, log files and configs and a dynamically created settings.py file that links to those files. So no more "copy example/cmdset.py up one level, then change your setting file to point to it ..." as we use in master. Since the game directory is created on the fly it is not a part of Evennia's version control which means that you can change it and restructure it as you please without being afraid of running into merge conflicts down the line. You can also easily create multiple games in different folders (as long as you change their ports to avoid collisions).
Here is an example of creating a new game with evennia once the library has been installed:
So why this change? The main advantage (and goal) of the package restructure is that this makes it easier to distribute Evennia in a more accessible form. Once we have worked out the kinks, it means that we can distribute Evennia in pypi and that those of you who are not interested in git will be able to do something like "pip install evennia" without much fuzz. Also getting evennia into other package systems (like debian) should be easier. It will also lead to Evennia adopting a more formal release schedule with version numbers (more on this in the future). Cool cats will of course still be able to follow and help using the bleeding edge git version as before.
What is left to do? Apart from testing, the starting game directory template probably needs some more starting docs. Docs in general needs to be updated (both for the new typeclasses and for the new library system). Migrations need to be written, so people can just drop in their old databases into the new game directory and run from there.
Comments and feedback are appreciated!
.
Griatch
This is a little preview of what is coming to Evennia-world in the future.
Over Christmas I found some time to work on Evennia and the current state of affairs is currently visible in the "devel" branch of the Evennia repository. The "devel" update introduces two changes to Evennia, one behind the scenes (typeclasses to proxy models) and one very much visible to you as an end user (Evennia into a library). It should be remembered that this is not yet ready for production!
For those who want to see quick progress I encourage users to help me by pullng the devel branch and toying with it already. I need a lot more eyes on it before considering merging it into master. Whereas both features have some minor things which I know is still missing, what both are massively missing is more testing. Feedback and comments you can give should best come as soon as possible while this is still in devel branch.
Changing typeclasses to use proxy models
To understand this change you need to be a little familiar with how typeclasses work in master. A typeclass is a Python class whose __setattr__ and __getattributes__ methods are overridden in such a way that whenever you store something on it, it actually stores that data on its connected database model instance. That database model instance is in turn cached and held in memory by Evennia. It stores a text string holding the python path to the typeclass associated with it. For example, when you @create a "Rock" object of the typeclass "src.objects.objects.Object", what happens is that a new database object (model ObjectDB) is created to represent this new line in the database table. This model then initialize an instance of its typeclass (src.objects.objects.Object in this example) and ties to it using its .typeclass property. Vice-versa, this new typeclass ties back to the database model instance via its .dbobj property. By simply storing different typeclass-paths in the database one can this way represent any number of different game entities using very few database models.
A drawback with the typeclass system in the master branch is that it introduces some custom limitations on its classes. Notably you need to use the create_object, create_script etc functions to create typeclassed objects (you cannot overload __init__) since the database model and typeclass must be tied together and initialized correctly at creation. Furthermore there is a constant exchange between the two objects with lookups on the typeclass often leads to the lookup on its dbobj and also the other way around - this introduces a small but noticeable performance hit over time. Furthermore, since django knows nothing about this typeclass business querying objects in the database deals with database models and not typeclasses (this is a small limitation since we implement our own managers and there are also suggested other means of modifying queries to address this particular issue). When it comes to understanding the inheritance of typeclasses this is also a bit cumbersome since the typeclass inheritance tree is not actually stored in the database and can thus not be searched easily through it (such as when wanting to find all objects of a given typeclass and its children).
Enter proxy models. Proxy models has been around in django for a while but they were not there when I first started with Evennia and they had slipped under my radar until user Volund made me aware of them in chat. At the time I was working on another typeclass revamp using the existing system - I threw that aaway after looking into proxy models. A proxy model is basically described in the django docs as a way to expand a django model with alternative Python code without needing to change the database schema - in short a proxy class using the database model for storage. Sounds familiar? As it turns out django's proxy models suit our typeclass needs very well, without the need of our custom overlying implementation. So during Christmas I have converted Evennia's typeclass system to use proxy models. Below are the advantages and features of this new typeclass implementation:
- - typeclasses now inherit directly from their database models. All such models are automatically treated as a proxy (this is implemented using some (if I may say so) pretty sophisticated metaclass magic, so creating a new typeclass requires no extra boiler plate except inheriting from the right parent.
- - There is no longer any difference between the database model and the typeclass (we still call the proxy children of the model "typeclasses" though). This means that the .dbobj and .typeclass properties does not make sense any more and were removed. All methods were moved from the database model to the typeclass and are now available directly through inheritance, so you should have much less need to go to the database model than you had. You can however still reach a typeclass' parent model at any time using typeclass.__dbclass__, which is the new standard along with the built-in __class__ property.
- - You can now query typeclasses directly, instead of only the main model. For example, if you have a typeclass "Rock" you can do Rock.objects.all() to get all the rocks. Conversely, if you do ObjectDB.objects.all() (or Rock.__dbclass__.objects.all()) you will (same as before) get all ObjectDB-derived instances, independent of typeclass. There are now also the new all_family, filter_family and get_family manager methods that allow you to query the database directly for an typeclass and all its subclasses, such as Rock.objects.all_family().
- - You can now create new instances of typeclasses using normal initialization. So rock = Rock() will now get you a correct typeclass (you need to do rock.save() to actually store it, same as any django model). Through internal signalling saving this will still trigger the correct startup hooks. It should be noted that the create_* functions still offers more functionality, since they can accept more arguments and add things like permissions at creation time (using the plain construction you'd need to add such things manually).
- - Interestingly, the new system requires no changes to the database schema, so the actual change needed in your code is not so big as one might think.
- - The new typeclass system is a lot easier to explain and should also be more efficient. Furthermore, it should be easier to cache using one of the many cache solutions available to Django (such as memcached) or using threading for supported database (more testing is needed of this though).
There are two caveats of the new typeclass system:
- - Django's proxies does not _quite_ fit our needs. I have modified the query system to return typeclasses rather than database models. I have also introduced a django patch to allow proxies multiple inheritance as long as they all stem from the same model. This patch is included in Evennia and introduced transparently, but it should hopefully soon be a part of future django versions so we don't need this hack.
- - There can only exist one proxy model of a given name for a given base model. This means that we can no longer do stuff like "from src.objects.objects import Object as BaseObject" and then create a class Object(BaseObject) (as was done in the gamesrc/ example files). Django interprets these as two proxy models based off ObjectDB, both named "Object", something which is not allowed. For this reason the default typeclasses are now called DefaultObject, DefaultRoom, DefaultExit etc, to allow end users the possibility to use the shorter Object, Exit, Room etc. This is one reason (apart from legacyy) that the classes are still called "typeclasses".
So what changes does the new typeclass system require from you, the end user. Surprisingly little. The main thing is expected to be to remove your use of .dbobj and .typeclass and to change eventual imports of the default Object, Exit etc to instead be named DefaultObject, DefaultExit etc.
... But I'm not done yet. what WILL require some more changes is the next new change. Read on ...
Evennia becomes a library
Once I did the typeclass revamp I though I could just as well continue and add the other big change that has been discussed for a long time - converting Evennia to a proper package/library structure. The library change is also operational in devel now.
The package change means that Evennia itself takes on the role of a library with an "evennia" executable to do operations. The "game" directory is no longer shipped with the system but is created on-the-fly for each new game using the evennia launcher. This allows the evennia library to exist completely separately from the game implementation in the same way as django does. The freshly created game folder has empty starting modules for the common game entities, log files and configs and a dynamically created settings.py file that links to those files. So no more "copy example/cmdset.py up one level, then change your setting file to point to it ..." as we use in master. Since the game directory is created on the fly it is not a part of Evennia's version control which means that you can change it and restructure it as you please without being afraid of running into merge conflicts down the line. You can also easily create multiple games in different folders (as long as you change their ports to avoid collisions).
Here is an example of creating a new game with evennia once the library has been installed:
evennia --init mygame # This creates a new "mygame" directory and creates all
# folders and a custom settings file that you may now tweak if you like.
cd mygame
evennia migrate # Creates the database based on your (possibly tweaked) settings.
evennia -i start # start the server
Main differences when coding using the new Evennia library:# folders and a custom settings file that you may now tweak if you like.
cd mygame
evennia migrate # Creates the database based on your (possibly tweaked) settings.
evennia -i start # start the server
- - src/ is no more. The src/ folder has been renamed evennia/ and the library is expected to be imported simply as "evennia" in your code.
- - ev.py is no more. All of the flat API has been included in evennia.__init__, which means that you can get to most common things directly via just the evennia import (such as evennia.DefaultObject).
- - game/ is no more, obviously. This is now the dynamically-created folder. The old examples, such as the red button has been moved to a new contrib/tutorial_examples/ folder.
- - game/manage.py was merged with game/evennia.py into the new bin/evennia executable. For now, call it explicitly with python <path-to-evennialib>/evennia. This will need to be made automatically available on $PATH down the line and linux users can do so manually if they want.
- - the default typeclass paths have changed to be located in the new game dir rather than in the evennia source tree. Migrations for this are not yet finished so use a fresh database to test.
So why this change? The main advantage (and goal) of the package restructure is that this makes it easier to distribute Evennia in a more accessible form. Once we have worked out the kinks, it means that we can distribute Evennia in pypi and that those of you who are not interested in git will be able to do something like "pip install evennia" without much fuzz. Also getting evennia into other package systems (like debian) should be easier. It will also lead to Evennia adopting a more formal release schedule with version numbers (more on this in the future). Cool cats will of course still be able to follow and help using the bleeding edge git version as before.
What is left to do? Apart from testing, the starting game directory template probably needs some more starting docs. Docs in general needs to be updated (both for the new typeclasses and for the new library system). Migrations need to be written, so people can just drop in their old databases into the new game directory and run from there.
Comments and feedback are appreciated!
.
Griatch
Griatch Art
Jan 10, 2015, 12:36:40 PM1/10/15
to eve...@googlegroups.com
I got asked some questions on how to actually get the devel branch of Evennia. So here is how!
- Go to your downloaded Evennia repo (or clone a fresh one from us if you want to keep things completely separate)
- git fetch
- git checkout devel
If you know how, you can now add the evennia launcher (bin/evennia) to your system path ($PATH) (linux only at the moment). If "system path" means nothing to you, you can instead start the launcher manually like so:
- Linux: python <path-to-your-evennia-repo>/bin/evennia
- Windows: python <path-to-your-evennia-repo>\bin\evennia
This should show you some text about evennia. Use -h to get help on using the launcher to create your game directory.
.
Griatch
Griatch Art
Jan 16, 2015, 6:50:51 PM1/16/15
to eve...@googlegroups.com
Update:
With the latest additions from Kelketek, the devel branch now has a correct python-package structure which means that to use it you need to "install" it with pip after you cloned it (in the future pip will download and install it for you, this is for development). If you want to try devel it's highly recommended that you do this is in a virtualenv (see our GettingStarted wiki page on how vitualenv works). So after you cloned the devel repository as described in the previous post you need to activate your virtualenv with the right requirements, cd to your downloaded repository and give the following command:
Note the period at the end, it must be there and means that you should install stuff from the current directory. The -e flag installs evennia using symlinks so you can tweak the repo and see results (otherwise it would be copied into the python install dir). From here on, evennia should be available in your path to use as described in the original post.
.
Griatch
With the latest additions from Kelketek, the devel branch now has a correct python-package structure which means that to use it you need to "install" it with pip after you cloned it (in the future pip will download and install it for you, this is for development). If you want to try devel it's highly recommended that you do this is in a virtualenv (see our GettingStarted wiki page on how vitualenv works). So after you cloned the devel repository as described in the previous post you need to activate your virtualenv with the right requirements, cd to your downloaded repository and give the following command:
pip install -e .
Note the period at the end, it must be there and means that you should install stuff from the current directory. The -e flag installs evennia using symlinks so you can tweak the repo and see results (otherwise it would be copied into the python install dir). From here on, evennia should be available in your path to use as described in the original post.
.
Griatch
Yijun Lu
Jan 20, 2015, 12:05:06 PM1/20/15
to eve...@googlegroups.com
Thank you for your great work! These changes can make the structure more clear and make the server more efficient. With Evennia library, developers can distribute their games in separate packages.
Griatch Art
Feb 20, 2015, 5:40:04 AM2/20/15
to eve...@googlegroups.com
Devel update:
The development of the devel-branch of Evennia (as described above) is coming along fine. With the help of contributors fixing code and testers finding bugs it's getting nearer to merging into master. You can get an overview of the devel-branch merge in https://github.com/evennia/evennia/pull/658. From that thread, here are some brief points of what remains to do be done before merging:
- Resolve remaining merge requests against devel (e.g. pull request #659)
- Discuss with Kelketek what needs to be done to close #645
- Add a more lenient mechanism when introducing syntax errors in typeclassed modules. It's currently raising a traceback (which can kill the entire cmdset loading too) rather than loading a temporary base class and logging the error as before.
- Merge my local rewrite of the tutorial_world (this is done, but it does not currently build correctly)
- Fix Windows installation to work using
pip install -e . - I also need to take a pass over the web features to make sure they work okay.
.
Griatch
Griatch Art
Feb 26, 2015, 6:03:40 AM2/26/15
to eve...@googlegroups.com
Hi folks, some more updates about the work going on in devel branch:
.
Griatch
- The development branch was moved ahead with the merger of Kelketek's expanded and reworked unittest suite - it's not only refactored to use modern django, it also makes it easy to run individual sub-tests of various systems.
- The procpool contrib is currently unfunctional with devel and to avoid delaying devel, this large contrib system was for now moved out into a separate repository here.
- With the aid of contributor acharles, dependency on the @ic command was removed from the character typeclass to make it easier to remove some default commands for a custom game.
- I have now finally taken the time to go through the Tutorial World from the ground up. Rewriting a lot of the underlying code, I've updated it to use the latest Evennia systems and recommended workflow. This lead to finding and fixing a lot of bugs and edge-cases on the way. Note that there is more one can do here in the prettifying/usability department (such as adding more color and/or following the road of Yijun Lu and introduce MXP for clickable links everywhere) - I will not take this one on right now though (maybe someone else is willing?)
.
Griatch
Reply all
Reply to author
Forward
0 new messages