django-mptt

• Nested Set trees.
• A register() call is used to set things up; it ads the necessary fields to the model.
• A tree model still has a foreign key to itself. This is the API you use to manage the tree. Signals are used that the hidden tree fields are updated when the parent ForeignKey changes. No add_child() required.
• Using the foreign key to self means that deletion is handled automatically be Django and/or the database. The other libraries need to implement a custom Queryset subclass to handle deletes.

django-treebeard

• Has an awesome name.
• In addition to the common Nested Set/MPTT approach, supports two other tree implementations. Materialized Path in particular is interesting.
• You inherit your models from abstract base classes, which I like.
• The tree has to be managed manually, that is, there are specific APIs like add_child() you have to call.
• Unfortunately, those APIs are classmethods on the model rather than the Django-way, putting them into the model manager.

django-easy-tree

• Apparently a fork of django-treebeard, but only supports Nested Set trees.
• But has a prettier API that fits very well into Django: Nicer class names, properly puts methods into the manager when they belong there, options are specified inside “Meta” rather than on the model itself.
• Has an interesting concept of validators. Included is a SingleRootAllowedValidator.
• No tests!

Clearly, somebody needs to write a django-treebeard that uses the django-easy-tree API design and django-mptt’s signal approach.

At critify, we do a lot of copy & paste. Sometimes, that means copy & pasting a date from another site into a Django DateTimeField on our site. This can be tedious, because Django only supports a fixed number of strict input formats, while the actual input can at times be something quite freeform. So I decided that what was needed were date form fields that would make a good attempt at parsing the input.

Here’s what you need:

• python-dateutil, which provides the parser.
• This file, which implements DateTimeField and DateField form field classes which use python-dateutil to process the input.

With that available, the actual code is quite simple:

from djutils.forms.fields import formfield_callback

class YourForm(ModelForm):
    formfield_callback = formfield_callback
    class Meta:
        model = YourModel

The formfield_callback will ensure that for every DateField or DateTimeField that your model has, the appropriate form field with dateutil-parsing support will be generated.

Of course, you can use these fields manually as well:

from django import forms
from djutils.forms.fields import DateField

class MyForm(forms.Form):
    std_date = forms.DateField() # < - Uses the standard DateField
    parsed_date = DateField()  # <- Uses the custom DateField

This should work well enough; we can now put something like 8th of January into the form, and it’ll be parsed into the correct date.

“8th of January”? My site is in German! Fortunately, while not supporting it out of the box, python-dateutil is flexible enough to allow for internationalization as well. You need to get:

• Yet another file, which adds support for German date strings to python-dateutil (i.e. “Janauar”, “Montag”, ….). If you need to support a different language, you can use that file as a template.

The way internationalization is done with python-dateutil is that you implement a parserinfo class for a given language. We need to tell our custom form fields to use the German parserinfo provided by the file above:

from djutils.forms.fields import get_formfield_for
from pyutils.date import GermanParserInfo

class YourForm(ModelForm):
    def formfield_callback(field):
         # If it's a date-related field, get one of our custom form fields 
         result = get_formfield_for(field, parserinfo=GermanParserInfo()) 
         # Otherwise, use the default form field.
         return result or field.formfield()
    class Meta:
        model = YourModel

That’s basically it. You might want to create a base form class that you can inherit from for DRY purposes.

By the way, my module containing the German parser info also has a MultiParserInfo helper that may be useful to some:

parserinfo=MultiParserInfo(parsers=[GermanParserInfo()])

This will give you a parser that supports both German date names, and the original English date names as a fallback. If you need to support even more languages, you could too:

parserinfo=MultiParserInfo(parsers=[GermanParserInfo(), FrenchParserInfo()])

Ever since I decided to use SpamStopsHere (they’re excellent, by the way) I noticed that some emails sent to me by one of my site’s contact form didn’t pass their SPF check.

It turns out that apart from your normal “From” sender, an email message also has an envelope sender specified in the Return-Path header. The first is considered to be the author, the second refers to who is responsible for actually sending the message.

So my site sent those contact form emails to me using as the sender whatever the user specified as his email address in the form – in both the From and Return-Path path fields. Whenever the user’s email domain had SPF records installed, the validation would run the user’s email against my mail servers and thus obviously fail.

The solution then is to use as the Return-Path an address of your own. How to do this in Django? It’s really simple, but totally non-obvious. This is per Ticket 9214 and Changeset 9842 (so you need at least 1.1).

connection = SMTPConnection(fail_silently=fail_silently)
headers = {'From': 'users@email.address'}  # From-header
from_email = 'bounce@mysite.com'           # Return-Path header
EmailMessage(subject, message, from_email, recipient_list, 
                   connection=connection, headers=headers).send()

Note that we need to work with “EmailMessage“ directly – django.core.mail.send_mail doesn’t provide an option for custom headers.

If you are using django-contact-form, it unfortunately doesn’t support custom headers out of the box either, so you need to apply a patch. Then, the following should work:

class MyContactForm(AkismetContactForm):
    def __init__(self, *args, **kwargs):
        super(self.__class__, self).__init__(*args, **kwargs)        
            self.subject = "Contact Form"
            self.template_name = "site/pages/contact_email.txt"
            # Return-Path header, e.g. bounce@mysite.com
            self.from_email = settings.DEFAULT_FROM_EMAIL   
            # From-header
            self.headers = 
                lambda: {'From': "%s " % 
                    (self.cleaned_data['name'], self.cleaned_data['email'])}

Since I’ve been playing a round with Jinja a bit the last few days, I needed django-assets to support it as well. So it can now be used with both Django and Jinja templates, even simultaneously. So far, the code is only in bzr, not packaged up. Details in the Readme, though it’s all pretty straightforward.

I initially had some trouble porting the Django template tag to a Jinja extension, partly due a seemingly limited number of examples out there – so if you’re in the same boat, maybe this will help.

After putting it off again and again, I decided that it would be finally time to make an effort and start writing some actual tests for my views (rather than just the code behind them). Beyond Django’s neat, but somewhat limited test client, the tool to do this seems to be twill.

Twill supports a WSGI intercept that can be hooked up to Django, bypassing the need to go through the network. There’s an entry on django snippets that shows how to use this in context of unit tests.

I’ve slightly improved upon it to remove the need to deal with the local dummy host in urls. I also added a login/logout feature, as known from the Django’s own test client. The result can be found on BitBucket.

Quick example:

from djutils.test import twill

# install the wsgi hook
twill.setup()

try:
    # those will go to Django
    twill.go('/')
    twill.follow('login')
    twill.code(200)

    # once we're browsing away...
    twill.go('http://google.com')
    # ...urls will be relative to that
    twill.go('/services')
    # now at http://google.com/services

    # default=True brings us back to Django
    twill.go('/list', default=True)

    # you can also pass along reverse args
    twill.go('proj.app.views.func', args=[1,2,3])

    # login as some user
    twill.login(username="foo", passwort="bar")
    twill.go('/django_admin')
    twill.logout()

    # alternative: login without credentials
    any_admin = User.objects.filter(is_superuser=True)[0]
    twill.login(user=any_admin)
finally:
    twill.teardown()

For the benefit of people searching for this, I should also mention that when calling twill.setup(), a workaround is applied to make twill function with documents documents that identify themselves as XHTML. Per default, twill explicitly fails on those with a “Not viewing html” error. Looking at the source, this is apparently because support for XHTML is considered incomplete and/or broken. However, at least for two XTHML sites of mine everything seems to work just fine.

Finally, I recommend you set DEBUG_PROPAGATE_EXCEPTIONS = True while testing with twill, as explained here. Otherwise, not only will debugging tests be more difficult, in some instances tests failures may easily go unnoticed (imagine you’re testing for string containment, but inadvertently dealing with your 404 or 500 error page).

Copying a model instance, i.e. creating a new database row based on an existing one (copying all the data except the primary key), doesn’t seem to be a very common need – it’s not directly supported by the Django ORM, and Google doesn’t turn much up, either.

But maybe that’s because, in most cases, it really is pretty simple:

>>> obj = Model.objects.get(pk=1)
>>> obj.pk = None    # (or: obj.id=None)
>>> obj.save()
>>> obj.pk
2

That’s how I’ve done it myself in the past. However, it turns out that this doesn’t work when model inheritance is used. When copying a subclass model, obj.id refers to the parent model object’s primary key, and obj.pk points to the parent_link OneToOne relationship. Both need to be nulled for a copy to be created, otherwise the save() logic will cause the missing one to be reset to the original value (I haven’t gone through all of the code, but that seems to be the effect).

>>> obj = SubclassModel.objects.get(pk=1)
>>> obj.pk = None
>>> obj.id = None
>>> obj.save()
>>> obj.pk
2

While that works, I felt like it would be relying on implementation details a bit too much, so currently I am using this function:

from django.db.models import AutoField
def copy_model_instance(obj):
    """Create a copy of a model instance.

    M2M relationships are currently not handled, i.e. they are not
    copied.

    See also Django #4027.
    """
    initial = dict([(f.name, getattr(obj, f.name))
                    for f in obj._meta.fields
                    if not isinstance(f, AutoField) and
                       not f in obj._meta.parents.values()])
    return obj.__class__(**initial)

Which basically initializes a new model instance with the data from the old, while leaving out all id and parent link fields. For example:

>>> obj = SubclassModel.objects.get(pk=1)
>>> obj = copy_model_instance(obj)
>>> obj.save()
>>> obj.pk
>>> 2

So far it works great, and I’m pretty happy with it – I’d love to hear other ideas, though.

Yes, yet another search app. However, unlike other projects, say this year’s GSoC project, it doesn’t try to be generic. Rather, it is specific to Xapian, making it possible to make full use of all the advanced features the excellent Xappy library offers. Xappy is a high-level interface to Xapian (as opposed to the tedious-to-use lowlevel Python bindings), and supports some pretty nice stuff like facets, tags or ranges.

Yes, I also know about djapian, but… I really wanted to use Xappy!

So, here’s how it works. First, define your index (note that since I really built this for my own use where indexes usually span multiple models, it might not be as simple as it could be if you want to search through only one – at least, right now).

from django_xappy import Index, action, FieldActions

class MyIndex(Index):
    location = '/var/www/mysite/search-index'

    class Data:
        @action(FieldActions.INDEX_FREETEXT, spell=True, language="en")
        @action(FieldActions.STORE_CONTENT)
        def name(self):
            if self == auth.models.User:
                return self.content_object.username  

        @action(FieldActions.SORTABLE, type="date")
        def date(self):
            if self == app.models.Book:
                return self.content_object.released_at
            elif self == user:
                return self.content_object.date_joined

MyIndex.register(Book)
MyIndex.register(auth.models.User)

This says where the index lives, what fields it has, what Xappy actions to use for each, and from where to get the data for those fields (note that not all models have to provide data for each field).

Now, the way django-xappy works is, it logs all changes in a database table instead of updating the index directly. This means that your index won’t always be up-to-date, but also, that the rest of your site’s functionality will never be affected by troubles with your search engine. Instead, you regularly apply the changes to your index (e.g. using a cronjob). The easiest way to do that is using the management command.

Let’s create the index for the first time:

PS G:...trunkexamplessimple> .manage.py index --full-rebuild
Creating a new index in "index-1218575904"...
Indexing 11 objects of type "Book"...
Indexing 2 objects of type "User"...
Switching "index-1218575904" to live index...
Done.

Then, after doing some changes, say in the admin:

PS G:...trunkexamplessimple> .manage.py index --update
Updating 1 index with 18 changes...
Done.

Good. Now that we have an up to date index, let’s search:

results = search("searchterm", page=1, num_per_page=10)

Pass results to your template:

    {% if results %}
        {% for result in results %}
            {{ result.content_object }}
        {% endif %}
    {% endif %}

Done! For more information, see the readme – and don’t forget to check out the Xappy docs as well.

As usual, the code is available on Launchpad – and now also PyPi. Via bzr:

bzr branch lp:django-xappy

For questions and support, use the django-apps Google Group and prefix your message with (xappy).

While trying to add a simple sorting feature to the critify game listing, I went off on a strange train of thought involving a bunch of future functionality I only have a very vague picture of, and decided that it would be best to choose the most complex approach possible and create a separate, overarchitected abstraction layer for that very purpose.

Well, something like that, though hopefully not quite as bad. Basically, the idea is:

• to making rendering tabular data a little easier by encapsulating some repetitive parts.
• make working with tabular data a lot easier when it comes to the user interacting with the table.

You would define a table, which is sort of a cross between a model and a form, define it’s columns (i.e. fields), and then tell it how you want to sort, filter and group the data.

Some snippets of actual code should probably explain it best.

First, let’s define a table:

import django_tables as tables

class BookTable(tables.ModelTable):
    id = tables.Column(sortable=False, visible=False)
    book_name = tables.Column(name='title')
    author = tables.Column(data='author__name')
    class Meta:
        model = Book

The table is based on the Book model; thus, it will have a column for each model field, and in addition the locally defined columns will override the default ones, or add to them, very much like newforms works (you’ll find that even the internals are at times very similar to the newforms code).

So, now that we have defined the table, let’s create an instance:

initial_queryset = Book.objects.all()
books = BookTable(
         initial_queryset,
         order_by=request.GET.get('sort', 'title'))

We tell the table to operate on the full book data set, and to order it by whatever the user sends along via the query string, or fall back to the default sort order based on the title (book_name) column.

Finally, you would send the table to the template:

return render_to_response('table.html', {'table': books})

Where it is easy to print it out:

<table>

<tr>
  {% for column in table.columns %}
  <th>
    {% if column.sortable %}
      <a href="?sort={{ column.name_toggled }}">
        {{ column }}
      </a>
      {% if column.is_ordered_reverse %}
        <img src="up.png" />
      {% else %}
        <img src="down.png" />
      {% endif %}
    {% else %}
      {{ column }}
    {% endif %}
  </th>
  {% endfor %}
</tr>

{% for row in table.rows %}
  <tr>
  {% for value in row %}
    <td>{{ value }}<td>
  {% endfor %}
  </tr>
{% endfor %}

</table>

The above template code generically renders any table you give it, restricted to it’s visible columns, and allows each column to be sorted in ascending or descending order (so long sorting is not disabled for a column). It gives you those nice arrow icons too.

At this point you will probably wonder if it’s not a lot simpler to just say:

Books.objects.all().order_by(request.GET.get('sort'))

And you’d be right, it is, and I indicated as much at the beginning of the post. There are however a couple of nice things about the table abstraction:

• While Django’s ORM already protects you against SQL injections, you still might like to play it safe and limit the possible values of the sort parameter (which will also ensure users won’t be able to guess your database schema by trying different values). Using django-tables, this is built in.
• It easily allows you to expose your fields under a different name, e.g. date instead of published_at. True, it’s just cosmetics, but personally I am (probably unhealthily) particular about stuff like that.
• Both previous points are especially relevant when you want to order via a relationship, e.g. author__birthdate. The double underscore doesn’t look all that nice, gives a rather clear insight into your database layout and also exposes that you are using Django, which may be undesirable.
• It easily allows you to move control over what fields to expose from your templates into python. Your templates will need to deal to a lesser extend with what to render, but rather on how to render it.
• Boilerplate code required to let the user sort the table, particularly when it comes to allowing toggling between descending/ascending, or even multiple sort fields, is reduced.

Additionally, it might be worth noting that there is a non-model implementation as well (use Table instead of ModelTable), that bascially does the same thing with static python data, e.g. a list of dicts.

To be perfectly honest, at this point I have no idea myself if much of this makes any sense or how useful it actually is. The fact that I haven’t really thought that much beyond the (already implemented) sorting functionality is not helping either (i.e. grouping, filtering…). I am going to toy with it for a bit though, and you are invited to do so too.

The project is maintained in bazaar and can be retrieved via Launchpad:

bzr branch lp:django-tables

There is also a source code browser, where you’ll also find the readme with a lot more information.

For questions/discussions/support use the django-apps Google Group and prefix your message with (tables).

There are already a couple asset management addons for Django out there, and so I feel somewhat bad for coming up with another one.

In my defense, non of the other approaches appealed to me: django-yslow apparently simply searches recursively through the project directory, finding media files and merging them into one single package. django-assetpacker has you define your assets in the database, which is a definite no-go for me. The database is installation-specific data, my media file configuration I consider to be application/project data (although I realize that that could be argued). Finally, django-compress uses the settings file for setup, which, while better, I’m still not all that happy with either.

So what then, you ask, do I want? See, right now, the only thing that references my JS and CSS files are my templates (only a limited number of base templates in fact). I’d like it to stay this way, i.e. assets should be defined and created solely through templates. So for example:

{% load assets %}
{% assets filter="jsmin" output="packed.js" "jquery.js" "common/file1.js" %}
    http://%20ASSET_URL%20
{% endassets %}

Which would merge jquery.js and file1.js, apply jsmin and store the result as packed.js. Since the tag simply renders it’s contents using the right ASSET_URL, you are free to reference your media files any way you want to, e.g. specify the correct media-type for CSS files.

Changes to the source files can automatically be detected (only by timestamp, right now), and the assets recreated. As an alternative, there is a management command that can be used to update manually:

./manage.py assets rebuild --parse-templates

As you can see from the command line, and this is the approach’s downside, since there is now no central repository of available assets, the command needs to parse your templates and look for usage of the assets tag. I find that a bearable compromise, however.

I’ve implemented a couple of the more popular minifiers and compressors, including YUI, jsmin, jspacker and cssutils. GZip is supported as well, and you can apply multiple filters:

{% assets filter="jsmin,gzip" ... %}

Another notworthy feature might be the ability to rewrite relative urls in CSS files. If your output file is in a different location than your CSS source files, relative url() references will break. The cssrewrite filter will fix this:

{% assets filter="cssrewrite,yui_css" output="cache/default.css" common/base.css extjs/css/extjs-all.css %}

I’ve put the whole thing up as a Google Code project. I’ve also created a Google Group and since I’m planning on releasing a couple other things in the future, generically called it django-apps – I don’t see the point of having a dozen empty groups, not to mention the inconvenience of jumping between them all. Feel free to use the group for your own projects – I suggest messages should be prefixed with (appname), e.g. (assets).

Problem: You need the filesystem location of the current project, but your code doesn’t know which project is is used by (think 3rd party app).

Answer:  Define “project location” as “location of settings file”, and you can do:

from django.conf import settings
os.path.dirname(os.path.normpath(os.sys.modules[settings.SETTINGS_MODULE].__file__))