Actually, the title of this post should be Python: What I learned this week, as I’ve been trying to debug this problem for at least that long. What problem, you ask?

First let me say that I’m writing this down mostly as a recap and for my own reference, but if it can help a lonesome Google wanderer at some point, even better.

With that, let’s consider the following:

import copy
import types

class X(object):
    def foo(self): print "foo"
a = X()

# add dynamic method to class
def bar(self): print "bar"
a.bar = types.MethodType(bar, a, X)

# create shallow copy
b = copy.copy(a)

# print results
print 'foo: %s' % (a.foo.im_self == b.foo.im_self)
print 'bar: %s' % (a.bar.im_self == b.bar.im_self)

If you are a Python noob like myself, the output might surprise you:

foo: False
bar: True

b.bar does in fact still think it’s a method of the a instance! Obviously, that means if called, it will pass a as self instead of b – probably not what we want.

I ran into this issue when I recently upgraded my Django checkout – I was using a very old revision previously, like ~r5600. I’ve been using this function that wraps around a Select Widget’s render() method, and inserts HTML that makes the items searchable (in fact, I posted a version of this to DjangoSnippets). So basically, I was doing something like this:

widget = FormClass.base_fields['fieldname'].widget
widget.render = types.MethodType(new_render, widget, widget.__class__)

forminstance = FormClass()
forminstance.fields['fieldname'].widget.choices = ...
return render_to_response(....., {'form': forminstance})

As I said, this worked perfectly with the old checkout. Now, see the second to last line, where widget.choices is modified? After the update, changes like that stopped having any effect on the output – “like that” meaning: after the form was instantiated.

Can you guess what happend? forminstance.fields[‘fieldname’].widget.render, as a bound method, was in fact still referring to the widget instance of the form class (i.e. FormClass.base_fields[‘fieldname’].widget). It turns out that when you create an instance of a form, Django deepcopies the base_fields array of the form class to create the new fields array of the instance (*).

It would seem that copying an object does not update the self-pointer of it’s methods. But looking at the original example, it works just fine for foo, the statically declared method. Why not for the dynamically added bar? I set to find that out, and soon discovered descriptors. I had heard about them before, but didn’t really bother. Apart from the python documentation itself, this article by Raymond Hettinger helped me understand.

Now, it makes perfect sense. b.foo actually invokes X.foo.__get__(b, X), which returns a bound method object with the correct self. On the other hand, b.bar just returns the method instance we created previously for a from b‘s instance __dict__ – but which is still referring to a.

Ok, so now that we know what the issue is, it should be easy to fix, right? Well, it turns out, apparently not so. If not to say impossible, although I might very well be missing something.

Remember, we want to change the method specifically on an instance of the class (i.e. the Widget or X), and not the class itself (that would be easy and would affect all instances). So we need to get a method object (i.e. of type instancemethod) assigned to the instance, and only the instance, that always returns with the correct self. Just a plain function wouldn’t work, as that would require explicit passing of self, which obviously breaks Django’s existing code, which expects a bound method).
Unfortunately, any assignment to the instance ends up in the instance’s own __dict__ and the descriptor protocol doesn’t seem to be in effect here.

1. When assigning a descriptor object to the instance, we just get the object itself back, it’s __get__ is never called:
   >>> a.xyz = property(X._get)
>>> print a.xyz
<property object at 0x015691E8>
2. Assigning to a.bar.im_func (or widget.render.im_func) isn’t possible, as it’s a readonly attribute, according to Python. Even it that were different, I’m not sure what would actually be changed. Is the instancemethod object regenerated or updated each time a function is accessed as an attribute? Not sure about the internals here.

To finally come to an end, I found a solution after all. It’s a little bit hackish, but also neat in a way, and definitely preferable over only supporting class-wide changes (to the render method), or only supporting changes after a Form instance has been created. Here it is:

def new_render(self, name, value, attrs=None, choices=()):
    custom_html="..."
    return mark_safe(
        custom_html+
        super(RenderSubclass, self).render(name, value, attrs, choices))

RenderSubclass = type('RenderSubclass',
        (widget.__class__,),
        {'render': new_render})
widget.__class__ = RenderSubclass

(*) Django’s Field and Widget base classes actually implement the __deepcopy__ hook, which in the case of Widget just does a shallow copy. Maybe something changed here between my old checkout and the current version, although I am not sure what, because whenever I tried to do deepcopy() an object with a dynamically added method, it didn’t really work either.