Pylint is an excellent static analysis tool for checking Python code. One word that is not normally associated with Python, however, is 'static' and this manifests itself especially when trying to use pylint with Django projects. To improve the output of Landscape, I created a plugin called pylint-django to enhance pylint's ability to analyse codebases using Django.

What's the Problem?

Django does a lot of metaprogramming, by which it manipulates and changes several objects at runtime. Consider the following code:

class ExampleModel(models.Model):
    name = models.CharField(max_length=100)

e = ExampleModel.objects.create(name='example')

>>> type(e.name)
<type 'unicode'>

This will show you that, while name is defined as a CharField, at runtime it is in fact a unicode object (in Python 2). How? Django magic!

(Aside: there was once a branch in the Django repository called 'magic-removal'. What we have now is less magic than it used to be. Gosh.)

While all this is very clever and produces a very nice web framework, it does however play havoc with tools which rely on reading sourcecode without executing it, such as static analysis tools.

Enter the Plugin

The typical way to improve pylint's understanding of your codebase is to spend time configuring a .pylintrc file, tweaking options and turning errors off. It is a bit of a blunt instrument however, as any option is applied globally. To prevent warnings about objects not existing on model classes, you can either ignore all 'no such attribute' warnings or ignore all attribute warnings about things called objects.

There is however a better way. Pylint has a little known feature that allows you to create your own plugins. This is tremendously useful, and a lot more powerful than using a configuration file, because it allows much more nuanced adaptation of pylint's behaviour.

Dealing with Model Fields

Using the example model as before:

class ExampleModel(models.Model):
    name = models.CharField(max_length=100)

    def get_name_uppercase(self):
        return self.name.upper()

Using the default pylint, this will result in a warning:

$ pylint example/
E:  7,15: Instance of 'CharField' has no 'upper' member (no-member)

That is not ideal, so what pylint-django does is to replace (or transform) attributes on models and forms with a hybrid type of the django field itself and the type it is replaced by. That is to say, a CharField is replaced by an object which inherits from both CharField and str, which allows pylint to both understand the effective type of the field when used in other areas of the code, but also the constructor when defining the field.

Trying again with the plugin, the no-member error is no longer there:

$ pylint --load-plugins pylint_django example/

Transforming ForeignKeys

Another transformation happens with foreign key relationships and similar things such as OneToOneField. Here's another simple example model:

class Author(models.Model):
    pen_name = models.CharField(max_length=200)

class Book(models.Model):
    author = models.ForeignKey(Author)

    def get_author_name(self):
        return self.author.pen_name

    def get_author_age(self):
        return self.author.age

Pylint will warn here, as it sees self.author.pen_name and spots that ForeignKeys do not have a pen_name attribute - which is true. But of course, once Django has done its thing at runtime, self.author is no longer an instance of ForeignKey but is in fact an instance of Author. With the default behaviour of pylint, you will get this:

$ pylint example/
E: 14,15: Instance of 'ForeignKey' has no 'pen_name' member (no-member)
E: 14,15: Instance of 'ForeignKey' has no 'age' member (no-member)

Using the pylint-django plugin however, you get this:

$ pylint --load-plugins pylint_django example/
E: 14,15: Instance of 'Author' has no 'age' member (no-member)

As you can see, the spurious warning has been removed, and pylint is also now able to correctly warn about attributes that are not present on instances of Author.

The relevant code is here; essentially, pylint-django replaces a node in the AST generated by pylint.

Model Relationships and Guesswork

Python is duck typed - if it walks like a duck, talks like a duck, then it must be a duck. As such it seems not entirely unreasonable to apply similar logic to errors which are not always errors.

An example where this is useful is for model relationships. As part of a ForeignKey or ManyToManyField definition on a model, Django will create a reverse relation. So if you have a ForeignKey from Book to Author, then Author will have a book_set attribute added allowing you to filter and fetch books. It's possible to name this differently too, using the related_name argument to the constructor.

This causes a problem for analysis: it's not possible to know what the relationship names are without analysing the graph of model relationships. That's tricky at the best of times but pylint also has the property of "streaming" errors as they're found. It's not easy to pre-process all files.

This means it's not possible to perfectly solve issues like this:

class Author(models.Model):
    def count_books(self):/
        return self.book_set.count()

class Book(models.Model):
    author = models.ForeignKey(Author)

Django's generated book_set member is not available until runtime, so is unknown to pylint:

$ pylint example/
E: 18,15: Instance of 'Author' has no 'book_set' member (no-member)

The solution in this case is basically to guess. If an attribute of a model is requested which does not exist (book_set), then pylint-django looks at what happens to it. If it seems like it's used as a ModelManager - that is to say, if you call all() or filter() or similar on it - then it is assumed to be a relationship that the plugin was not able to determine, so the error is supressed. Quack.

Ignoring Errors

Some errors raised by pylint are simply not errors in the context of Django. This is the easiest type of change - there are a list of code patterns that are perfectly valid in Django but not in regular Python. For example, models have an objects attribute; it's quite easy to simply supress all errors from pylint complaining about non-existant attributes if the name is objects.

New Warnings

For debuging and use in the Django admin, it's very useful if models specify a __unicode__ method for display. pylint-django will issue a warning (W5101) if one is not defined. This can of course be turned off just as other pylint errors.

How to get pylint-django

The plugin is available on PyPI so you can simply install using pip:

$ pip install pylint-django

Then when running pylint, include the --load-plugins flag:

$ pylint --load-plugins pylint_django [other pylint options]

Alternatively, you may want to consider using prospector. This is another tool I wrote as part of improving code quality checks for Landscape. It runs pylint as well as several other tools such as mccabe complexity and pyflakes. Its purpose is to provide tweaked configuration out of the box, making all of these tools a bit easier to get up and running. It will automatically enable pylint-django if it detects Django as a dependency in your project.

And of course, there's always using Landscape itself, if you would like to have continuous code quality metrics!

What Next?

There are many things left to do, and there are many frameworks which could benefit from a dedicated pylint plugin. As work continues on Landscape and more errors are unearthed, the plugin will improve. And, of course, pull requests and feedback are always welcome!

Also, I will write another blog post explaining more about how to create plugins for pylint. If you are interested in that, you can sign up below to get email notifications of new blog posts. Until then, you can also take a peek at the source code of pylint-django.

If you enjoyed this article and would like to receive email notifications when new articles are published, sign up below:

About Landscape.io

Landscape.io is a tool to measure and track code quality and technical debt in your project. It can analyse Python code to point out errors and problems, and provides continuous metrics so you can see if your code is deteriorating.

You can sign up today with a free 14 day trial - no credit card required.

Get Started Now!