Pylint is an excellent static analysis tool for checking Python code. One word that is not normally associated with Python, however, is 'static' and this manifests itself especially when trying to use pylint with Django projects. To improve the output of Landscape, I created a plugin called
pylint-django to enhance pylint's ability to analyse codebases using Django.
What's the Problem?
Django does a lot of metaprogramming, by which it manipulates and changes several objects at runtime. Consider the following code:
class ExampleModel(models.Model): name = models.CharField(max_length=100) e = ExampleModel.objects.create(name='example') >>> type(e.name) <type 'unicode'>
This will show you that, while
name is defined as a
CharField, at runtime it is in fact a
unicode object (in Python 2). How? Django magic!
(Aside: there was once a branch in the Django repository called 'magic-removal'. What we have now is less magic than it used to be. Gosh.)
While all this is very clever and produces a very nice web framework, it does however play havoc with tools which rely on reading sourcecode without executing it, such as static analysis tools.
Enter the Plugin
The typical way to improve pylint's understanding of your codebase is to spend time configuring a
.pylintrc file, tweaking options and turning errors off. It is a bit of a blunt instrument however, as any option is applied globally. To prevent warnings about
objects not existing on model classes, you can either ignore all 'no such attribute' warnings or ignore all attribute warnings about things called
There is however a better way. Pylint has a little known feature that allows you to create your own plugins. This is tremendously useful, and a lot more powerful than using a configuration file, because it allows much more nuanced adaptation of pylint's behaviour.
Dealing with Model Fields
Using the example model as before:
class ExampleModel(models.Model): name = models.CharField(max_length=100) def get_name_uppercase(self): return self.name.upper()
Using the default pylint, this will result in a warning:
$ pylint example/ E: 7,15: Instance of 'CharField' has no 'upper' member (no-member)
That is not ideal, so what
pylint-django does is to replace (or transform) attributes on models and forms with a hybrid type of the django field itself and the type it is replaced by. That is to say, a
CharField is replaced by an object which inherits from both
str, which allows pylint to both understand the effective type of the field when used in other areas of the code, but also the constructor when defining the field.
Trying again with the plugin, the
no-member error is no longer there:
$ pylint --load-plugins pylint_django example/
Another transformation happens with foreign key relationships and similar things such as
OneToOneField. Here's another simple example model:
class Author(models.Model): pen_name = models.CharField(max_length=200) class Book(models.Model): author = models.ForeignKey(Author) def get_author_name(self): return self.author.pen_name def get_author_age(self): return self.author.age
Pylint will warn here, as it sees
self.author.pen_name and spots that
ForeignKeys do not have a
pen_name attribute - which is true. But of course, once Django has done its thing at runtime,
self.author is no longer an instance of
ForeignKey but is in fact an instance of
Author. With the default behaviour of pylint, you will get this:
$ pylint example/ E: 14,15: Instance of 'ForeignKey' has no 'pen_name' member (no-member) E: 14,15: Instance of 'ForeignKey' has no 'age' member (no-member)
pylint-django plugin however, you get this:
$ pylint --load-plugins pylint_django example/ E: 14,15: Instance of 'Author' has no 'age' member (no-member)
As you can see, the spurious warning has been removed, and pylint is also now able to correctly warn about attributes that are not present on instances of
The relevant code is here; essentially,
pylint-django replaces a node in the AST generated by pylint.
Model Relationships and Guesswork
Python is duck typed - if it walks like a duck, talks like a duck, then it must be a duck. As such it seems not entirely unreasonable to apply similar logic to errors which are not always errors.
An example where this is useful is for model relationships. As part of a
ManyToManyField definition on a model, Django will create a reverse relation. So if you have a
Author will have a
book_set attribute added allowing you to filter and fetch books. It's possible to name this differently too, using the
related_name argument to the constructor.
This causes a problem for analysis: it's not possible to know what the relationship names are without analysing the graph of model relationships. That's tricky at the best of times but pylint also has the property of "streaming" errors as they're found. It's not easy to pre-process all files.
This means it's not possible to perfectly solve issues like this:
class Author(models.Model): def count_books(self):/ return self.book_set.count() class Book(models.Model): author = models.ForeignKey(Author)
book_set member is not available until runtime, so is unknown to pylint:
$ pylint example/ E: 18,15: Instance of 'Author' has no 'book_set' member (no-member)
The solution in this case is basically to guess. If an attribute of a model is requested which does not exist (
pylint-django looks at what happens to it. If it seems like it's used as a
ModelManager - that is to say, if you call
filter() or similar on it - then it is assumed to be a relationship that the plugin was not able to determine, so the error is supressed. Quack.
Some errors raised by pylint are simply not errors in the context of Django. This is the easiest type of change - there are a list of code patterns that are perfectly valid in Django but not in regular Python. For example, models have an
objects attribute; it's quite easy to simply supress all errors from pylint complaining about non-existant attributes if the name is
For debuging and use in the Django admin, it's very useful if models specify a
__unicode__ method for display.
pylint-django will issue a warning (
W5101) if one is not defined. This can of course be turned off just as other pylint errors.
How to get
The plugin is available on PyPI so you can simply install using pip:
$ pip install pylint-django
Then when running pylint, include the
$ pylint --load-plugins pylint_django [other pylint options]
Alternatively, you may want to consider using prospector. This is another tool I wrote as part of improving code quality checks for Landscape. It runs pylint as well as several other tools such as mccabe complexity and pyflakes. Its purpose is to provide tweaked configuration out of the box, making all of these tools a bit easier to get up and running. It will automatically enable
pylint-django if it detects Django as a dependency in your project.
And of course, there's always using Landscape itself, if you would like to have continuous code quality metrics!
There are many things left to do, and there are many frameworks which could benefit from a dedicated pylint plugin. As work continues on Landscape and more errors are unearthed, the plugin will improve. And, of course, pull requests and feedback are always welcome!
Also, I will write another blog post explaining more about how to create plugins for pylint. If you are interested in that, you can sign up below to get email notifications of new blog posts. Until then, you can also take a peek at the source code of pylint-django.