Improving Django: Generic views for RESTful web services
Django is my web programming framework of choice. The following 1.500 words describe what I would add to the core framework in order to make machine-to-machine communication (APIs) for almost every web application a matter of minutes.
Abstract
I suggest a general method offering a public and private API for existing Django models. New generic views will simplify data retrieval and modification via different web services in a resource-centric REST architecture, providing model data in formats such as XML, JSON and YAML with very little custom code.
Implementation
The most widely used browsers don't support PUT and DELETE in web forms without using AJAX, therefore it is impossible to achieve a pure REST implementation. Nonetheless a more RESTful resource-centric approach makes sense: With a unique hyperlink address for every resource, resource discovery from content becomes much easier. With all resources sharing a common interface, programs written in any language can talk to Django instances in a standardized way. The partial lack of browser support for PUT and DELETE can be compensated by adding an extra hidden field to forms that carries the method information.
The code that tells Django which web service should be provided for which model at which URL will not to be attached to the individual model classes but will live at the view level. There are two reasons for this:
- REST is only one way of looking at a Django application and therefore should not be part of the underlying model.
- There is no one-to-one mapping between resources and models. Resources have more to do with the user interface than with the underlying database structure. Sometimes it will be sufficient to apply the CRUD actions to models, but sometimes there will be the need for more complex resources (e.g. one resource that consists of several related models, service discovery). It is important to me that there is an obvious way of using both the generic views and custom developments consistently in one Django project.
I will solve this by writing a general Resource class as an interface. Each method of Resource raises a Http404 exception by default. Custom resources inherit from Resource and may implement the methods __call__ (thus providing the equivalent of current view functions), create, retrieve, list, update, delete, get_create_form and get_update_form (cf. djangocollection). The ModelResource class that covers all the cases where there is a one-to-one mapping between a model and a resource inherits from the class Resource.
When using the ModelResource class we need to be able to set a few options:
- Which data should be accessible?
- Which CRUD actions should be possible?
- Which model fields should be visible?
- Which type of authentication is required?
- How should the output look like (xml, json, html, yaml, ...)?
I imagine the urls.py to look like this (similar to a draft by Jacob Kaplan-Moss):
-
from mysite.polls.models import Poll
-
from django.conf.urls.defaults import *
-
from django.contrib import restapi
-
-
poll_api = {
-
queryset : Poll.objects.all(),
-
allowed_operations : (restapi.READ, restapi.UPDATE),
-
expose_fields : ('id', 'question', 'pub_date'),
-
authentication_method : restapi.APIKeyAuthentication,
-
output_function : restapi.json_response
-
}
-
-
urlpatterns = patterns('',
-
(r'^json/polls/(?:(?P<id>\d+)/?)?(?:;(?P<noun>\w+))?/?$', 'django.contrib.restapi.ModelResource', poll_api)
-
)
In order for our resource implementation to be REST-friendly, each resource needs to have a unique URL. If the content format is not specified by the URL and if there is more than one content format per app, looking at the HTTP headers may be sufficient to choose which content to deliver (we could set output_format to restapi.DETECT_FORMAT).
The functionality of the new generic RESTful views will comprise a large part of the functionality of the current generic create_update and list_detail views. This includes the ability to supply the views with a filtered queryset, pagination, response mimetype, allow_empty, post_save_redirect and post_delete_redirect.
In contrast to the current generic views the ModelResource class will not use templates by default. Instead, in order to follow the DRY principle, they could call a user-supplied output function (e.g. xml_response, json_response) depending on which type of response (xml, json, ...) to return. I'm still thinking about how the most elegant solution will look like. I'll be happy to hear suggestions.
Another issue that still needs to be resolved is how to get authentication right. The consensus of ticket #115 seems to be that HTTP Basic Authentication over SSL is "secure enough" but that the risk of people not using SSL is quite high. Both public and private web services will be possible.
Web development shouldn't be monotonous. Setting up APIs can be. I'd like to automate this as far as possible without taking away the freedom to extend or replace the API with more complex solutions.
Existing Code
There exists a fair amount of code that might be useful for this project.
There is john@sneeu.com who wrote at ticket #2553:
I've written a RESTful API (similar urls to the 'oldforms' Admin), it needs a bit of a tidy up, and a few more tweaks but I'm more than happy for it to be incorporated in to Django in any way.
Jason Huggins mentions in the discussion of ticket #115 (Models CRUD via web services):
I have a fair enough amount of code already written for this ticket. If possible, I'd like a branch in subversion set up so I can check in my code and we can start talking about real example code.
Jacob Kaplan-Moss to whom the ticket is assigned may have started writing some code that can be used for both REST and RPC. He wrote:
Jason (and others): if you've got code that's not attached to this ticket or to #356, #547 or #552 please attach it to this one as I'm going to try to get some work done on this Any Day Now™.
More recently, Adam Smith has started working on django-restful-model-views which intends to make it easier to create ReSTful Web applications with Django. In the current stage this is still more of a proof-of-concept, but the code written by Adam Smith might turn out to be a good point to start.
The wsgicollection-inspired project djangocollection which provides a set of generic RESTful urls for all the models of a project and a GenericCollection which uses the django generic views illustrates a few interesting and relevant ideas, too.
Before getting started I will talk to those who have written possibly useful code and to the Django community in order to find out how my suggested approach can be improved and how much reusable code already exists.
Benefits to Django
There will be a standard interface for applications to communicate with Django projects. Making it easier for Desktop applications to use Django as a server-side backend will encourage Desktop applications to use Django as a backend. Yay.
An API that can return data in JSON format will make AJAX integration much easier — both for the admin app and for Django users who complain about AJAX not being as easy as with competing frameworks.
The value of having a dead easy way to get data in and out of Django databases is obvious. Let's make extending existing applications with a fully-fledged REST API a matter of less than five minutes.
Success Criteria
- A few lines of code in urls.py are sufficient to make model data CRUD-accessible via XML and JSON.
- It is possible to create custom API resources that don't correspond directly to models. These should be integrated seamlessly into the rest of the API.
Roadmap
- Assess together with those who have already written similar code whether it makes sense to start from scratch. Get feedback from the community on issues such as how ModelResource should generate different output formats and how authentication should look like.
- In-depth planning "on paper".
- Initialize a restapi branch.
- Implement the Resource "interface" class.
- Implement the ModelResource class and an elegant way to return output in at least xml and json.
- Add authentication.
- Write and perform unit tests.
- Prepare the code to be merged to trunk. The Django API will likely change before v1.0. Some of the things that will be improved in a possibly backwards-incompatible way and that might play a role for this project are validation, serialization and the authentication framework.
About me
I'm a 21 year old bachelor student of Cognitive Science at the University of Osnabrück in Germany. When I'm not in Osnabrück, I live in the small town of Dinkelscherben which is located in Bavaria. After finishing school about two years ago I decided to postpone University and spent the next year developing web applications instead. At that time I had two years of Python experience. Django became — and still is — my web framework of choice.
During that year, I spent quite some time making sure I understood how Django works under the hood. I participated in the Django discussion groups, wrote a few patches and submitted a few tickets. My largest Django project to date is the e-learning application Mindpicnic which is fairly popular in Germany.
I have been planning to add a RESTful API to Mindpicnic for about as long as the project exists. I'm committed to seeing this proposal become reality. I believe I would enjoy getting more involved with the Open Source community a lot.
My proposal got accepted, the latest source code is available at Google Code.
Artikel
Andreas,
This is a really nice overview and proposal–I wish you had written it a few months ago! I didn’t know about many of the existing attempts to do REST more easily in Django.
Your ideas sound very close to my current thinking–well, actually, I think you are many stages beyond my thinking at this point. Although I started out just trying to leverage the existing model/generic views code in Django, I have more recently come around to the idea of a more general REST interface that the model/generic view approach would implement, and I had only started thinking about how to configure less model-centric resources, to which you seem to have well formed ideas. I was also trying to think about how to respond in different formats, for which again, you seem to have a pretty well developed ideas for how to do this.
I have been using the Ruby on Rails approach as a source of ideas for evolving my own code, as it seems to be a pretty straightforward and relatively mature implementation. And there might be opportunities for sharing knowledge between the two communities. Otherwise, my main concerns have been using a test driven approach to coding it and trying to figure out how to make it as easy as possible for developers to implement the approach. (That was the other nice thing about focusing on models and generic views, much of it could be introspected automatically.)
For example, I think it would be great to see code like your poll_api example inside a Python module that follows a name/location convention like urls.py and other Django modules, then programmatically build the urlpatterns from that (or at least, have that as an option that can be overridden for more complex implementations) in a way that injects as little code as possible into urls.py and elsewhere so that the code stays clean and people can start using it productively without a steep learning curve or drastically changing how they use Django already. In particular, I think that the urlpatterns are pivotal to making REST work elegantly in Django; because the regular expression syntax is so dense and because there may be so many RESTful urls in an application, this seems like a key place to ease a developer’s burden, by finding a way to generate these automatically from other configuration settings or conventions.
So, as I think my approach is very compatible with what you are proposing, would you like to work together to develop this into a full fledged Django contribution? As your ideas are significantly ahead of mine, and you seem to have significantly more Django experience than I, I would be glad to have you be the project lead. Or, if you want to work on your own implementation alone, but plan to produce working code in the near future, I could also use the time I would have spent developing my code and spend it testing what you produce, filing bug reports and suggestions, generating documentation, etc. For me, the important thing is to learn Django in more depth and begin building RESTful applications, and I’d like to put my energy behind the best approach, even if that turns out not to be mine. I also want to give back to the community in the way that is most appropriate for my level of knowledge and experience.
If you are so inclined, my blog posts, which I link to from my project site, chronicle the process I have gone through so far to develop the code. My last blog post on the subject from a few weeks ago goes into many similar ideas to what you have in your proposal, and the django restful-model-views site itself was updated to reflect those ideas. The code itself is strong in places–its pretty fully unit tested, and some aspects of it are well thought out and well implemented I think, and could be a good starting point for developing the REST API you have envisioned. At the very least, it might be food for thought, such as the approach I took to organizing the view code for example.
If you are interested in working, please feel free to contact me directly.
adam smith
asmith@agile-software.com
http://code.google.com/p/django-restful-model-views/
[…] also feel obliged to point to a blog post that I suspect many people may not have seen: Improving Django: Generic views for RESTful web services. The author’s proposal is very close to what I proposed in my last blog post on the subject, […]
Thanks for your comment!
If my ideas sound similar to your current thinking that’s because I learned a lot reading your blog posts on REST and Django.
Yes, we need to think about URL refactoring ideas such as Nicolas Lara’s before implementing the REST/API proposal. On the other hand, RegEx URL patterns are one of the things I really like about Django. We might end up providing shortcuts for the most common REST URL patterns while leaving the basic URL functionality unchanged.
I want to point out that, although I describe one specific approach that connects models, urls and APIs (the Resource/ModelResource class approach), nothing is set in stone yet. There are probably many ideas I haven’t even considered and once we start writing code, new ideas will inevitably crop up.
I would definitely like to work with you and I have mentioned this in a comment on the SoC website. Collaboration makes a lot of sense: You seem to have spent more time thinking about REST in general, you are a native English speaker (I would not want to read documentation written by myself) and you know things about web development that I don’t (my knowledge of Ruby on Rails is quite shallow).
Hello again and sorry for the late response, I’ve been busier than usual with freelance work, and I just got around to checking your site again.
I’m glad you’re open to collaborating! I think together we will come up with very robust solution. Have you given thought as to how you want to proceed?
(Incidentally, I think your English is excellent–better than many native speakers I know… but don’t hesitate to let me know if I am ever unclear.)
Here are a few preliminary thoughts I have. Your blog post prompted me to make a few quick updates to my existing code to factor out the model specific approach more, so that its now ready (I think) to allow developers to define resources that are not directly tied to models. If you like the basic approach I took, we could use this as a starting point, just let me know if there are areas that you would like clarified, or if there are areas you think should be cleaned up more. I also believe that nothing is set in stone and would be prepared to start from scratch also if that makes the most sense.
Either way, we should probably start a new project site and deprecate the one I started. The Google project hosting seems to be easier to use than Source Forge, but I suspect that you can’t just rename an existing project. Do you have a preference?
My knowledge of Ruby on Rails comes purely as a spectator–I haven’t used it yet myself, although that might happen as part of ongoing research into the ideas they’ve come up with. And as I said, because I am also new to the REST philosophy, and still very new to Django, and because you seem to be more experienced with Django at least, I would be happy to follow your lead on this project. Besides, my best ideas may already be in the existing code, while I think you have a strong vision for how to take it to the next level and beyond.
Also, I have a significant project in the works for which I am planning to use Django, and I could use the developing REST API underneath so we would have that as an extra reality check.
Let me know your thoughts on this. I think I am ready to proceed, even if its slowly at first. I’m really excited about the possibilities here.
Feel free to contact me directly at my email address below.
adam smith
ajs17@cornell.edu
[…] I mentioned a few weeks ago, Andreas Stuhlmueller has proposed a general REST API for Django, and as his ideas and mine seemed very compatible, we began talking about collaborating on a joint […]
Vielleicht solltest du doch “Captchas” einbauen :-)