ReSTafarian

“Advertising leads to Portalisation”

Any website that displays advertising will be drawn towards portalisation of their properties in a misguided attempt to maximise advertising revenue at the expense of user needs and goals.

I’ve just spent the evening exploring the idea of asynchronous event driven architectures for building scalable web applications. The idea is to break the traditional request/response cycle that traditional web app development contrains you to by moving the vast majority of the application into a series of separate, fault tolerant components. The heart of this is a rather nifty library called ZeroMQ (or 0mq). ZeroMQ is a library that abstracts away a lot of the messy details of BSD socket management, leaving you with a nice, familiar layer that allows you to focus on the way you want your networked application to talk with other networked applications. I wont go into too much detail here about ZeroMQ as the official website explains things much better than I ever could.

Mongrel2 is a ZeroMQ enabled web server. We can use Mongrel2 as a connector to our archipelago of inter-networked applications. For example, Mongrel2 can connect to our handler app pr push requests to it. We can process those requests, fire them off into our event driven system asynchronously, and respond to the user quickly. In situations where the processing may take some time, we can return a HTTP 201 Accepted, and continue in the background, firing off events as we process the request for other components to pick up and process. Add to this GO’s awesome go routines, we can get some pretty cool multi threaded, fault tolerant, load balanced systems built, all with a traditional  HTTP front end.

Recently I hit my first real problem with Ubuntu Server 8.04: a lack of a compiled apr_dbd_mysql. This dbd driver provides apr_dbd access to mysql, but for some reason (which I believe is due to the incompatibility between the GPL and Apache license) its not compiled by default. My real dread was having to compile Apache from scratch, which, whilst is a pretty painless experience on OSX, I really am relying on Ubuntu to handle a lot of the basic stuff for me, as I’m still not comfortable to go spreading files all over the place

However, to the rescue was a sequence of instructions researched by Francois Pesce, which enable you to recompile only apr-utils with mysql enabled. You can find his instructions at http://jok.is-a-geek.net/jblog/index.php/post/2009/02/12/How-to-cleanly-build-apr_dbd_mysqlc-on-Ubuntu-Server. But as it took me a good week to discover his blog, Francios has kindly given me permission to reproduce these steps here, in an effort to increase the footprint for this solution. So here they are:

$ sudo apt-get install devscripts build-essential fakeroot
$ apt-get source libaprutil1
$ sudo apt-get build-dep  libaprutil1
# The apr-util-1.2.12+dfsg name is susceptible to be different according to updates.
$ cd apr-util-1.2.12+dfsg
# Change one parameter in the configure script command line
# To add a --with-mysql in the following line
$ vi debian/rules

$(CONFFLAGS) ./configure --host=$(DEB_HOST_GNU_TYPE) --build=$(DEB_BUILD_GNU_TYPE)
--enable-layout=Debian --with-apr=/usr/bin/apr-1-config --with-ldap=yes
--with-dbm=db46 --with-sqlite3 --with-pgsql=/usr  --without-gdbm
--without-sqlite2 --with-mysql --with-berkeley-db

NB: In my case, the full version number for this package was 1.2.12+dfsg-3,
thus I’ve incremented the revision from -3 to -4 in order to update cleanly
the currently installed package 1.2.12+dfsg-3

# Add a message, and change the revision number into changelog and files :
$ vi debian/changelog

apr-util (1.2.12+dfsg-4) unstable; urgency=medium
* Add mysql support
 -- Francois Pesce <francois.pesce@gmail.com>  Wed, 11 Feb 2009 23:51:32 +0100

$ vi debian/files

libaprutil1_1.2.12+dfsg-4_i386.deb libs optional
libaprutil1-dev_1.2.12+dfsg-4_i386.deb libdevel optional
libaprutil1-dbg_1.2.12+dfsg-4_i386.deb libdevel extra

NB: If unit tests failed, as it was the case for me, just comment out the C
code line from the incriminated test suite. (In my case, I have commented out
reslist test like this:

//    abts_run_test(suite, test_reslist, NULL);

# Launch the rebuild
$ debuild -us -uc
# Now the package should have been created in upper directory.
$ cd ..
$ sudo dpkg -i libaprutil1_1.2.12+dfsg-4_i386.deb
$ sudo dpkg -i libaprutil1-dev_1.2.12+dfsg-4_i386.deb

Nice one Francois!

Recently I’ve been working on my own web service. I don’t want to go into too much detail about the service itself, but I do want to talk about some of the custom modules I’ve been developing in order to make developing ReSTful web services a lot easier. The first is an Apache module call mod_authz_resource.

Mod_authz_resource (a terrible name I know, it will have to do for now) is an Apache module that allows developers to prevent access to resources due to the permissions granted on the HTTP methods the request contains. For example, say you want to provide the world access to a resource your web service provides, but want to limit access to the PUT and DELETE methods to those users who are a member of a specific named group; the Administrators group.

Now, you can do this currently in Apache quite simply using the Limit directive, but I’ve found this to be quite brittle, and doesn’t support dynamic resource creation easily (say, when a user accesses a publicly available resource with a POST method to create a new resource). I wanted to provide a similar level of granularity in access as on UNIX or LINUX, with the owner, group and world access restrictions, whilst providing a database back end for manipulation of these permission structures.

Mod_authz_resource is an authorization module, which means it works with the existing Authentication modules provided with Apache, so you can stick mod_authn_digest and mod_authn_dbd on your server to authenticate the user against a database over HTTP digest, and then use another (or the same) database to determine the permissions and groups this user belongs to.

The module has been implemented using the Provider pattern in Apache module development, meaning that like the authentication modules in Apache, its back end can be driven by a database, file, LDAP etc. Currently I have implemented a file based and database provider. The database provider used the apr_dbd library, so any Apache apr supported database driver should work out of the box, meaning MySQL, PostGreSQL, SQLite etc.

Each back end provider must implement the mod_authz_resource.h interface of two functions: get_groups_by_username() and get_permissions_by_uri(). The first function, get_groups_by_username(), returns an array of groups the that the username, provided by the authentication module in play, is a member of. These groups are used to determine if the user is a member of the resources group. Secondly, the get_permissions_by_uri() function returns a structure of permissions for the owner of the resource, the members of the resources group and the rest of the world.

This provides a very granular permissions system for resources. For example, we can say the owner of the resource has the username ‘bob’, and the resources group name is ‘admin’. If the user is ‘bob’, they are the owner of the resource, and may HTTP DELETE the resource, whilst members of the group ‘admin’ may HTTP PUT to the resource, whilst everyone else may GET and POST to the resource.

You can specify multiple allowed HTTP methods for each permissions level, and permissions are inherited up the stack (for example, we the owner also may GET and POST to the resource, as well as the group members, but in addition to this, the owner also inherits the group privileges as well).

Hopefully, this module will free your web service code from having to manage authorization. That is, your PHP script will only  execute its GET method handler if the authenticated user has authorization to. This should simplify the development of web services by moving this logic out of your code.

Finally, there are some additional features of note. Mod_authz_resource can be moved to an authentication proxy, sat infront of your backend web servers. Again, this allows you to draw a clean line of seperation from your web service code and the authorization code. In fact, this module has been written specifically to act as part of a load balancing authenticating reverse proxy for my own web service.

Another feature is that world permissions allow authentication to be optional. That is, if the world has permission to GET your resource, they will not be presented with a authentication challenge. Only those user attempting to access methods restricted by owner or group will be challenged.

The file provider reads from existing Apache authorization group files, and is fully compatible, so adding the permissions layer on top will allow you to use your existing group files.

It supports OPTIONS requests, returning the methods allowed by the user (again, based on authentication).

And finally, for successful authorizations, all the users authorization details will be passed through to your backend service as environment variables or x-headers, including the list of allowed methods, the users access level (owner, group or world) etc; so should your backend web service code need to know, it can (for example, if you want to display additional data for the owner or group members, but not the world).

I’m currently having some success compiling the modules over two different architectures (OSX Tiger and Ubuntu 8.04) and hope to have them tested properly, but the eventual goal is to get them out and open sourced ASAP.

If you’re interested in these modules, drop me a comment below and I will answer your questions.