Logging in the cloud for free with Heroku and Mongolab
A few weeks ago I get very pissed of by the new interface of Delicious and by their way of handling the switch to (another) new owner (they lost my bookmakrs and found then again, then they lost my password but where unable to let me reset it for two days…). So I developed No Fuss Bookmarks as a personal replacement of Delicious – I'm not teaching this semester (long story), so I have some spare time to experiment with things I've lurked at for ages and learn new stuff.
Without going into the details of such application (I'll reserve for a forthcoming post), following the suggestion of a very knowledgeable friend of mine, I decided to deploy it using Heroku. Once it was in production (I'm proud to have four users, feel free to join!), I decided that I wanted to keep track of some logs, both "application" logs (so to say, the one generated using the Python logging module), and "web" logs (the usual Apache style access log and error log).
One of the beautiful things of Heroku is that it provides many incredible addons addressing a wide variety of needs an application can face; all of them are straightforward to use and most of them come for free (at least for "developer" usage quotas). The Loggly addon looked very promising, even though the free quota (allowing for a single day of log retention) seemed a bit to restrictive. Since my applicaiton already used mongodb as a databse (kindly offered for free and nicely integrated as an Heroku addon by mongolab), I decided to pursue the MongoDB is Fantastic for Logging hint I found on the mongodb blog itself.
In the following part of this post I will describe at high level the bits of code needed to endow my application with cloud logging; you can look at the very simple heroku-log4mongo for a detailed, completely worked out example.
The key piece of software is the very nice log4mongo-python library that bridges quite easily Python logging and mongodb (by the way, there are log4mongo variants for many logging API of different programming languages).
Since log4mongo-python does not support mongodb URI, that on the other hand are the way Heroku store your database access configuration, one need a bit of code to get the URI from the environment and parse it; nothing more than standard library functions though
from os import environ
from urlparse import urlparse
MONGOLAB_URI_PARSED = urlparse( environ[ 'MONGOLAB_URI' ] )
MONGOLAB_CONF_DICT = dict(
host = MONGOLAB_URI_PARSED.hostname,
port = MONGOLAB_URI_PARSED.port,
database_name = MONGOLAB_URI_PARSED.path[ 1: ],
username = MONGOLAB_URI_PARSED.username,
password = MONGOLAB_URI_PARSED.password
)
Once you have parsed the URI, you can configure a logger to use the database it denotes for instance with
from logging import getLogger, DEBUG
from log4mongo.handlers import MongoHandler
logger = getLogger( name )
logger.addHandler( MongoHandler( level = DEBUG, collection = 'application-log', **MONGOLAB_CONF_DICT ) )
and your log messages to logger will go to the databse.
My application (as many on Heroku) is served by the gunicorn webserver; to convince it to send its access and error log to mongodb was a little trickier.
First of all it seems that you need to "enable" logging from the command line (I was not able to find a way to do this programmatically, but perhaps I was not looking hard enough). For such purpose, I run it as
gunicorn <MY_APP> --logger-class=GunicornLogger --access-logfile=/dev/null --error-logfile=-
where <MY_APP> is the WSGI application endpoint and GunicornLogger is the following class
from gunicorn.glogging import Logger
from log4mongo.handlers import MongoHandler, MongoFormatter
class GunicornLogger( Logger ):
def __init__( self, cfg ):
super( GunicornLogger, self ).__init__( cfg )
access_handler = MongoHandler( level = INFO, collection = 'access-log', **MONGOLAB_CONF_DICT )
error_handler = MongoHandler( level = INFO, collection = 'error-log', **MONGOLAB_CONF_DICT )
self.error_log.addHandler( error_handler )
self.error_log.setLevel( INFO )
self.access_log.addHandler( access_handler )
self.access_log.setLevel( INFO )
This is a kind of monkey patch, depends on the actual implementation of gunicorn.glogging.Logger as found in version 0.13.4 and not on any public APIs, so it may stop woring in future releases.
To retrieve your logs, something very simple (and rough) as this will suffice
from os import environ
from urlparse import urlparse
from pymongo import Connection
MONGOLAB_URI = environ[ 'MONGOLAB_URI' ]
da = Connection( MONGOLAB_URI )[ urlparse( MONGOLAB_URI ).path[ 1: ] ]
for entry in db[ 'error-log' ].find():
print entry[ 'timestamp' ].as_datetime(), entry[ 'message' ]
for entry in db[ 'access-log' ].find():
print entry[ 'message' ]
for entry in db[ 'application-log' ].find():
print entry[ 'timestamp' ].as_datetime(), entry[ 'message' ]
As a concluding remark, a very nice feature of mongodb are capped collections (a very high performance auto-FIFO) that can be used as a straightforward way of controlling the size of stoder logs.