Saturday, February 12, 2011

Insights from ‘The Design of Design’ - Part I

I have good fortunate of meeting/interacting with some great software designers while working in Geometric Ltd and during last two years as independent consultant. I am always intrigued by how an expert software designer thinks and how he learns. As Sir Ken Robinson says the key skill in today's world is ‘knowing how to learn new things’. I think of myself as ‘Thinking Craftsman’ (i.e. someone who is thinking about his trade/craft and strives to continuously improve his/her skills). Hence ‘how an expert designer learns and becomes an expert’ is a key question for me in the quest of improving my own skills.

The Design of Design’ is a new book from Fred Brooks (Author of Mythical Manmonth). As expected, it has some great/some obvious insights but most importantly it has great explanations of these insights. In this article, I am going to discuss about insights related to how ‘expert designers become experts’.

Insight One: Exemplars in Design
This is what Fred Books says about 'exemplars'

Exemplars provide safe models for new designs, implicit checklists of design tasks, warnings of potential mistakes and launching pads for radical new designs. Hence great designers have invested great efforts in studying their precedents. Bach took a six month unpaid leave to study the work and ideas of Buxtehude. Bach proved to be much greater composer but his surpassing excellence came from comprehending and using the techniques of his predecessors and not ignoring them

I argue that great technical designers need to do likewise but that the hurried pace of modern design has discouraged this practice. ... Technical design disciplines eager to produce great designs need to develop accessible bodies of exemplars and knowledgeable critiques of them (page 154-155)
Certainly lazy or slack designer can minimize his work by picking an exemplar and just modifying it to fit. By and large, those who just copy do not draw on ancient or remote exemplars but only on those that are most recent and fashionable’ (page 162).
There are two things that came to my mind after reading this article. First is ‘Design Patterns’. “In software engineering, a design pattern is a general reusable solution to a commonly occurring problem in software design. A design pattern is not a finished design that can be transformed directly into code. It is a description or template for how to solve a problem that can be used in many different situations.”.

One purpose of documenting the ‘design patterns in software’ is to 'make experts insights available to a novice’. In this sense, ‘Design patterns’ fit perfectly into what Fred Brooks calls ‘accessible bodies of exemplars’. Personally my thinking about the software design changed after I studied patterns (especially advantages and limitations of each pattern). Even today I read and reread GoF Design pattern book, articles by Robert Martin, articles and books of Martin Fowler, books like Effective C++ and More Effective C++. Every time I gain some small new insights which enrich my own ‘body of knowledge’. Another recent good book of exemplars is ‘Beautiful Code: Leading Programmers Explain How They Think’. There are positive and negative reviews on this book. For me, this book is invaluable for its insights into how various developers think about a problem and how they come up with a solution.
Second thought was ‘about Googling’. Recently (past few years) I see many developers just google about a problem, find something and copy that code. Many times they just pick up some pattern and copy the sample code for that pattern found on the internet. But since they don’t have any real understanding of the pattern, they end with more problems. While conducting programs on ‘design patterns’ I see participants eager to get sample code rather than eager to understand the pattern, participants eager to get the ‘power point slides’ rather than reading the books and articles. So far I have not found any solution to this ‘lazy or slack designer syndrome’. Bigger problem is many of these lazy/slack designers are considered ‘good/great designers’ in their company because they can rattle of latest technology and design buzz words.

For many years I regularly read blogs, various book and studied how other good designers think out of habit.But I could not clearly explain/articulate a new comer why I am doing this. Now I know how this habit has helped me and how it can help a new comer to study his craft.

Saturday, February 05, 2011

Optimizing Django database access : some ideas/experiments

Originally published on BootstrapToday Blog  

As we added more features to BootStrapToday, we started facing issues of performance. Page access was getting progressively slower. Recently we analyzed page performance using excellent Django Debug Toolbar and discovered that in worst there were more than 500 database calls in a page. Obviously that was making page display slow. After looking at various calls, analyzing the code and making changes, we were able to bring it down to about 80 calls and dramatically improving the performance.  Django has excellent general purpose caching framework. However, it’s better to minimize the calls and then add caching for remaining queries. In this article, I am going to show a simple but effective idea for improving the database access performance.

In Django, a common reason for increased database calls in ForeignKey fields. When you try to access the variable representing foreign key typically it results in a database access. Usually suggested solution to this problem is use of ‘select_related’ function of Django query set API. It can substantially reduce the number of database calls. Sometimes it is not possible to use ‘select_related’ (e.g. you don’t want to change how a third party app works or changing the queries requires significant change in the logic etc).

In our case, we have Foreign Key fields like Priority, Status etc on Ticket models. These fields are there because later we want to add ability to ‘customize’ these fields. However, the values in these tables rarely change. Hence usually this results in multiple queries to get the same data.  Usually these are ‘get’ calls. If we can add a simple caching to ‘get’ queries for status, priority etc, then we can significantly reduce the number of database calls.

In Django, a new connection is opened to handle a new request. Hence if we add model instance cache to ‘connection’ then query results will be cached during handling of one request. New request will make the database query again. With this strategy we don’t need complicated logic to clear ‘stale’ items from the cache.

from django.db import connection
from django.core import signals

def install_cache(*args, **kwargs):
    setattr(connection, 'model_cache', dict())

def clear_cache(*args, **kwargs):
    delattr(connection, 'model_cache')


class YourModelManager(models.Manager):
    def get(self, *args, **kwargs):
        Added single row level caching for per connection. Since each request
        opens a new connection, this essentially translates to per request
        model_key = (self.model, repr(kwargs))

        model = connection.model_cache.get(model_key)
        if model is not None:
            return model

        # One of the cache queries missed, so we have to get the object from the database:
        model = super(YourModelManager, self).get(*args, **kwargs)
        if model is not None and is not None:
        return model

As a side benefit, since the same model instance is returned for same query in subsequent calls, number of duplicate instances is reduced and hence memory foot print is also reduced.