Ideas on on sessions /courses as Prof of Practce for Chandigarh University

 Sometime back I posted an update on LinkedIn about joining Chandigarh University as Professor of Practice.   

I am joining CHANDIGARH UNIVERSITY Computer Science dept as "Professor of Practice". This is part time. I plan to spend about 2 weeks every semester on university campus. And also want to do some online lectures/workshops. There are no particular time commitments. It more about Industry + Academia partnerships and sharing the experiences of with Students.

I also asked about ideas on sessions/courses that students will be interested in. I received lots of comments and suggestions. Here I am collecting all the ideas/suggestions at one place. Over time, I will keep on posting updates whenever I implement an idea/suggestion.

Nitin's Ideas

1. Few sessions in current academic topics related to software architecture and design but with sharing of experiences.
2. A 2 week hands-on workshop for 2nd/3rd year students on "professional software development" starting from using version control, and how to write a spec doc to coding, testing, etc.
3. A 1 week workshop for student teams who want to participate in hackathons.
4. Few sessions on diverse topics like 
• Algorithms in Daily life, 
• Costliest bugs in history of software, 
• Women Pioneers in Computer Science, 
• Basics of Cascading Style Sheets (CSS), 
• Hidden Life of a Google Search Request 
• (re)Inventing the Assembly Language.
• Do/Doing/Done is NOT Kanban
• Using Version Control Effectively

Ideas from Harish Walke

1. few topics around "Agile" too! Fundamentals and basics of Agile
• Typical day or sprint in the life of a Developer
• built-in Quality and what it means in Agile
• Scaling agile 
• Some useful concepts of Value streams, Lean Portfolio, Strategic Themes, WSJF for prioritization, Real life examples of Epic, Capabilities, Features
2. Topics related DevOps
• DevOps with CI/CD.
• TDD, BDD. Different compliances and their inclusion in Features as NFR (Non Functional Requirements)
• SecDevOps with mindset of "Built-in Security"
3.  Innovation mindset related
• Lean Startup, 
• Design thinking, 
• Think Wrong (Don't find solutions for the problems that do not exist).h
4. Use of Tools while doing the course projects
• Task or backlog management tools
• version control tools

Ideas from Anirudha Raste

1. OO eyes!
   Look around you, in your real life to find many OO concepts. People think all OO and other software concepts were 'Created' by software engineers and architects. Whereas, they are already present in real life. Need to see them in software way! Then one will find that software is not so 'alien' to real world.
2. 'Importance of logging while troubleshooting problems at customer end, where you do not have access to customer's environment, customer data, debugging

Ideas from K K George (KKG)

The importance of unit testing and show them how to measure coverage of the unit tests.

Ideas from Manish Ramrakhiani   

• Agile principles
• Rise of Data and related roles in the Data Engineering/Analytics space
• Sharing some of real successful archicture/design of your favorite projects (Nitin's comment ; I was also thinking on these lines. So probably will do this early)

Ideas from Nikhil Karkare

• Automation: automating testing, RPA
• Would sound basic, but using powershell or bash for basic scripting (using awk for example) - (Nitin's comment :  In my opinion, absolutely needed)
• Using any code editor to its full potential (Visual Studio / Replit)

Ideas from Laxmidhar Gaopande

 Some projects split in small teams to work on for 2-3 months after your sessions and review by making them presentation to you along with all.  (Nitin's comment : CU already have a concept of Project Based Course, where students learn by doing a project rather than attending lectures. Will be good ideas to incorporate these in the same framework)

Ideas from Vivek Gupta

1. code readability/ maintainability, 
2. logging/eventing/traceability, log analysis (Nitin's comment : Also suggested by Anirudha Raste)
3. ease of troubleshooting, 
4. importance of code reviews, 
5. threat modeling etc. 

Ideas from Surjit Patra

Go with very basic and core topic like Aniruddha Raste said OO things. But also go 1 level below 

• "How to realize or relate the software way of thinking to a real problem". 
• Or How to visualize the problem, is it from outside/ inside. 
• What is the best approach to find the possible solutions.
•  How to pick the best one out of it. 

If you have any other ideas, please add in comments

Where are the 'average engineer students' ?

I am pondering on this question for last 2 months. I will really like to hear from more people.

I work with Smart India Hackathon and was mentor to teams in Smart India Hackathon and Singapore India Hackathon. Here I work students who are enthusiastic and ready to learn. They are ready to experiment, they are confident, they are not afraid to fail initially to succeed later. Its a joy to work with them.

Obviously they are high in demand in job market and command premium for salaries. Recently I was recruiting for my startup and salaries of many of these students do not fit in our budget. Also lot of our work do not require high end talent. We are looking for students good in attitude and programming aptitude but may not be the best. We are also looking for interns.

For internship, we ask students for solve some small programming problems. I got 40+ responses from 2-3 colleges. 35+ students have copy/pasted the code from internet. Come on guys, I am using google since before your birth. I can find from where you copy/pasted your code in 5 mins. I still talked with 3-4 students and realized that they have not taken any efforts to understand the code. They don't want to take any efforts. I have similar experiences of interacting students with few colleges from Pune (not naming the colleges). These students are essentially at the bottom and practically unemployable.

Where did the average students disappear ??? Expected "normal" distribution will mean small set of students at both extremes and large number of average students in middle. However, now we have 'skewed' distribution with really large number of unemployable students at one end and small number of top notch students at another end and nothing in between. How did we land up in such a bad situation ??

Manual Code Review of every single change is OVERRATED

Many software companies have this policy that every change has to be manually reviewed and approved by some peer (or senior) developer either before it is commited or before it is merged in main development branch. This practice is recommended by many software luminaries. However, when implemented in a project/product, I found many practical issues with 100% manual code reviews and reached a conclusion that "manual code review of every single change is OVERRATED"

Over years, I realized that manual code review of every single change is OVERRATED. It does not appreciably reduce post shipment defects but does introduce many inefficiencies in your development process. Manual Code Review of every single change usually have -ve ROI. I find that nobody writes about bad side effects of forcing developers into 100% manual code reviews. Recently I found one article and that triggered me to write this blog post.

So here is the list of problems that I found in 100% Manual Code Review process

Problems in Manual Code Review of every single change

1.  ROI on Code Review - 
   
   Is reviewer finding kind bugs in code review such that if they remain undetected will take more efforts than the time spent in Code Review ? If yes, then ROI of Code Review is +ve else ROI of Code Review is -ve.  Most of the Code Reviews have -ve ROI.  
   Reviewer may be a good developer but he may not be really good or may not have an 'eye' for defect. Some years back I did an informal study of 'code review' comments reported in my company's code review system. Majority of the code comments were about 'coding style' or 'comments' or 'naming conventions'. Very rarely a serious bug is detected. Effectively Such code reviews have a '-ve ROI'.
2. Reviewers time and attention.
   
   Typically Reviewer is mostly one of the senior developers. He/She also has their own commitments on new feature development. Usually these commitments take priority over code reviews. Hence Code Reviews remain stuck in queue and total feature development time increases unnecessarily (elapsed time from start to feature is delivered in end user's hand).
3. Team's Most productive developers are 'reviewing' rather than 'writing new code'.
   On the other hand, if Senior developers spend majority of their time doing reviews, then 'most productive developers' are effectively not writing any code. Either case results in lower productivity for the entire team.
4. Lower developer productivity while fixing bugs reported during code reviews.
   Code review is helps only if review happens within short time writing the code (preferably within a day). However, in general actual review happens after many days. In this time, developer loses the context of code. When finally the code is reviewed and he/she has to fix something in the code, it takes longer because now developer has to 'get in the context' again. Effectively it lowers the developer's productivity again.

Over year's almost every team that I have worked with has made this mistake and effectively reduced the productivity. How do we fix this ? Whats the alternative ?

How to regain the productivity lost by "manually review every code change policy" ?

1. Institute "Zero Warnings" policy for your code.
   This is easiest and most effective way to eliminate bugs. Every new version of compiler improves in ability to detect potential problems and report them as 'warnings'.  Set your compiler to 'highest warning' level. If you are using Visual Studio, turn on 'report warnings as errors" setting. Fixing warnings is 'practically no-brainer'. Warnings have to fixed immediately. DO NOT wait till end of sprint or end of release to fix warnings.
2. Take advantage of Static Analysis Tools for your programming language.
   Every programming language now has static analysis tools (linters, tools like Klocwork, Coverity etc). This Wikipedia article will give you a list of static analysis tools. Make static analysis part of your 'CI/CD' pipeline. Make sure that static analysis bugs are fixed within day. DO NOT wait till end of sprint or end of release for fixing static analysis errors.
3. Identify your critical files and review only those manually.Stop reviewing very single line of change. Identify your critical files (use complexity analysis, analyze version history to find most frequently changed files etc).  Usually 80-20 rule applies 20% are your critical files. Mark those files in system. Any commits in these files should trigger your manual code review process. Manually review these critical files ONLY after warnings and static analysis bugs are fixed.

Development processes/practices followed in a typically software development team usually have serious inefficiencies. By using 'common sense' (rather than following common practices) it is possible to achieve an 'order of magnitude' (2x to 10x) improvement in the team's productivity and delivery quality. It DOES NOT require a change in methodology (like move to Agile) or any new expensive tools. It requires some disciple and taking advantage of many low hanging fruits. The results are usually visible in about 3 months. 

[Shameless Plug]If you want to know how to improve your team's productivity by 2x to 10x, I am available for 'consulting/coaching'.

References

1. Code Reviews Do Not Find Bugs - How the Current Code Review Best Practice Slows Us Down
2.  

Things you should do after joining your first job

This blog post is for my friends who recently graduated and now joined their first job. 

You have now joined a company and started working. Your responsibilities have now changed. You are now an earning member of your family. The habits that you develop in this first year will determine your future (in your career, in your earnings, etc). So what are the first things you need to do ?? 

Given below is a partial list of things that you need to do immediately or start doing regularly.

1. Buy a medical insurance. 
   You will already be covered in your company's medical insurance or you may be covered in your parent's medical insurance. If you are covered in parents medical insurance, that cover may now be over.  Also it is unlikely that you will stay in the same job. If you change that job, start freelancing, may be between the jobs then you may NOT have medical insurance cover during that period. As you become older, it will be more difficult to get medical insurance. So Buy a Medical Insurance on your own. See if you can get 'family insurance' plan. Add your parents to this insurance.
   
2. Start Investing and building your Financial Portfolio
   Don't spend all your salary on parties and malls. Learn the basics of investing in Mutual Funds, Shares and other financial 'instruments'.  Few years later when you want to buy a house, get married, go to honeymoon, do it with your own money. Don't depend on your parents.
• At bare minimum start creating Fixed or Recurring Deposits in your bank account. 
• All investment companies like ICICI Direct has short courses  that introduce you to basics of investing. Attend those courses.
• Register for SIPs (Systematic Investment Plans) for Mutual Funds/Shares. SIPs will allow you to start investing in small amounts but regularly.
  
3. Start Investing and Building your Knowledge PortfolioBuilding your knowledge portfolio as more important than building your financial portfolio.Sometimes your company will reimburse some of these expenses. Remember The money you spend on books, courses etc are NOT expenses. They are investments. Invest in learning. These investments pay off many times over.
• Buy Books. Don't depend on company or other libraries. You must create your own library of favourite books and books critical for your work. For example, if you are in a programmer you must have your copy of 'Code Complete2'
• Join online courses like Coursera, EdX, Udacity. Pay for those certificates
• Attend conferences. 
• Join Professional Associations like ACM, Computer Society of India, Stepin Forum and meet-ups
  
  
4. Start Investing in your Communication Skills and Business Etiquette
   Good communication skills are critical for your career advancement. One of my mentors gave me a critical advice at the start of my career. He said
   

   "Nitin, what you know is important. But who knows what you know is EQUALLY important"

   "Who knows what you know" gives you opportunities. For that to happen you must be able to clearly articulate your thoughts and ideas.
   
   
   • Practice to write good professional emails and documents. Don't write 'bcos, yo, etc' (i.e.typical sms/whatsapp slang) in your official mails, mails to your boss or colleagues.
   • Practice to speak fluently in your mother tongue and in English. Consciously avoid saying 'You know' or 'ya' in every sentence. 
   • Join organization like Toast Masters where you are practice your presentation skills
     
     
5. Learn to deal with A**holes
   You will find some a**holes in every organization. You have to learn to deal with these on your own. Your parents, brothers/sisters etc are not going to help you now. You will encounter 'passive aggressive bosses', 'vining co-workers', people trying to emotionally blackmail you for various reasons/agendas.
• Read books of Prof. Bob Sutton (No Asshole Rule,  The Asshole Survival Guide: How to Deal with People Who Treat You Like Dirt )
• Learn some meditation. You can do it on your own. You don't have to attend some fancy/expensive class. Checkout Quiet Kit
• Develop your own support system - Your friends, colleagues with whom you can talk and share your problems, who can help you with suggestions and intelligence.
  
6. Find a Mentor
   Your first team/Manager has a HUGE and disproportionate impact on your career. The habits you develop in your first project/team/job, stay with for life long. Make sure you develop 'Good habits'. Make sure to work in a team/manager who will help you learn and develop the good habits.
   
   Check this comment from Prashant Kale. I fully agree with this advice
   
   Do you feel you can reach to the expertise level of your manager/current mentor in 3-4 years? THEN you have 'bad role model'.  Find out someone where you must feel "To reach to level him/her, it will take me 10 years'.  When you find such person, stick with him/her. Stay continuously in touch with him. Meet him/her regularly. Get his/her advice. Discuss your problems/career choices/aspirations openly with them. 
   It is really really difficult to find such Gems. When you find one, make sure you don't ignore or lose them. (NOTE : I was fortunate that I found many such Gems in my initial years in Telco and In Geometric Ltd. It shaped my career. I still keep in touch with my mentors.)
   If you feel your current mentor/team is not adding value to you, change the team (may be even the company). 

If you have any other suggestions to add to this list, please write a comment.

[Update]  I posted this article on LinkedIn. There are many thoughtful comments and advice posted  on this article. Check the replies and suggestions on LinkedIn 

May be Indian Software Developers are just not good in programming

 Today I was reading an article 'Maybe today’s Navy is just not very good at driving ships' about problems by US Navy in light of two recent ship collisions.


What stuck me the how similar the problems are with the problems faced by the Indian Software Industry.

1. Compromising on training of new comers (in the name of saving cost) and assuming that they will learn 'on job'. Typically these guys then screw up 'on the job' which has much higher cost. But this cost is never really visible. It remains hidden behind individual project failures/cost.
2. Project Managers and Team leads afraid to go to seniors/mgmt with anything that might look bad for them. They did everything to protect their own reputation to ensure that they get promotion. 
3. Environment is not about becoming competent software developer/project manager but about looking out for yourself.
4. Decisions made for saving money have now “posed more leadership, logistical and administrative problems,” 
5. (Navy) culture rewards checking the box on qualifications, passing inspections and stacking resumes with career-minded assignments.

For short term company gets 'reduction in cost and hence better profitability'. In Long term company is doomed. (growth stops, good people start leaving, company is sold, etc).

Unfortunately in Indian context this is not just problem of few companies. Its a 'pandemic'.  That is why we (i.e. entire Indian software industry especially small companies) are in BIG trouble.

 

We desparately need lot of Good Passionate Coders. But we also need companies which really really value these passionate programmers.

I cannot do much about companies really giving importance to core coding skills. However, from Sept I will do my bit to develop the next generation of Passionate Programmers. If you want to know how, read about the Concept of CodeGym.

Fast track programs for managers and developers - Selection and First Quarter

When we announced the program we got almost 150 applications for this program. We wanted about 20 participants in this program. So we have to short list our 20 selected candidates from 150 applications.  We used following selection process

• Online test composed of General Intelligence Test, Analytical ability test, English comprehension test.
• About 50 candidates were selected from the online test. 
• These 50 candidates have to write an essay about 'things they will like to change' in the organization.
• These candidates have to undergo an interview by 2 interview panels. 
• About 25 candidates were selected based on interviews and their essay.
• These candidates had to  undergo final interview and in the end we selected 18 candidates.

First quarter was difficult. Participants were also somewhat confused and we were also learning about how to run such program.

First quarterly meeting changed lot of things. The participants started to gel into a team. We did sessions on basics of business finance, mind-mapping, creativity techniques, software design principles, basics of PLM systems, etc. 

They have also started on online management certification program by McIntire School of Commerce.  This program covered the theory while the candidates were getting a hands-on experience of all the concepts that they were learning.

One key component of Quarterly Meet was 'lunch with senior management'. Every quarter we arranged a lunch with CFO, COO, Head of HR and other senior management team members. This way candidates were getting direct access to senior management team and they were able to ask questions/discuss and understand their thoughts behind various company initiatives and ideas.

Every quarter we evaluated the candidates on various parameters. Anyone with C grade in two consecutive quarterly is to be removed from the program. So far (almost 6 quarters are done) we have not removed anyone and every quarter we are increasing the baseline

Part 1 of this article

Fast track programs for managers and developers - Background

Last nine months I am mentoring/working with a team of 17 extremely enthusiastic people in a program we call 'Accelerated Techno Manager' program. It is an amazing experience/journey. Hence I thought I will write about these experiences.

Many companies have 'fast track' programs for their employees. Usually it means company identifies potential 'high performers' and then groom them someway. Nine months back we started a fast track program in Geometric Ltd. I took initiative in defining the program and I am now mentoring 17 potential candidates for last 9 months. I thought, I will write about general structure of program, how the program started, what benefits I see. I hope it will help others in starting/running similar program. If you have similar programs in your company, please share your experiences.

The fast track program is called 'Accelerated Technical Manager' program (ATM program, pun unintended).  Program is inspired by Google APM program or TAS (Tata Administrative Services) program in Tata Group. The content and structure and especially scale is lot different than these programs. 

As the name suggests the program is designed to develop 'technical project manager'. Manager who are experts in project management but also understand software development and technology and can teach a thing or two to engineers/developers working with them. They are multi-talented managers who are good at all aspects of project management, (customer communication, tracking, estimation techniques, negotiations) but are also good at problem solving, leadership, critical thinking and innovation techniques, strategy, strong software development and/or engineering fundamentals, etc etc. 

It is designed to be 'tough program' of 2 years. It is targeted towards 2-3 year experienced employees. The selection criteria is tough. There is an evaluation at every quarter. Every quarter based on performance evaluation some participants can be eliminated. It is expected that about 50-60% will get eliminated at various stages and only 40-50% will pass. The participants who pass will become Managers (i.e. they will get promotion in 2 years that usually takes 8 to 10 years ).

The program includes one year online Management Certification Course by McIntire School of Commerce paid by the company. The program includes a 3 day meet every quarter. There are class room sessions by internal and external faculty. During these 3 days participants get a chance to meet and interact with senior management team (e.g COO, CFO, Head of HR, etc).  Overall program consists of four Phases. Each phase consist of ‘on-job’ mentoring and formal class room trainings

1. Phase 1 (6 months) - Focused on Problem Solving and Software development skills
2. Phase 2 (3 months) - Demonstrating Fast learning skills in a new domain or role
3. Phase 3 (9 months) - Project management skills and Mentoring from Sr. Mgmt team and rotation in various functions.
4. Phase 4 (6 months) - Deputation as 'assistant project manager' in some project.

We selected 17 participants from different locations and domains from over 150 applications. We have completed the Phase 1 & 2. And I can see serious positive change in the thinking of all participants.

In the next part, I am going to write about preparation for program and selection process, experience of first six months.

Introspection : am I really a good programmer ?

Sometime back Prateek Jain posted a link to an article title 'Signs that you're a good programmer' on Geometric's internal portal. It has list of 'signs' that you are a good programmer. It made me introspect and see how many signs actually apply to me. Turns out that by this checklist, I am a reasonably good programmer. :-)

I realized I have done following from the list.

1. side projects.
2. Dabbling in other programming languages, especially ones from a different "family".
3. A tendency to suggest wacky and unrealistic solutions in meetings.
4. Willingly throws away weeks or months of work in order to adopt another programmer's superior code.
5. Refers to it as "the code" rather than "my code", unless accepting blame 
6. Doesn't take the spec by its word and tries to find out who wrote it and what they were thinking
7. Owns a book written by a guy called Martin Fowler. (Actually I own multiple books by Martin Fowler)
8. At least 10% or more of their commits reduce the line-count of the project without adding new functionality
9.  Shoves through a crowd at a party to get near someone who just used the word "Bayesian"
10. Envies but doesn't resent people with degrees in something they don't know 
11. Blogs about their work 
12. Not hesitant to pick up a marker and approach a whiteboard 
13. Commits changes to the repository that consist only of comments (but not commented code)
14. Is oblivious to how many times their cubicle-mate has gone for coffee, the bathroom, or the hospital (I don't even hear any sound if I am concentrating on the code).
15.  Not bothered by office politics 
16.  Can predict a bug before the code is ever run  (Done that a few times)
17.  Assumes their own code is the source of a bug before blaming the compiler, library or operating system
18. Disinterested by the outcome of elections 
19. Stock options and bonuses are ineffective 'retain'-ment techniques 

I have done all the signs/symptoms in section "Thinks In Code"

1. In casual conversation their readiest metaphors come from programming constructs (some time back I gave an example of classes/instantiation while explaining 'business offerings')
2. Spends the majority of their time "goofing off", but commits more bug-free code each day than their colleagues
3. Glances over your shoulder and points at a bug in your code with their finger
4. Correctly diagnoses bugs over the phone while drunk or in bed
5. Comes up with their best code while taking a shower*
6. When confronted with an obstinate bug, their instinct is to get up and go for a walk
7. They suddenly pause and stare into space in the middle of a conversation, then abandon you to hurry back to their terminal with no explanation (AKA "A Columbo Moment" or "Gregory House behavior")

Also few signs/symptoms in "Indifferent to Hierarchy"

1. Getting into arguments with the CEO (done that, probably multiple times, still in Geometric because I like working with Geometric CEO, Manu Parpia)
2. Quitting on principle 
3. Organizing teams without permission (I believe its easier to say 'sorry' than get permission)

Overall not a bad score. 

My code review checklist

I am not a fan of checklists (especially for code reviews). Code review checklists start small and then slowly become really large and unwieldy. After sometime checklist becomes a bottleneck and instead of improving effectiveness of your process, these lengthy checklists start reducing the effectiveness. 

However, there are situations where I used checklists and they worked very well. For example, a Customer Release checklist. There are many small small things that you need to do before sending the new release to customer. Its easy to miss few critical steps. A release checklist has always worked very well.

I was not sure why in typical organization sometimes checklists did not work well (for example, in cases like code review) while sometimes it really worked. What exactly is the difference ?

Sometime back I read Atul Gawande's book 'Checklist Manifesto'. It triggered my interest in Checklists again. As first step I extracted a Code Review checklist from my code review training content. I have used this 'mental' checklist for a many-many years. It has worked well for me even with different programming languages (C/C++, Java, Python, C#, Javascript) and technologies. 

Here is my code review checklist.

Code Review Checklist from Nitin Bhide

PS :  Based on my experiences, information from Atul Gawande's book and from information internet, I have now prepared a 4 hour hands-on session on creating and improving the checklists. Contact me if you are interested.

Optimizing Django database access : Part 2

Originally published on BootstrapToday Blog 

Few months back we moved to Django 1.2.  Immediately we realized that our method 'optimizing' database queries on foreign key fields doesn't work any more.  The main reason for this 'break' was Django 1.2 changes to support multiple databases. Specially the way 'get' function is called to access value of a ForeignKey field.

In Django 1.2 ForeignKey fields are accessed using rel_mgr. See the call below.

rel_obj = rel_mgr.using(db).get(**params).

However, manager.using() call returns a queryset. Hence if the manager has implemented a custom 'get'  function, this 'custom get' function is not called. Since our database access optimzation is based on 'custom get' function. This change broke our optimization.

Check Django Ticket 16173 for the details.

Also our original code didn't consider multidb scenarios. Hence query 'signature' computation has to consider the db name also.

from django.db import connection
from django.core import signals

def install_cache(*args, **kwargs):
    setattr(connection, 'model_cache', dict())

def clear_cache(*args, **kwargs):
    delattr(connection, 'model_cache')

signals.request_started.connect(install_cache)
signals.request_finished.connect(clear_cache)

class YourModelManager(models.Manager):
    def get(self, *args, **kwargs):
        '''
        Added single row level caching for per connection. Since each request
        opens a new connection, this essentially translates to per request
        caching
        '''
        model_key = (self.db, self.model, repr(kwargs))

        model = connection.model_cache.get(model_key)
        if model is not None:
            return model

        # One of the cache queries missed, so we have to get the object
        # from the database:
        model = super(YourModelManager, self).get(*args, **kwargs)
        if model is not None and model.pk is not None:
            connection.model_cache[model_key]=model
        return model

There are minor changes from the first version. Now this manager class stores the 'db' name also in 'key'.

Along with this change you have to make one change the core Django code. In file django/db/fields/related.py in in __get__ function of 'ReverseSingleRelatedObjectDescriptor' class, Replace the line

         rel_obj = rel_mgr.using(db).get(**params)

by line

         rel_obj = rel_mgr.db_manager(db).get(**params)

This will ensure that YourModelManager's get function is called while querying the foreignkeys.

Optimizing Django database access : some ideas/experiments

Originally published on BootstrapToday Blog  

As we added more features to BootStrapToday, we started facing issues of performance. Page access was getting progressively slower. Recently we analyzed page performance using excellent Django Debug Toolbar and discovered that in worst there were more than 500 database calls in a page. Obviously that was making page display slow. After looking at various calls, analyzing the code and making changes, we were able to bring it down to about 80 calls and dramatically improving the performance.  Django has excellent general purpose caching framework. However, it’s better to minimize the calls and then add caching for remaining queries. In this article, I am going to show a simple but effective idea for improving the database access performance.

In Django, a common reason for increased database calls in ForeignKey fields. When you try to access the variable representing foreign key typically it results in a database access. Usually suggested solution to this problem is use of ‘select_related’ function of Django query set API. It can substantially reduce the number of database calls. Sometimes it is not possible to use ‘select_related’ (e.g. you don’t want to change how a third party app works or changing the queries requires significant change in the logic etc).

In our case, we have Foreign Key fields like Priority, Status etc on Ticket models. These fields are there because later we want to add ability to ‘customize’ these fields. However, the values in these tables rarely change. Hence usually this results in multiple queries to get the same data.  Usually these are ‘get’ calls. If we can add a simple caching to ‘get’ queries for status, priority etc, then we can significantly reduce the number of database calls.

In Django, a new connection is opened to handle a new request. Hence if we add model instance cache to ‘connection’ then query results will be cached during handling of one request. New request will make the database query again. With this strategy we don’t need complicated logic to clear ‘stale’ items from the cache.

from django.db import connection
from django.core import signals

def install_cache(*args, **kwargs):
    setattr(connection, 'model_cache', dict())

def clear_cache(*args, **kwargs):
    delattr(connection, 'model_cache')

signals.request_started.connect(install_cache)
signals.request_finished.connect(clear_cache)

class YourModelManager(models.Manager):
    def get(self, *args, **kwargs):
        '''
        Added single row level caching for per connection. Since each request
        opens a new connection, this essentially translates to per request
        caching
        '''
        model_key = (self.model, repr(kwargs))

        model = connection.model_cache.get(model_key)
        if model is not None:
            return model

        # One of the cache queries missed, so we have to get the object from the database:
        model = super(YourModelManager, self).get(*args, **kwargs)
        if model is not None and model.pk is not None:
            connection.model_cache[model_key]=model
        return model

As a side benefit, since the same model instance is returned for same query in subsequent calls, number of duplicate instances is reduced and hence memory foot print is also reduced.

Notion of Repeatable processes

From the recent discussion on PragmaticProgrammer Mailing List:

(From Mail from Andrew Hunt)

Here's a side question for ya'll on the notion of repeatable process: I want a formula of how to make a hit movie. Perfectly repeatable, with no consideration of the talent involved. Just turn the crank and rely on the process to work.

Now replace the 'hit movie' with Software Product/Project. I never seen the concept explained with such a simple and thought provocating metaphor.