Transforming a Struggling Software Team into a High-Performance Team - Part 3

In this  third installment of our series, "Transforming a Struggling Software Team into a High-Performance Team," we delve into the pivotal third step of our transformation: improving our version control practices.

Our team's journey with Git, took a dramatic turn as we shifted from the somewhat chaotic 'developer branches' to the harmony of a unified 'trunk' based development approach. This strategic move was not just a change in process, but a cultural shift towards collective code ownership and streamlined workflows for a typical small to medium-sized team.

The creation of two  branches, 'develop' and 'bugfix', from the 'main' branch, marked the beginning of a new chapter. Developers merged their individual efforts (i.e. the individual 'developer branches' into the single 'develop' branch, resolving any conflicts along the way, and then bid farewell to their isolated 'developer branches'. This unification meant that all developers were now committing directly to the 'develop' branch, fostering an environment of collaboration and immediate integration.

To ensure a seamless workflow, we introduced critical practices such as syncing with the latest code at the start of the day and post-lunch, as well as pushing changes before breaks. This routine not only kept each developer codebase fresh but also minimized integration headaches.

Our build strategy also saw a transformation with the introduction of a daily build from the 'develop' branch, a QA build from the 'bugfix' branch for our diligent testers, and a customer release build triggered upon merges to the 'main' branch. This structured approach led to a more stable codebase, with integration issues being swiftly identified and rectified during local feature testing.

Join us as I continue to share this journey in the quest to achieve software excellence. Stay tuned for more insights and strategies that can help transform your team from struggling to high-performing.

Elevating our software team's game, we've changed our version control protocols, catapulting code stability to new heights. Developers now unearth and rectify integration snags swiftly through meticulous local feature testing. We've instituted  release naming conventions, build number tracking, and a streamlined hot fix process. This strategic overhaul has slashed the influx of trivial bugs to QA, empowering our engineers to zero in on pivotal feature and behavior anomalies. The result? A seismic reduction in customer-facing bugs, showcasing our commitment to excellence. (P.S. These insights are versatile, meshing seamlessly with any version control system, be it Subversion, Mercurial, or Perforce.)

NOTE :  This article was originally posted on LinkedIn and expanded.

• Part 1 of this series
• Part 2 of this series

Transforming a Struggling Software Team into a High-Performance Team - Part 2

 🚀 Transforming a Struggling Software Team into a High-Performance Team - Part 1 

One Proverb all project managers and tech leads should member is _"Road to Hell is Paved with Good Intentions"._ But in software development, Good Intentions can also lead to a maze of bugs." This is a mantra that every project manager and tech lead should remember. Implementing a policy with good intentions does not guarantee positive outcomes for your project.

Take the common practice of using try/catch blocks in Java and .NET projects. Developers often add these blocks intending to prevent crashes. However, if the catch block merely logs the error and the exception is not properly handled, it can create chaos for the entire team.

Let's dissect a hypothetical scenario to highlight the issue:

x = func1();
y = func2(x);
z = func3(y);

In an ideal world, if  `func1()` throws an exception, neither `func2()` nor `func3()` should be called. But what if a developer wraps `func1()` in a try block followed by an empty catch block? The exception is caught, and the program erroneously proceeds to `func2(x)`, operating on a potentially incorrect or corrupted state of 'x', which could lead to another exception. To "fix" this, another empty catch block is added, and the cycle continues until `func3(y)` crashes.

The real bug, however, likely resides in `func1()`. This means the developer will waste a significant amount of time tracing back to the actual cause of the problem.

As a Tech Lead, by advocating a 'no-crash' policy without proper exception handling, you've inadvertently constructed a 'road to bugfix hell'. It's a cautionary tale that underscores the importance of thoughtful and effective error management in software development. 

Lets get back to our Case study of  "Transforming a Struggling Software Team into a High-Performance Team"

Imagine a codebase, a labyrinth of about 500 try/catch blocks, most leading to dead ends of empty catch blocks. This was the daunting maze the "Struggling Software Team." was working with. Determined to bring order to chaos, I (along with Project Manager and Team lead) championed a cleanup crusade. The result? A leaner, meaner set of 50 catch blocks, each a sentinel effectively guarding against errors, no longer just placeholders but active defenders of code integrity.

The moment of truth came when `func1()` threw an exception. Instead of the usual obfuscation, what emerged was a crystal-clear call stack, pinpointing the 'correct' error. This clarity was win for the developers, project manager and technical lead, This singular fix slashed bug fix times from a sluggish weeks to a swift days.

This narrative isn't unique. Many software teams, with the best intentions, erect barriers to their own success. Intelligent teams risk devolving into dysfunction, not for lack of skill, but from practices that stifle rather than stimulate productivity and quality.

The linchpin in this equation is the project manager and the management team. I've witnessed customers miss out on significant productivity and quality gains simply because these practices weren't taken seriously, weren't enforced. The cost? Not just in subpar productivity and quality, but in the ballooning development costs that inevitably follow.

Join the conversation and share your experiences in transforming software development practices. Let's inspire and learn from each other's journeys.

Sane Branch Management of Version Control Systems for Teams

[NOTE : This is still a 'draft', please point out mistakes and suggest changes/improvements]
Some time back Nagaraj Mali asked a question about 'best practices for repository branch management' on one of our whatsapp groups. This is commom query. 'bad branching strategy' or 'no strategy about branching' are very common mistakes in project teams. These mistakes can seriously impact the teams productivity and quality. Unfortunately very few manager/scrum masters or tech leads really understand devastating impact of 'bad version control practices'. In this blog, I am attempting to explain my views and logic behind various practices that I recommend.

Mistake : Too many branches

A developer need to understand the difference between a 'branch' and a 'tag'. Many times teams use 'branch' where tag is sufficient. 'branch' is an 'active line' of development. Common practice is product teams is to support 3 previous versions (i.e major releases) for bug fixes and one new release'. For example your current release is Ver10 and you may supporting major Versions9, 8 and 7. Essentially you have 4 'lines of development'. Then you need 4 branches. Ideally you should checkout 'only' the  branch that you are working on. There is no need to checkout all branches. Since 'branches' represent 'live lines of development', once you stop supporting a particular major release, you should 'delete' the branch for that release. Typically you will have 4 to 6 branches and many/many tags.

Mistake : Creating new branch for every minor release

Assume a major release is 'version 9'. Then 'version 9.1' is developed on 'version 9'  branch ONLY. Remember 'Version 9' is the supported major release and 'live development line'. Version 9.1 is usually a 'NOT live development line'. Any code changes/bug fixes for Version 9.1 should be developed in 'Version 9' branch'

Common complaint : We spend long time in merging.

This is another symptom of bad branching and bad practices followed by team. Consider following branch scenarios.

1. Release branches for past releases
   
   Lets assume that new releases are developed in 'master' and team is supporting 2 previous release 'Version1' and 'version2'. Now team have done a bug fix in 'Version1'. Obviously customers will expect that same bug fix is available on 'patch release on Version2' and also available when new Version3 becomes available. So every day branches must be merged 'upstream'. i.e. merge Version 1 commits to Verson 2. And merge version 2 commits to 'master'. If you get any merge failure fix them. Daily changes are typically small and can be easily merged. This simple practice ensure that time required to do merges are drastically reduced and bug introduced because of merge problems are almost entirely eliminated.
   
   
2. Feature branches
   
   Many teams create 'feature' branches for every new feature or bug fix. However, they don't delete the feature branch one the feature development is over. At end of 'feature' development feature branch must be merged into 'master' and then delete the feature branch. It is best to use 'git flow' plugin/workflow for working on feature branches.
   
   Typical feature branch flow will be 
   • create feature branch from 'master' (so 'parent branch' will be 'master')
   • keep making changes and commiting in feature branch. Its ok to push feature branch.
   • every day merge 'parent branch' to 'feature' branch. This way any changes to 'parent' branch done by other developers are available to you and any conflicting changes are detected early.
   • once the feature is done, merge the 'feature branch' to 'parent branch' (usually 'master'). Since you are regularly merging the 'master' to 'feature', you will not see any conflicts when you merge 'feature' to master. Now delete the feature branch
   
   In both scenarios 'key' is the regular (preferably every day) merge from 'release branch' to master' or 'master to feature' branch.

Remember It is OK to delete a branch

Remember in version control there is nothing like 'permanent delete'. When a branch is deleted, you will not see it 'branches list'. But that does not mean all the history of such branch is also deleted. Recommended practices is to create a 'tag' from branch and then delete the branch. This will keep the history intact. It will also allow to restore a branch from the tag, if you need to do some critical bug fix in an older (now unsupported) release

Bottomline

Remember 'version control' is 'productivity tool' for the development team and NOT just a backup tool. Learn how to take advantage of the version control system and you will see signficant increase in productivity of your team.  Start by defining policy for 'branch creation/branch deletion and daily merge'.

Manual Code Review of every single change is OVERRATED

Many software companies have this policy that every change has to be manually reviewed and approved by some peer (or senior) developer either before it is commited or before it is merged in main development branch. This practice is recommended by many software luminaries. However, when implemented in a project/product, I found many practical issues with 100% manual code reviews and reached a conclusion that "manual code review of every single change is OVERRATED"

Over years, I realized that manual code review of every single change is OVERRATED. It does not appreciably reduce post shipment defects but does introduce many inefficiencies in your development process. Manual Code Review of every single change usually have -ve ROI. I find that nobody writes about bad side effects of forcing developers into 100% manual code reviews. Recently I found one article and that triggered me to write this blog post.

So here is the list of problems that I found in 100% Manual Code Review process

Problems in Manual Code Review of every single change

1.  ROI on Code Review - 
   
   Is reviewer finding kind bugs in code review such that if they remain undetected will take more efforts than the time spent in Code Review ? If yes, then ROI of Code Review is +ve else ROI of Code Review is -ve.  Most of the Code Reviews have -ve ROI.  
   Reviewer may be a good developer but he may not be really good or may not have an 'eye' for defect. Some years back I did an informal study of 'code review' comments reported in my company's code review system. Majority of the code comments were about 'coding style' or 'comments' or 'naming conventions'. Very rarely a serious bug is detected. Effectively Such code reviews have a '-ve ROI'.
2. Reviewers time and attention.
   
   Typically Reviewer is mostly one of the senior developers. He/She also has their own commitments on new feature development. Usually these commitments take priority over code reviews. Hence Code Reviews remain stuck in queue and total feature development time increases unnecessarily (elapsed time from start to feature is delivered in end user's hand).
3. Team's Most productive developers are 'reviewing' rather than 'writing new code'.
   On the other hand, if Senior developers spend majority of their time doing reviews, then 'most productive developers' are effectively not writing any code. Either case results in lower productivity for the entire team.
4. Lower developer productivity while fixing bugs reported during code reviews.
   Code review is helps only if review happens within short time writing the code (preferably within a day). However, in general actual review happens after many days. In this time, developer loses the context of code. When finally the code is reviewed and he/she has to fix something in the code, it takes longer because now developer has to 'get in the context' again. Effectively it lowers the developer's productivity again.

Over year's almost every team that I have worked with has made this mistake and effectively reduced the productivity. How do we fix this ? Whats the alternative ?

How to regain the productivity lost by "manually review every code change policy" ?

1. Institute "Zero Warnings" policy for your code.
   This is easiest and most effective way to eliminate bugs. Every new version of compiler improves in ability to detect potential problems and report them as 'warnings'.  Set your compiler to 'highest warning' level. If you are using Visual Studio, turn on 'report warnings as errors" setting. Fixing warnings is 'practically no-brainer'. Warnings have to fixed immediately. DO NOT wait till end of sprint or end of release to fix warnings.
2. Take advantage of Static Analysis Tools for your programming language.
   Every programming language now has static analysis tools (linters, tools like Klocwork, Coverity etc). This Wikipedia article will give you a list of static analysis tools. Make static analysis part of your 'CI/CD' pipeline. Make sure that static analysis bugs are fixed within day. DO NOT wait till end of sprint or end of release for fixing static analysis errors.
3. Identify your critical files and review only those manually.Stop reviewing very single line of change. Identify your critical files (use complexity analysis, analyze version history to find most frequently changed files etc).  Usually 80-20 rule applies 20% are your critical files. Mark those files in system. Any commits in these files should trigger your manual code review process. Manually review these critical files ONLY after warnings and static analysis bugs are fixed.

Development processes/practices followed in a typically software development team usually have serious inefficiencies. By using 'common sense' (rather than following common practices) it is possible to achieve an 'order of magnitude' (2x to 10x) improvement in the team's productivity and delivery quality. It DOES NOT require a change in methodology (like move to Agile) or any new expensive tools. It requires some disciple and taking advantage of many low hanging fruits. The results are usually visible in about 3 months. 

[Shameless Plug]If you want to know how to improve your team's productivity by 2x to 10x, I am available for 'consulting/coaching'.

References

1. Code Reviews Do Not Find Bugs - How the Current Code Review Best Practice Slows Us Down
2.  

Understanding DevOps Core Concepts - presentation on Slideshare

Few days back I published my understanding of DevOps on slideshare.

DevOps - Basics from Nitin Bhide

My code review checklist

I am not a fan of checklists (especially for code reviews). Code review checklists start small and then slowly become really large and unwieldy. After sometime checklist becomes a bottleneck and instead of improving effectiveness of your process, these lengthy checklists start reducing the effectiveness. 

However, there are situations where I used checklists and they worked very well. For example, a Customer Release checklist. There are many small small things that you need to do before sending the new release to customer. Its easy to miss few critical steps. A release checklist has always worked very well.

I was not sure why in typical organization sometimes checklists did not work well (for example, in cases like code review) while sometimes it really worked. What exactly is the difference ?

Sometime back I read Atul Gawande's book 'Checklist Manifesto'. It triggered my interest in Checklists again. As first step I extracted a Code Review checklist from my code review training content. I have used this 'mental' checklist for a many-many years. It has worked well for me even with different programming languages (C/C++, Java, Python, C#, Javascript) and technologies. 

Here is my code review checklist.

Code Review Checklist from Nitin Bhide

PS :  Based on my experiences, information from Atul Gawande's book and from information internet, I have now prepared a 4 hour hands-on session on creating and improving the checklists. Contact me if you are interested.