Elevating DevOps: A Journey from novice to Master craftsman

 In the dynamic world of DevOps, the gap between theory and practice can sometimes seem as vast as space itself. Reflecting on my journey through countless interviews with DevOps engineers, a pattern emerges: a disconnect between claimed experience and practical knowledge. It's a concerning trend that sees the essence of DevOps, a craft of precision and innovation, being diluted by superficial engagement.

Rewind to the days before DevOps became the industry buzzword. I was at the helm of creating a comprehensive DevOps platform for Geometric Ltd., integrating both commercial and open-source tools. This experience was more than just 'clicking buttons'; it was about architecting a seamless workflow that propelled projects forward.

For those aspiring to master DevOps, here's a distilled essence of my experience:

1. Begin with the roots. Watch John Allspaw and Paul Hammond's seminal presentation at the Velocity 2009 conference. Let it be your DevOps genesis, and absorb its insights multiple times.
2. Dive deep into Version Control Systems (VCS). Whether it's Git, Mercurial, or Subversion, understanding the intricacies of VCS is non-negotiable. They are the backbone of any robust DevOps strategy.
3. Ensure everything is traceable. From source code to CI/CD configurations, every element should be version-controlled, allowing you to pinpoint changes with precision.
4. Immerse yourself in the wisdom of pioneers. My recommended reads include 'Release It' by Michael Nygard, 'The Phoenix Project' by Gene Kim, and 'Accelerate' by Gene Kim and Jez Humble.
5. Learn from the best. Study the engineering blogs of tech giants like Netflix, Google, and Facebook, Uber to stay abreast of cutting-edge practices.

DevOps is not just a role; it's a mindset of continuous improvement and relentless pursuit of excellence. Let's commit to upholding the true spirit of DevOps and nurturing the next generation of software craftsmen. 

Posted on LinkedIn  https://www.linkedin.com/posts/activity-7215942686271238144-7XjI

Ideas on on sessions /courses as Prof of Practce for Chandigarh University

 Sometime back I posted an update on LinkedIn about joining Chandigarh University as Professor of Practice.   

I am joining CHANDIGARH UNIVERSITY Computer Science dept as "Professor of Practice". This is part time. I plan to spend about 2 weeks every semester on university campus. And also want to do some online lectures/workshops. There are no particular time commitments. It more about Industry + Academia partnerships and sharing the experiences of with Students.

I also asked about ideas on sessions/courses that students will be interested in. I received lots of comments and suggestions. Here I am collecting all the ideas/suggestions at one place. Over time, I will keep on posting updates whenever I implement an idea/suggestion.

Nitin's Ideas

1. Few sessions in current academic topics related to software architecture and design but with sharing of experiences.
2. A 2 week hands-on workshop for 2nd/3rd year students on "professional software development" starting from using version control, and how to write a spec doc to coding, testing, etc.
3. A 1 week workshop for student teams who want to participate in hackathons.
4. Few sessions on diverse topics like 
• Algorithms in Daily life, 
• Costliest bugs in history of software, 
• Women Pioneers in Computer Science, 
• Basics of Cascading Style Sheets (CSS), 
• Hidden Life of a Google Search Request 
• (re)Inventing the Assembly Language.
• Do/Doing/Done is NOT Kanban
• Using Version Control Effectively

Ideas from Harish Walke

1. few topics around "Agile" too! Fundamentals and basics of Agile
• Typical day or sprint in the life of a Developer
• built-in Quality and what it means in Agile
• Scaling agile 
• Some useful concepts of Value streams, Lean Portfolio, Strategic Themes, WSJF for prioritization, Real life examples of Epic, Capabilities, Features
2. Topics related DevOps
• DevOps with CI/CD.
• TDD, BDD. Different compliances and their inclusion in Features as NFR (Non Functional Requirements)
• SecDevOps with mindset of "Built-in Security"
3.  Innovation mindset related
• Lean Startup, 
• Design thinking, 
• Think Wrong (Don't find solutions for the problems that do not exist).h
4. Use of Tools while doing the course projects
• Task or backlog management tools
• version control tools

Ideas from Anirudha Raste

1. OO eyes!
   Look around you, in your real life to find many OO concepts. People think all OO and other software concepts were 'Created' by software engineers and architects. Whereas, they are already present in real life. Need to see them in software way! Then one will find that software is not so 'alien' to real world.
2. 'Importance of logging while troubleshooting problems at customer end, where you do not have access to customer's environment, customer data, debugging

Ideas from K K George (KKG)

The importance of unit testing and show them how to measure coverage of the unit tests.

Ideas from Manish Ramrakhiani   

• Agile principles
• Rise of Data and related roles in the Data Engineering/Analytics space
• Sharing some of real successful archicture/design of your favorite projects (Nitin's comment ; I was also thinking on these lines. So probably will do this early)

Ideas from Nikhil Karkare

• Automation: automating testing, RPA
• Would sound basic, but using powershell or bash for basic scripting (using awk for example) - (Nitin's comment :  In my opinion, absolutely needed)
• Using any code editor to its full potential (Visual Studio / Replit)

Ideas from Laxmidhar Gaopande

 Some projects split in small teams to work on for 2-3 months after your sessions and review by making them presentation to you along with all.  (Nitin's comment : CU already have a concept of Project Based Course, where students learn by doing a project rather than attending lectures. Will be good ideas to incorporate these in the same framework)

Ideas from Vivek Gupta

1. code readability/ maintainability, 
2. logging/eventing/traceability, log analysis (Nitin's comment : Also suggested by Anirudha Raste)
3. ease of troubleshooting, 
4. importance of code reviews, 
5. threat modeling etc. 

Ideas from Surjit Patra

Go with very basic and core topic like Aniruddha Raste said OO things. But also go 1 level below 

• "How to realize or relate the software way of thinking to a real problem". 
• Or How to visualize the problem, is it from outside/ inside. 
• What is the best approach to find the possible solutions.
•  How to pick the best one out of it. 

If you have any other ideas, please add in comments

Manual Code Review of every single change is OVERRATED

Many software companies have this policy that every change has to be manually reviewed and approved by some peer (or senior) developer either before it is commited or before it is merged in main development branch. This practice is recommended by many software luminaries. However, when implemented in a project/product, I found many practical issues with 100% manual code reviews and reached a conclusion that "manual code review of every single change is OVERRATED"

Over years, I realized that manual code review of every single change is OVERRATED. It does not appreciably reduce post shipment defects but does introduce many inefficiencies in your development process. Manual Code Review of every single change usually have -ve ROI. I find that nobody writes about bad side effects of forcing developers into 100% manual code reviews. Recently I found one article and that triggered me to write this blog post.

So here is the list of problems that I found in 100% Manual Code Review process

Problems in Manual Code Review of every single change

1.  ROI on Code Review - 
   
   Is reviewer finding kind bugs in code review such that if they remain undetected will take more efforts than the time spent in Code Review ? If yes, then ROI of Code Review is +ve else ROI of Code Review is -ve.  Most of the Code Reviews have -ve ROI.  
   Reviewer may be a good developer but he may not be really good or may not have an 'eye' for defect. Some years back I did an informal study of 'code review' comments reported in my company's code review system. Majority of the code comments were about 'coding style' or 'comments' or 'naming conventions'. Very rarely a serious bug is detected. Effectively Such code reviews have a '-ve ROI'.
2. Reviewers time and attention.
   
   Typically Reviewer is mostly one of the senior developers. He/She also has their own commitments on new feature development. Usually these commitments take priority over code reviews. Hence Code Reviews remain stuck in queue and total feature development time increases unnecessarily (elapsed time from start to feature is delivered in end user's hand).
3. Team's Most productive developers are 'reviewing' rather than 'writing new code'.
   On the other hand, if Senior developers spend majority of their time doing reviews, then 'most productive developers' are effectively not writing any code. Either case results in lower productivity for the entire team.
4. Lower developer productivity while fixing bugs reported during code reviews.
   Code review is helps only if review happens within short time writing the code (preferably within a day). However, in general actual review happens after many days. In this time, developer loses the context of code. When finally the code is reviewed and he/she has to fix something in the code, it takes longer because now developer has to 'get in the context' again. Effectively it lowers the developer's productivity again.

Over year's almost every team that I have worked with has made this mistake and effectively reduced the productivity. How do we fix this ? Whats the alternative ?

How to regain the productivity lost by "manually review every code change policy" ?

1. Institute "Zero Warnings" policy for your code.
   This is easiest and most effective way to eliminate bugs. Every new version of compiler improves in ability to detect potential problems and report them as 'warnings'.  Set your compiler to 'highest warning' level. If you are using Visual Studio, turn on 'report warnings as errors" setting. Fixing warnings is 'practically no-brainer'. Warnings have to fixed immediately. DO NOT wait till end of sprint or end of release to fix warnings.
2. Take advantage of Static Analysis Tools for your programming language.
   Every programming language now has static analysis tools (linters, tools like Klocwork, Coverity etc). This Wikipedia article will give you a list of static analysis tools. Make static analysis part of your 'CI/CD' pipeline. Make sure that static analysis bugs are fixed within day. DO NOT wait till end of sprint or end of release for fixing static analysis errors.
3. Identify your critical files and review only those manually.Stop reviewing very single line of change. Identify your critical files (use complexity analysis, analyze version history to find most frequently changed files etc).  Usually 80-20 rule applies 20% are your critical files. Mark those files in system. Any commits in these files should trigger your manual code review process. Manually review these critical files ONLY after warnings and static analysis bugs are fixed.

Development processes/practices followed in a typically software development team usually have serious inefficiencies. By using 'common sense' (rather than following common practices) it is possible to achieve an 'order of magnitude' (2x to 10x) improvement in the team's productivity and delivery quality. It DOES NOT require a change in methodology (like move to Agile) or any new expensive tools. It requires some disciple and taking advantage of many low hanging fruits. The results are usually visible in about 3 months. 

[Shameless Plug]If you want to know how to improve your team's productivity by 2x to 10x, I am available for 'consulting/coaching'.

References

1. Code Reviews Do Not Find Bugs - How the Current Code Review Best Practice Slows Us Down
2.  

Understanding DevOps Core Concepts - presentation on Slideshare

Few days back I published my understanding of DevOps on slideshare.

DevOps - Basics from Nitin Bhide

Agile Burn Down Chart

I am guiding a project team on some architecture changes. Its a new product. However, I was not getting any clear picture on the progress. I checked Burn down chart. Even that did not give me clear picture. I checked with project team as well Process team on the burn down chart. AND they successfully confused me. :-)

Process team told me that they track the burn down chart based on the 'remaining effort'. Every day each team member update how much time he/she worked any story. The 'remaining time' is the used to plot the 'burn down chart'. 

Now consider the scenario below

1. Original estimate of Story A is 40 hours.
2. Team member X, worked on Story A on first day for 8 hours. So he logged 8 hours and 'worked' and 32 hours as remaining.
3. 2nd day he worked for 8 hours and updated 24 hours as remaining
4. 3rd day he worked for 8 hours and updated 16 hours as remaining
5. 4th day he worked for 8 hours and realized that many more things are still do be done to actually complete the story. So he updated 36 hours as remaining.

The chart will show that till day 3 you are on track. However, on day 4 you are suddenly 'delayed'. At this point, the graph will shoot upwards (Q: Do I still call it burn 'down' chart ??). In my opinion, this type of burn down chart gives a misleading picture to project team. On top of this,  consider that sprint duration is 1 month and stories are estimated at 40 to 60 hours duration. Now you have completely inaccurate picture of project progress. Many cases, by the time PM or Scrum master realizes something is wrong, team is very near to end of Sprint and has very little time to take any corrective action. Exactly the scenario that you are trying to avoid by implementing Scrum. !!!!

So what a Scrum team should do to get better picture in "burn down chart" ??

This is what I will do

1. I will not use remaining time (or remaining story points) to update the burn down chart.
2. If the story is not done, then entire estimated time is 'shown' as 'remaining' in burn down chart.
3. When the Sprint starts, I will ensure that stories are broken down to level where they can be completed in 2-3 days (irrespective of I am using story points or hours for 'story sizing').
4. I will ask which stories may get delayed during the Daily Huddle, but will not use 'remaining hours' estimate by developer to update the burn down. I will use this info only to decide who may need help and how to give that help.

Now my burn down chart will look like a ladder. 

For the above scenario,  till first 3 days, story will remain open and at 40 hours remaining time. On 4th day, when story is complete, remaining time will go to 0. The burn down chart will now look like this.

 

What do you think ? How do you calculate the burn down chart in your project ?

Why does Agile work ?

Why does Agile work ?  Yesterday I missed a session on 'why Agile' works in Agile Dev (West) conference.  (https://adcwest.techwell.com/program/concurrent-sessions/why-agile-works-and-how-screw-it-0).

Just wanted to share my thoughts on "why Agile works". 

Waterfall Assumes and expects "perfection"

Waterfall model assumes 'perfection'. It expects perfect requirements, then perfect design, then perfect code. Usually that doesn't happen but going back and correcting becomes difficult because of this expectation of 'perfection'.

Agile assumes and expects "mistakes will happen"

Agile assumes that 'mistakes will happen'. So all Agile practices focus on 'detecting mistakes early and fixing them as required'. It assumes that requirements will change. Hence Agile practitioners advise that  "don't spend time on big upfront requirement and design". If requirements change then this big upfront requirement spec and design is a 'waste'. To minimize such waste all Agile practices are use some kind 'fail fast' mechanism. This implementation of 'fail fast' mechanisms reduces waste and hence cost at the same time it improves the quality of outcome. Because of this 'mistakes will happen' mindset, Agile works better than waterfall in practice. Remember Toyota production systems is also based on implementing 'Fail Fast' mechanisms

So what is the 'guaranteed' way of ensuring 'Agile failure' ?  

Bring the 'perfectionist' mindset of Waterfall to Agile. Managers and teams who try to bring this 'perfectionist mindset' to Agile ensure that Agile/Scrum do not work for them.