Sane Branch Management of Version Control Systems for Teams

[NOTE : This is still a 'draft', please point out mistakes and suggest changes/improvements]
Some time back Nagaraj Mali asked a question about 'best practices for repository branch management' on one of our whatsapp groups. This is commom query. 'bad branching strategy' or 'no strategy about branching' are very common mistakes in project teams. These mistakes can seriously impact the teams productivity and quality. Unfortunately very few manager/scrum masters or tech leads really understand devastating impact of 'bad version control practices'. In this blog, I am attempting to explain my views and logic behind various practices that I recommend.

Mistake : Too many branches

A developer need to understand the difference between a 'branch' and a 'tag'. Many times teams use 'branch' where tag is sufficient. 'branch' is an 'active line' of development. Common practice is product teams is to support 3 previous versions (i.e major releases) for bug fixes and one new release'. For example your current release is Ver10 and you may supporting major Versions9, 8 and 7. Essentially you have 4 'lines of development'. Then you need 4 branches. Ideally you should checkout 'only' the  branch that you are working on. There is no need to checkout all branches. Since 'branches' represent 'live lines of development', once you stop supporting a particular major release, you should 'delete' the branch for that release. Typically you will have 4 to 6 branches and many/many tags.

Mistake : Creating new branch for every minor release

Assume a major release is 'version 9'. Then 'version 9.1' is developed on 'version 9'  branch ONLY. Remember 'Version 9' is the supported major release and 'live development line'. Version 9.1 is usually a 'NOT live development line'. Any code changes/bug fixes for Version 9.1 should be developed in 'Version 9' branch'

Common complaint : We spend long time in merging.

This is another symptom of bad branching and bad practices followed by team. Consider following branch scenarios.

1. Release branches for past releases
   
   Lets assume that new releases are developed in 'master' and team is supporting 2 previous release 'Version1' and 'version2'. Now team have done a bug fix in 'Version1'. Obviously customers will expect that same bug fix is available on 'patch release on Version2' and also available when new Version3 becomes available. So every day branches must be merged 'upstream'. i.e. merge Version 1 commits to Verson 2. And merge version 2 commits to 'master'. If you get any merge failure fix them. Daily changes are typically small and can be easily merged. This simple practice ensure that time required to do merges are drastically reduced and bug introduced because of merge problems are almost entirely eliminated.
   
   
2. Feature branches
   
   Many teams create 'feature' branches for every new feature or bug fix. However, they don't delete the feature branch one the feature development is over. At end of 'feature' development feature branch must be merged into 'master' and then delete the feature branch. It is best to use 'git flow' plugin/workflow for working on feature branches.
   
   Typical feature branch flow will be 
   • create feature branch from 'master' (so 'parent branch' will be 'master')
   • keep making changes and commiting in feature branch. Its ok to push feature branch.
   • every day merge 'parent branch' to 'feature' branch. This way any changes to 'parent' branch done by other developers are available to you and any conflicting changes are detected early.
   • once the feature is done, merge the 'feature branch' to 'parent branch' (usually 'master'). Since you are regularly merging the 'master' to 'feature', you will not see any conflicts when you merge 'feature' to master. Now delete the feature branch
   
   In both scenarios 'key' is the regular (preferably every day) merge from 'release branch' to master' or 'master to feature' branch.

Remember It is OK to delete a branch

Remember in version control there is nothing like 'permanent delete'. When a branch is deleted, you will not see it 'branches list'. But that does not mean all the history of such branch is also deleted. Recommended practices is to create a 'tag' from branch and then delete the branch. This will keep the history intact. It will also allow to restore a branch from the tag, if you need to do some critical bug fix in an older (now unsupported) release

Bottomline

Remember 'version control' is 'productivity tool' for the development team and NOT just a backup tool. Learn how to take advantage of the version control system and you will see signficant increase in productivity of your team.  Start by defining policy for 'branch creation/branch deletion and daily merge'.

SVNPlot Version 0.9.0 Released

Today I am releasing SVNPlot Version 0.9.0.


SVNPlot now works on Python 2.7.x and on Python > 3.5.x. Current release also contains many small bug fixes esp. related to Unicode handling.



You can download the installers from Bitbucket Download Page.

Recovering from corrupted Subversion repository

As a habit I store all my personal data (training ppts, various experimental, etc) in version control. Since I am doing this for a many years, I am using Subversion to store all this data. Recently I upgraded to Subversion 1.8.  I tried to do a fresh checkout and realized that repository is corrupted. 'svnadmin verify' returned error and could not fix it. I could not do a fresh 'svn checkout' and I could not use TortoiseHg as client to Subversion. I was stuck. The error I was getting was 'E200002 : serialized hash missing terminator' error.

My first step was to Google for error message and see if I get any solution. The search results suggested running 'svnadmin verify'. It confirmed the error at revision 1266. Many others suggested taking backed-up repository dump and restored from the dump. I took backup of the repository and did not take a repository dump at the time of backup. The backup of the repository was also corrupt at the same revision. So this solution was not very useful for me.

My second step was to send a mail to Subversion users email list 'users@subversion.apache.org'. You can see the mail here. Stefan Sperling suggested 'fsfsverify' script. I tried it but it did not work for me.I also tried 'fsfsfixer' script. That also did not work. 

The best clue I got from 'Philip Martin'. I was not sure what the error 'serialized hash missing terminator' means. I tried to read the documentation for FSFS format. Based on the checked the data from 'revision' files. It seemed to be corrected. 

Philip explained the error as

"It means one of the repository files is corrupt. It could be a revision files in db/revs or it could be a revprop file in db/revprops. A serialized hash is a series of K/V pairs followed by END:"

The key was "it could be a revrop file db/revprops". So far I missed on checking the 'revprop' files. I checked the revprops file for revision 1266 and found that it is made of just Zeros. There was no content.

Now the problem was clear. The revprop files are corrupt, I did not have the backup of 'uncorrupted' revprop files. Hence only was to somehow recover the revprops. Typical revision prop file looks like this

K 10
svn:author
V 2
pm
K 8
svn:date
V 27
2013-09-05T18:00:22.881511Z
K 7
svn:log
V 1
m END

Typically it contains information about 'author', 'date' and svn commit message. It may also contains merge information. In my case, mergeinfo was not there. Since it was a personal repository only person committing code was me. Hence svn:author will always be me. Now its question of creating message and date. So decided to do take a simpler approach for date use the 1 minute after previous valid revisions date. And message as 'this revision is recovered from corruption' kind of message.

I wrote a  python script, which takes repository path and revision number and does the following

1. Runs the 'svnadmin verify -r ' and checks 'serialized hash missing terminator error'. 
2. If the error is reported, the script reads the revision properties of revision just before that (i.e. revno-1) and add One minute to the time stamp of this revision.The log message is changed 'recovered from corruption' message.
3. Now original corrupted revision property file is copied to a backup location and corrected revision property is written in its place.
4. The process repeats till get it 'valid revision'. At this point it stops.

After this I ran the 'svnadmin verify' on entire repository and confirmed that all revisions can be read. Then dumped the repository contents to dump file and reloaded the content into a new empty repository.

I checked the log and diff of the revision 1266 and few subsequent revisions. Things seem to be on track.

PS> The python script I wrote is very specific to my needs. However, if you need it, send me a mail.

svnplot - one year later

About one year back (Dec 2008) I changed job. Between two jobs I had some free time. I wrote first version of svnplot during these 4-5 days. Then I released it as 'open source' project on Google Code. Soon many people started using it. I started getting the bug reports filed on the project page. To me, this was a indication that people are really using this project. 

1. I got bug reports from developers/scientist working in places like CERN, AMD.
2. One of the bug reports mentioned that "I'm using SVNPlot on over 100 of my users' repositories"
3. Svnplot was mentioned in discussions on StatSVN forums
4. StatSVN developers added the feature of  tag cloud of commonly used words in commit messages inspired from the similar feature in svnplot. So as mentioned by  Benoit "it is now a bilateral inspiration". Since initially features in svnplot were inspired from excellent StatSVN project.

Many people contributed bug fixes and improvements to svnplot.

1. Chris Glasman added  support for repository authentication.
2. Oscar Castaneda developed/contributed code to convert SVN logs to output files can be used in CMU's ORA and Apache Agora as as part of Google Summer of Code 2009 (GSoC09). You can read the details of his contribution here.
3. kitpz2 contributed code for better pie-chart display of directory sizes.

I think the key advantage of  svnplot is it doesn't require a checked out copy of repository. Also it is easy to hack.

So what's next ?
 

I am now working on next version of svnplot (0.6). The key new feature will be graphs be generated on client side with javascript and HTML canvas. This will reduce the dependency on matplotlib and it will be easier for users to deploy it.  After checking few Javascript charting libraries like Flot, jquery.Visualize plugin, I decided to use jqPlot. I am planning to release Svnplot 0.6 in few weeks time.

Book recommendations for Software Developers

As I mentioned in my previous post on 'C++ Book Recommendations', I have now published a list of books for software developers. These are the books helped me in developing my ideas about Software Development (irrespective of technlogy or programming languages)

Check the list at "Book Recommendations for Software Developers"

Comparison of VSS, CVS and SVN

I have prepared a comparison among the three commonly used Version Control Systems Visual SourceSafe (VSS), CVS and Subversion.

The weights and scores are based on my judgement. I think this type weight scores based comparison may help you in convincing people (e.g. your project team, colleages, senior mgmt in your company) to use Subversion.

Total Score
Subversion : 251
CVS : 171
Visual SourceSafe(VSS) : 138

Check it out at Comparison of VSS, CVS and SVN

Using Social Network Analysis with Version control data

As I mentioned in the last post, am experimenting about using social network analysis (sna) on verision control data. Now with SVNPlot project, I have a way of converting the Subversion logs into sqlite database. It allows me to query the data in many different ways.

I used the Rietveld repository data and did some premilinary analysis. I am not an expert on SNA but Initial results look very interesting and promising. You can see the results on my website

Update : Oscar Castaneda has added SNA data extraction to SVNPlot as part of GSoC 2010 project. He has used these modifications to analyze Apache repositories and reported his findings in ApacheCon. Check the details at

1. Life After Google Summer of Code by Oscar Castaneda
2. Oscar's GSoC 2010 proposal 
3. Details on how to use his contributions in SVNPlot to extract the data.

SVNPlot - my first opensource project

During the 1 week gap between the two jobs, I finally started an opensource project. The project is called in SVNPlot. It is inspired by the excellent StatSVN Subversion Statistics generation package.

SVNPlot generates graphs similar to StatSVN. The difference is in how the graphs are generated. SVNPlot generates these graphs in two steps. First it converts the Subversion logs into a 'sqlite3' database. Then it uses sql queries to extract the data from the database and then uses excellent Matplotlib plotting library to plot the graphs.

I believe using SQL queries to query the necessary data is resulting great flexibility in data extraction. Also since the sqlite3 is quite fast, it is possible to generate these graphs on demand.

As tribute to python and author of Python-Guido van Rossum, I have generated the graphs for Rietveld project. Check it out here

SVNPlot is hosted on Google code (http://code.google.com/p/svnplot/) and licensed under New BSD license. For information on installation and usage, check the introduction page here

I am using python to implement SVNPlot. I am a novice to python. Hence any suggestions to improvement are welcome.