Like all software companies, we spend a fair share of our time trying to resolve bugs. Many people hear the word bug and it sends shivers up their spine. They envision some beast that eats all of their data never to return it again. Bugs are an acquired taste and for us we actually enjoy a good bug once we can track it down and fix it.

With a couple of major releases in July we hit some interesting bugs. By describing them in detail I think the process of debugging can be an opportunity to play Sherlock Holmes and then achieve that satisfying epiphany in an ah-ha moment. I do wish someone would create a museum or book of bug patterns. Maybe someone has. But a layperson's guide including the debugging process would be ideal.

Case 1: The case of the over green heat map

Biologists like to see data in heat maps. It feels natural for them to try to view the complexity of genes and samples with everything lined up in a colorful grid. We built a tool to view a grid of gene expression data. It was looking fine but we decided to modify our normalization routine to improve the accuracy of the results. Normalization is needed for heat maps because taking numbers from zero to N makes it difficult to set a scale for what should be red and what should be green. In a normal distribution using a bell curve, items that are in the extremes are bright colors and the ones in the middle are lighter versions of the same.

A problem was reported to us; our heat map was showing data that was too bright green, meaning the data wasn’t represented accurately. This caused some stir given that we had just improved it. So, we went to work trying to identify the cause. The first hypothesis suggested outliers. We proposed samples that were 3 standard deviations could be throwing off the whole scale of the heat map.

We took numbers greater than + or - 3 and set them to 3. This seemed to help but things were still too green. We then reviewed the data directly and it looked normal. However one thing was odd; the extremely small numbers were bright green. This led us back to the question of scales and outliers and reexamining our groupings of samples.

One engineer eventually cracked the bug. The small numbers were bright because they weren't recognized in the heat map as small. But why did this occur in the new heat map version? The small near-zero numbers were being read in scientific notation, e.g., .0000003 can also be 3 EXP-6. The heat map recognized it as 3. It changed because the new algorithm was more precise and was carrying more information in it through additional significant digits. More significant digits meant conversion into scientific notation more often.

We modified the database procedures to remove the unnecessary digits and verified the data wasn't presented through the code in the heat map using scientific notation.

Case 2: The mysterious failed report migration

Due to schema changes to a data mart, a number of Crystal reports were modified to encode logical updates to reflect the modified schema. Each were built in a development environment and all successfully tested against the development database.

We then needed to move the reports to QA. Upon migration of the database and schemas, the reports failed in the report server. This was quite odd; the reports worked in development when pointing the report server to the QA database, but failed in QA. We meticulously checked the configuration files and couldn't find a difference. However, there was a pattern in report failures, just the ones we had modified.

Upon verifying the files were the same, I came to the following conclusion: something was different in development vs. QA on the servers and something was different between modified vs. not modified reports.

Given that there was a big clue (error message "could not log on to database"), I compared both the report types modified vs. unmodified for the SQL connection properties. Everything was identical except for one thing. The modified report connection type had switched from SQL SERVER OLEDB to the SQL SERVER NATIVE driver. Upon switching it back in each report to OLEDB, the problem was resolved in QA.

Why? Development is a curious environment vs. QA. In QA, we keep the database and application server separate. In development, both are on the same machine. This meant that the SQL SERVER NATIVE driver was installed with SQL SERVER and was the difference in why it worked in development but not in QA. Case closed.

Dan Housman
Managing Director, Analytical Applications
Recombinant Data Corp.

Labels: Development, Quality Reporting