The major pitfalls of microarray analysis

In the second of a two part series we give you the low down on the final five most common problems that can arise when interpreting DNA microarray results

Pitfall 6:
Using experimental conditions that are different from the error model conditions
The goal of any measurement tool is to provide an estimate of quantitative truth. In the case of DNA microarrays, this ‘truth’ is of differential gene expression. And, like other quantitative tools, DNA microarray measurements are typically associated with an estimate of measurement error.

For DNA microarrays, this error can be systematic error (i.e., error that can be corrected for such as background subtraction or dye normalisation) or random error (i.e., error that cannot be captured, but which can be modelled). To estimate the random error associated with expression measurements from a particular microarray platform, one can perform many hybridisations under identical experimental conditions. With this approach, the normal level of noise associated with a microarray platform can be estimated, independent of biology or technician. Once the random noise associated with a microarray platform is understood, this information can be applied to the interpretation of future, smaller data sets.

While a complete explanation regarding the use of error models is beyond the scope of this paper (for review, see reference 1), the concept behind this pitfall is straightforward. Error models are only accurate under the identical conditions that were used to model the error. In other words, researchers who are implementing an error model in their analyses must use the same experimental conditions that were used for the initial development of the error model. This means using the same labelling protocols, wash conditions, scanner models, and manufacturer’s recommendations. Any changes introduced in the process or protocols may add noise that was not initially present in the experiments used to estimate the random error. As a result, the error model (random error estimation) may not be accurate.

Figure 1: Sample results from a poorly processed microarray experiment. a) The unsupervised hierarchical cluster of 9 microarray hybridisations (Normal vs. Reference; Treated vs. Reference) shows grouping of similar experiments based upon the day of processing rather than sample biology. Here processing noise is greater than biological noise. b) An alternative sample processing strategy randomises the sample handling across two days in order to minimise the processing bias. Here an equal number of untreated samples are processed on Day 1 as treated samples.

Pitfall 7:
Paying more attention to the magnitude of the Log Ratio than the significance of the Log Ratio
In early microarray experiments, many researchers filtered microarray data by defining an arbitrary fold-change cut-off for transcripts, such as a 2-fold cut-off. This refers to the ratio of cyanine 5-red labeled target to cyanine 3-green labeled target. If a microarray spot had an intensity of 10,000 counts in the red channel and 5,000 counts in the green channel it would have a ratio of 10,000/5,000= 2, with Log102=0.3. Because of the poor quality of early microarray technology, only transcripts with a fold change greater than 2 (i.e., x<-2 or x>+2) were considered biologically real. Less attention was given to transcripts with smaller fold changes. Therefore, users focused on the magnitude of the Log Ratio in defining ‘true’ transcriptional differences. This fold-change threshold is depicted in Figure 2.
arton, MJ., Witteveen, AT., Schreiber, GJ., Kerkhoven, RM., Roberts, C., Linsley, PS., Bernards, R. and Friend, S. Gene expression profiling predicts clinical outcome of breast cancer. Nature 415. pp.530-536. (2002).

Another approach for defining ‘true’ transcriptional changes focuses on the significance of the Log Ratio. This refers to a statistical definition of significance whereby an estimate is assigned for the probability (P) that a given Log Ratio could occur by chance alone. In other words, if we assume that no transcriptional differences exist between two RNA samples (i.e., average Log Ratio=0), statistics can be used to estimate if an observed change is consistent with this assumption. For example, a Log Ratio with the estimate P<0.01 would suggest that this Log Ratio measurement would be observed about 1% of the time in repeated samplings by chance alone, assuming that its true Log Ratio=0. The smaller the P-value, the less likely the measurement would be observed by chance alone and the more likely that the change is reflective of true differential expression (i.e., the assumption of Log Ratio=0 is not supported). Another way to think of P-value thresholds is the acceptable false-positive rate for a given microarray experiment. So setting a Log Ratio P-value threshold, such as P <0.001, is another approach to filtering microarray data. Only transcripts that pass this filter (i.e., have a P-value equal to or less than 0.001) are considered statistically significant.

As suggested by this pitfall, users should place equal or greater emphasis on the statistical significance associated with Log Ratio values rather than simply the magnitude of the Log Ratio values. The importance of this point is highlighted on the inset of Figure 2. Here two transcripts of similar mean intensities are shown with different Log Ratio magnitudes. Because the transcript with the larger fold-change also has a large measurement error, it is not considered statistically significant. In this example, the transcript with the smaller measurement error and Log Ratio magnitude is considered significant.

Figure 2: Log Ratio vs. Log Intensity plot of two microarray hybridizations, where each dot represents a transcript’s error-weighted averaged Log Ratio across two hybridisations. Blue dots represent genes that are not considered statistically significant at P<0.01, red dots represent genes that are significantly up-regulated and green dots are genes that are significantly down-regulated. The dashed line represents a 2-fold Log Ratio threshold.

Pitfall 8:
Automatically assuming that Q-PCR, northern blot or RPA analysis is needed to confirm every microarray result
In the early application of DNA microarray technology, it was common to confirm observed expression changes by an alternative technology such as Q-PCR, northern blots, or Ribonuclease Protection Assays (RPA). This confirmation approach was necessary to screen out false-positive results due to the poor quality of early printing methods and the inherent challenges of cDNA clone handling (contamination, PCR issues, re-sequencing, clone tracking/storage, etc.). Today, because of improved manufacturing quality and content quality (in situ oligo synthesis, ink jet technology, no cDNA clone handling), the downstream approaches to data confirmation are not strictly limited to these methods. Rather, the confirmation approach should be consistent with the scientific aim of the experiment.

Figure 3 highlights four experimental applications and the alternative confirmation methodologies that may apply. For example, if DNA microarrays are used to suggest a cellular phenotype that discriminates cluster groups of tumours (Figure 3a), then the confirmation approach may focus on the hypothesised phenotype rather than confirming the specific transcripts comprising the cluster. For example, Bittner et al., predicted differences in cutaneous melanoma spreading and migration based upon DNA microarray results and confirmed this prediction by a series of cellular assays to measure motility and invasiveness2. Similarly, if DNA microarrays are being used to develop a prognostic classifier for metastasis or BRCA1 mutations, then the confirmation approach of the classifier may include testing independent primary tumours or sequencing BRCA1 for putative mutations3.

If the scientific aim were to predict a deregulated cellular pathway following experimental treatment, then the downstream confirmation approach might include cellular assays that test the integrity of the suggested pathway, rather than performing Q-PCR of every transcriptional alteration comprising the pathway (Figure 3b). Other experimental aims (Figure 3c, d) may include functional confirmation and the use of RNAi for target validation.

In summary, the downstream confirmation methods should be consistent with the scientific aim of the experiment. This is not to imply that Q-PCR or similar technologies are no longer of value, but simply to suggest that technological improvements no longer necessitate the confirmation of every transcriptional change in a microarray experiment.

Figure 3: Different experimental goals may necessitate different confirmation methods. The four experimental aims represented here are described in the text.

Pitfall 9:
Cutting upfront costs at the expense of downstream results
Although focus is often placed on the cost of a microarray slide, this can be insignificant relative to the costs associated with sample acquisition and downstream experimentation (Figure 4). Sample acquisition costs may include obtaining precious tumour biopsies, developing animal knockout models, synthesising new compounds, or cloning transfection constructs, for example. These are the costs incurred prior to the microarray hybridisation. Downstream costs may include the time/labour/energy involved with interpreting microarray data as well as the resulting experimental steps that are pursued as a direct result of this interpretation.

Since it only takes about three days to perform a microarray hybridisation (compared to the weeks or months involved with sample acquisition and data interpretation) it is critical that the microarray results are an accurate reflection of biology and not of poor quality microarrays, inappropriate experimental design, or improper sample handling. The time and costs associated with a microarray experiment are generally lower than the time and costs associated with pursuing poor quality data.

One example of this pitfall was previously shown in the first section of this article in the May issue of Laboratory News whereby a scientist modified a labeling protocol in order to increase microarray intensity and to avoid the cost of an optimised commercial labelling kit. A second example involved a customer who substituted Cot-1 DNA blocker that was available in his lab for an empty tube of Cot-1 that was recommended by the microarray vendor. Unfortunately, different Cot-1 preparation methods result in different singleton purity levels that can cross-hybridise to DNA features and interfere with the true Log Ratio measurements. So while both modifications were cost effective and thought to be benign, they had a potentially large impact on the resulting data quality.

This is an important pitfall to consider because many researchers sacrifice the use of quality microarrays, reagents and equipment in an effort to minimise cost. However, by doing so they risk spending months interpreting data that may be less accurate than would have been obtained by a greater investment in the microarray experiment. This investment includes the use of quality microarrays, reagents and scanner, the rigorous adherence to optimised protocols, and the careful consideration of an experimental design that will maximise data interpretation.

Since it only takes a few days to perform a microarray hybridisation, users should invest in this process to maximise the value of the resulting data rather than cutting corners to minimise cost and risk generating data that is less reflective of true biological changes.

Figure 4: Relative costs associated with a microarray experiment.

Pitfall 10:
Pursuing one path in data interpretation
The proper interpretation of DNA microarray results should always be done within the context of biological information, experimental design, and statistical output, as shown in Figure 5. If pursued independently, each individual path in this figure could result in misleading biological interpretation.

First, supporting biological information (such as the experimental hypothesis, clinical information, literature, etc.) is invaluable for interpreting DNA microarray results. However, this knowledge cannot be the sole framework for interpretation in the absence of proper statistics or experimental design considerations. This can lead to biased conclusions or discounted transcriptional observations that conflict with the initial hypothesis. For example, imagine that a scientist predicted cyto-architectural changes resulting from a specific drug treatment in culture. If the data were interpreted solely within the context of the initial hypothesis, the scientist might simply look for cyto-architectural genes in the resulting data and mistakenly overlook other meaningful transcriptional changes. So although the biological context is important, the hypothesis should not bias the interpretation in the absence of statistical methods.

The converse of this is also true. Many microarray facilities employ statisticians to cull microarray data and to identify the relevant transcriptional changes. This is important in order to minimise the pursuit of false leads. However, following this path alone may lead to statistical candidate genes that do not make sense within the context of the experiment (i.e., due to the level of replication, hybridisation design, normal physiological range, etc.). So this path should not be pursued independently.

The true path to success in data interpretation is at the interface of the three paths shown in Figure 5. Data interpretation must be done within the context of biological information, statistical results, and experimental design. As a result, it is recommended that microarray biologists work very closely with statisticians to ensure that statistical interpretation is consistent with the biological/experimental framework of the project.

In summary, the power of DNA microarray technology is widely recognised for its utility in basic research, cancer prognosis, toxicogenomics, and drug discovery. However, its value as a research tool is dependent upon its proper use and appropriate data interpretation. By recognising the common pitfalls in data analysis, new users will minimise the time and costs associated with pursuing false leads and maximise the biological meaning present in microarray
data sets.

Figure 5: Path to success, described in text.

References

1. Delenstarr, G., Cattell, H., Connell, S., Dorsel, A., Kincaid, R., Nguyen, K., Sampas, N., Schidel, S., Shannon, K., Tu, A., and Wolber, P. Estimation of the confidence limits of oligonucleotide microarray-based measurements of differential expression. In: Microarrays: Optical Technologies and Informatics. Michael Bittner, Yidong Chen, Andreas Dorsel, Edward Dougherty, Editors, Proceedings of SPIE Vol. 4266; pp. 120-131 (2001).
2. Bittner, M., Meltzer, P., Chen, Y., Jiang, Y., Seftor, E., Hendrix, M., Radmacher, M., Simon, R., Yakhini, Z., Ben-Dor, A., Sampas, N., Dougherty, E., Wang, E., Marincola, F., Gooden, C., Lueders, J., Glatfelter, A., Pollock, P., Carpten, J., Gillanders, E., Leja, D., Dietrich, K., Beaudry, C., Berens, M., Alberts, D., Sondak, V., Hayward, N.and Trent, J. Molecular classification of cutaneous malignant melanoma by gene expression profiling. Nature 406, pp. 536-540 (2000).
3. Van’t Veer, LJ., Dai, H., Van de Vijver, MJ., He., YD., Hart, A., Mao, M., Peterse, HL., Van der Kooy, K., M

By Scot J. Vacha. Scot is an applications acientist at Agilent Technologies