When Your Research Results Don’t Love You Back

Living in Wellington, you quickly learn that the weather has a personality of its own. One moment it is sunny, the next it is raining.

Sometimes I think PhD life is a little like Wellington weather. There are exciting moments when you achieve something you have worked hard for, and there are days when you look at your results and think, what a mess! 😅 But no matter what, I think even the smallest achievements deserve to be celebrated along the way because it reminds us how far we have come, allows us to appreciate what we have accomplished, and gives us a little more motivation to face whatever challenge comes next.

Research is quite similar. It would certainly be much easier if the data always behaved as expected. Fortunately — or unfortunately — that is rarely how research works. 😅

However, an unexpected research result is not necessarily a bad one.

One experience that taught me a lot about dealing with unexpected results came from my PhD research on the New Zealand carbon market and stock returns.

I was examining how unexpected changes, or shocks, in carbon prices were associated with stock returns. My initial expectation was that a positive carbon price shock would be associated with lower stock returns, particularly because higher carbon prices can increase costs for carbon-exposed firms.

However, my main result was quite different. On average, I found a positive association between carbon price shocks and New Zealand stock returns in some periods. I also found some unexpected results at the industry level.

So, unfortunately, the data did not seem particularly interested in following the story I had in mind. 🙂

At that point, my responsibility was not to find a way to produce the result I wanted. It was to understand why I was getting that result and how much confidence I could place in it.

I needed to determine whether the unexpected findings reflected a genuine feature of the New Zealand market, something related to the data, or a limitation of my research design.

That meant going back to understand the data itself — its coverage, quality, potential biases, measurement issues, and limitations — as well as reassessing my assumptions, methodology, and model specifications.

The first thing I did was step away from the problem for a little while. I went to a concert and had some Vietnamese food afterwards. Apparently, a little music and Vietnamese food can also be part of the research process. 🙂 It was simply a break to refresh my mind, but it helped me come back to the analysis with a clearer and more objective perspective.

Then I approached the problem systematically.

First, I went back to the data itself. I checked the construction and coverage of my sample and looked more closely at the composition of different industries. This was particularly important because the New Zealand stock market is relatively small, and some of my industry groups contained only a small number of listed firms.

Second, I revisited my methodology. I checked my model specifications, ran robustness tests, introduced additional controls, and examined whether the findings were sensitive to different modelling choices. The purpose was not to find a specification that gave me the result I expected, but to understand whether the unexpected relationship was robust.

Third, I explored alternative explanations for the main result. Instead of assuming that the positive association must be wrong, I started asking whether the New Zealand context might be different from the settings studied in some of the existing literature.

For example, New Zealand has a high share of renewable electricity, so many firms may be less directly exposed to increases in carbon-related energy costs than firms in more fossil-fuel-dependent economies. Some firms may be relatively insulated from higher carbon prices, while others could potentially benefit from the transition or changes in relative competitiveness.

I treated these as possible explanations to investigate, rather than immediately presenting them as the explanation for my result. I compared my findings with the literature, discussed alternative mechanisms with my supervisors, and used additional analysis to see which interpretations were more consistent with the evidence.

The analysis therefore shifted from asking why the data did not produce the expected result to examining what the data actually showed and what could reasonably be learned from it.

This provided a clearer understanding of the findings and of how much confidence could be placed in different parts of the evidence.

For the main result, I realised that what initially appeared counterintuitive might reflect features of the New Zealand context that were different from my original assumptions.

At the same time, I became much more cautious about some of the industry-level results. For example, some industries contained only a small number of listed firms, so an unexpected coefficient could potentially be influenced by one or two large firms with particular business characteristics.

Overall, the process helped distinguish the findings that were supported more consistently by the evidence from those that required more cautious interpretation.

Perhaps the same applies to policy analysis. When proposing a new policy, we cannot know exactly how it will work in practice, so being clear about what the evidence supports — and what remains uncertain — is just as important as the recommendation itself.