Your friend, the question mark

Why, and how, to honestly record uncertainty in your wild counts.

written May 21, 2018 (last updated Oct 10, 2021) • by Jon Sullivan • Category: Wild Counting

This is one of a series of in-depth dives into the counting concepts used in WildCounts. We take a close look at usual taxon uncertain added to count, which is done with the mighty question mark.
Question this!
Be suspicious of wild count datasets that don't contain question marks. Even the sharpest experts can't reliably identify everything they see in the wild. In this excerpt of shorthand, a dark butterfly was briefly seen flying by that looked to be a New Zealand red admiral (Ra) but otherwise would have been a yellow admiral.

Guess what? No, don’t! When you’re counting the wild, you should never, ever, EVER guess. Guesses + data = data you can’t trust. Data you can’t trust are pretty much worthless.

It’s very important, when counting the wild, to keep your observations separate from your interpretation. Honestly recorded pure observations are data that can be accurately compared across observers. They can also be compared within one observer’s observations across many years. Recording your interpretations as your counts pollutes your data and riddles these comparisons with doubt.

I don’t trust wild counts that don’t contain uncertainty. It means that uncertainty in what was seen or heard has been replaced with the observer’s interpretations of what they thought they likely saw or heard. That applies just as much to counts you made ten years ago as counts made by other people. There’s no easy way to later untangle reality from mistakes made in interpretations. They need to be kept separate right from the start.

In wild counts that are free of interpretation, two equally diligent observers with differing identification skills should record completely consistent data, if they independently count the same moment in the wild. The more skilled observer’s observations might be more precise than the others’, but they should be equally accurate. That doesn’t happen when observations are polluted with observers’ interpretations of what they probably saw.

If you’re counting species, and you’re not sure what something is, the only correct thing to do is write down that you’re not sure. Don’t skip it and don’t guess. If it was probably a blackbird, you’ve got to write that it was probably a blackbird. You don’t write that it was a blackbird.

This is all particularly important for long term monitoring. Your expertise as an observer will likely improve over time, and your ability may decline as your eye sight and hearing gradually gets worse. Yet, you’ll need every detected wild individual to be recorded accurately, so you can reveal changes in abundance over time.

There’s one easy solution to this: when you’re not sure what something is, you’ve got to honestly record that you’re not sure.

How to do that? Use the mighty question mark!

Unsure? Question mark!

Let’s start with an example. If I’m not 100% sure something is a blackbird, then it gets entered with a question mark.

Blkb 1?

Even if I’m 99% sure it was a blackbird, it still gets a question mark, always. The WildCounts principle is “Never guess and never interpret”, not “Sometimes guess and sometimes interpret”.

If I’m 99% sure that it’s a blackbird, and I state with certainty that it’s a blackbird, then that’s my interpretation of what I observed. It’s not what I observed. Interpretation happens later in the data analysis.

My job when making wild counts is to honestly record exactly what I observe, nothing less and nothing more.

By adding a question mark in this way, when we’re analysing blackbird trends, researchers can look at the trends both including and not including my questionable birds. If we get the same result both times, we can be confident that a trend is robust.

But, wait, you might say, why are you even bothering to count birds if you can’t identify a blackbird? True, they are distinctive birds, but that’s not always the case. If you’re counting every individual of a species, no matter how easy it is to identify, sometimes all you’ll get is a brief glimpse.

For example, that happens when I’m moving along a transect and I glimpse the brown back of what looks like a female blackbird, foraging in the shade under some vegetation. In Christchurch, New Zealand, that’s going to be a blackbird, or perhaps a song thrush. Thrushes are lighter brown on the back (and very speckled on the breast) but that’s not always clear from a view of the back in the shade. If I had time and binoculars, I could work it out. If I don’t have time, or if it flies off, I can write the following and keep moving.

Blkb 1? (or thrush)

(I just say thrush, and not song thrush, here because in New Zealand that’s the only species of thrush we have.)

When you're counting every individual you see of your chosen species, sometimes you'll get only a glimpse.

Usually, I can see enough to enter a count as the species I think it is, just with a question mark. Other times, though, I may be less sure, such as when a small brownish bird flies quickly overhead. It can be too quick to see clearly, catch in binoculars, or photograph. At least it was definitely a bird. Like other birders, I use a generic code for these small birds. Mine is sbb. One small mostly brown passerine bird, flying over my path at my mid-distance, 20—80 m away, would be recorded as the following.

sbb om1f

The important thing for monitoring is that I didn’t ignore what I saw but instead recorded it honestly. During analyses, researchers can then ensure that the trends they uncover are robust to substituting my sbb with various plausible small birds previously seen in this area.

Ensure that the Future You can trust the Now You 100% when you’re counting the wild. Embrace the question mark!

Proportion of my IDs questioned over time
Here's a graph of the proportion of my observations each month that were identifications with question marks, spanning the years when I have been consistently recording many to all bird species, plus a wide range of other species. This is from a total of 472,375 shorthand observations (it excludes my many untranscribed voice recordings). The black points are months that included visits to California. All other counts were made in my home country of New Zealand. While my identification skills have definitely improved, I remain honestly uncertain of the species identification of about 5–10% of the observations. Many of these are birds and butterflies glimpsed flying by.

You’re really unsure!? Double question mark??

There’s also a more drastic scenario that you’ll eventually encounter when you’re counting every individual of your predetermined set of species. That’s when you’re not sure whether what you saw was a living thing at all. This is most likely to happen when making wild counts from a fast moving vehicle. Sometimes I glimpse things whooshing by that I’m really unsure about. That happens especially when I’m counting roadkill, which can be misshapen and broken and often difficult to identify when on the move.

Do I record something only when I’m sure it is, or was, alive? No, I can’t do that. That’s interpretation. What if I get better over time at recognising and identifying roadkill? That would create a false trend of increasing roadkill rates in my data.

Others could also be better, or worse, at recognising roadkill than I am. If we only recorded the roadkill that we confidently recognised, we wouldn’t be able to reliably compare our datasets.

The rule remains: our interpretations must be kept separate from our observations.

It’s in these cases that I use the WildCounts double question mark (??). This, again, encourages me to honestly record what I see without requiring me to record my interpretation of what I saw.

For example, I’m in a car and see something black and flat whooshing by me on the left road verge. It might have been a squashed roadkill blackbird, but could also plausibly have been a blackbird sized piece of cloth or other rubbish. I can record this as follows.

blkb?? lv unsure sq

It’s on the left verge (lv), I’m unsure whether the roadkill is less than 24 hours old (unsure) and it was squashed (sq). Note the double question mark here in blkb??. That’s quite different from blkb?, which would have meant a blackbird or similar looking bird, but definitely a roadkill. In other words, ? means that there was a roadkill but I’m unsure about the species. In contrast, ?? means that there might have been a roadkill.

When you're counting roadkill, sometimes something will whiz by you and you'll think it might have been a roadkill animal but then it might have just been some animal-shaped rubbish.

Embrace the uncertainty

This might all sound a bit silly and pedantic. If you’re experienced enough to know that it was probably a blackbird, why not just write down “blackbird”? Does it really matter whether or not you record when you’re uncertain about your identifications?

In some cases, it might not matter much. Real population trends, if strong and consistent, and will emerge from your data regardless. Any extra noise caused by varying, inaccurate interpretations will only mean that counts need to be collected for a little longer to reveal reliable population trends.

However, in other cases, it definitely will matter. Fake trends can emerge caused by gradual changes in observers’ skills. Real trends can be obscured by the noise of variability in the accuracy of observers’ interpretations.

Importantly, you won’t be able to tell whether or not your trends are real or fake. That is, unless your observations and interpretations are always kept separate.

It’s also just easier to honestly record exactly what you observe. You don’t need to get distracted considering what your best estimate is of what the species was that you just saw, or heard. Just write down that you’re unsure, with ? or ??, and keep on counting the wild.