Incoherent Comparison of Ratios
Ratios, rates, averages, and so on can be a bad way to compare two different things. The subjects may have very different scales in absolute terms, or they may suffer from different externalities. Here are some examples of this kind of mistake.
“The conversion rate of webpage A is worse than the conversion rate of webpage B, therefore we should work on webpage A.”
Ignores scale If we’re talking about purchases, ignores dollar value
Conversion rates are also sensitive to factors other than design:
- traffic source (search vs. email, etc)
- things being sold (tractors vs. sushi)
“Our mobile app users spend more on average than users of our web app, therefore we should ask web visitors to install the mobile app.”
It is very common in practice for teams to notice a correlation like this, and to become confused about the direction of causality.
Comparing the average value of users across platforms misses the fact that users are choosing different platforms. Mobile applications are often proactively installed by your most engaged users. The effect of the platform upon average value may be small or nonexistent. Users may use different platforms because they are different.
In other words, even if native mobile and web users are spherical, their respective radii may vary.
You will convert some visitors to native app users this way, and native facilities such as push notifications may alter their behavior in valuable ways. But you have to consider the entire system:
value = TK loss of conv on web, increase in LTV
“We are investing in this segment because it is the fastest growing.”
Ignores the relative scale - 1 to 2 miniscule improvements in conversion in your largest customer segment may be vastly larger than massive improvements in a small one
TK