Artificial Intelligence: what part does AI play in predicting consumer choice?

These are key questions for marketers wanting to make practical use of artificial intelligence – such as for leveraging AI in predicting consumer choice. Yet, before we dig into how artificial intelligence is best used, we need to consider what the term artificial intelligence actually means.

The definition is not as straightforward as it might first seem. Most of us would agree that it is something less than a robot, which does our tasks during the day and discusses philosophical questions with us at night . . .

But what is this? Some take a short cut by saying artificial intelligence is machine learning, which then poses the question of how we define machine learning. However, whether these two terms are the same or just overlap, there is one idea central to both: they involve a machine learning on its own about its environment, to increase its chance of success.

This becomes sharper still if we consider the use of artificial intelligence in predicting consumer choice. In this instance, the machines are computers, their environment is data and success is making better predictions.

AI reveals complex patterns in consumer choice data

By viewing artificial intelligence in this way, we then see that artificial intelligence is already working alongside us, to assist rather than direct. That is, once we define a problem and set the goals, these methods help us reach better decisions by revealing patterns in data that would otherwise elude us.

For example, real AI appears in classification tree analysis (also called CHAID, C&RT, QUEST, C4.5, C5.0, and J48 – among other acronyms). This method uses the values of a predictor variable to split a sample into smaller groups, so that some target variable is at higher levels in some groups and lower levels in others. It then looks at other variables to split these subgroups again repeatedly, until it reaches some stopping criterion. This leads to strongly contrasting groups.

Let’s give a concrete example:

Suppose you are selling a breakfast cereal, which we will call Cereal O. You want to understand how household demographics relate to consumption and you have 46 demographic variables, such as age of householder, income, occupation, numbers and ages of children, marital status and so on.

Overall, households that buy this type of cereal consume 22 boxes a year. Classification trees then scan all possible variables and find that the number of children at home is the best predictor. Households with no children living at home consume the least, followed by households with 1 or 2 children, then households with 3 children, and finally any household with 4 to (a staggering) 14 children. The biggest households ate 225% more of this substance as the smallest ones, with strong differences in between.

CHAID here looked at all possible ways to split this variable (which has 15 values) to determine that making these four precise groups gave the strongest contrast. And recall, it also looked at the other 45 possible predictors as well. These had anywhere from two values to 10 values.

Looking for the best way to split these many variables to find the strongest statistical difference, turns out to be a fiendishly difficult problem. In fact, the first algorithm that could do this successfully was presented at an early conference on artificial intelligence (Biggs, Deville, Suen, 1990).

And indeed, classification trees can do comparisons that would otherwise leave us completely dumbfounded. For instance, one analysis sorted 2500 US postal ZIP codes into 11 groups that differed sharply in purchase levels. (The difference was statistically significant at the 99.9999999999999% level, or tremendously significant.)

We find artificial intelligence (and/or machine learning) in the background of many other methods. Such already-powerful methods for predicting choices as discrete choice modelling, conjoint analysis and maximum-difference scaling have been tremendously strengthened by the application of machine learning approaches, in particular, one called Hierarchical Bayesian Analysis. This method is mind-bendingly complex, but in very broad outline, it learns from the data by ‘borrowing information’ to fill in estimates for individuals where their answers are sparse or missing. Your computer may need to churn away for hours to reach the results, but the estimates are amazingly accurate.

You can learn more about these approaches in Artificial Intelligence Marketing and Predicting Consumer Choice. Other methods we discuss include Bayesian Networks, a remarkably powerful set of methods that learn connections and strengths of effects in complex data sets, as well as reviewing several of the mysterious-sounding ensemble methods, the classification tree methods mentioned above and neural networks.

However, the recommendation systems that some find nearly synonymous with artificial intelligence may not involve any analysis at all. These are the systems that recommend products or services based on a history of purchases, and/or characteristics of a customer or prospect, such as those used by Amazon and Netflix.

However, some stunningly complex methods for recommending have been devised, their record appears spotty at best, even for the most intricate systems (Coffman, 2013; Said & Bellogin, 2014 – ref Comparative recommender system evaluation: benchmarking recommendation frameworks. Proceedings of the 8th ACM Conference on Recommender Systems. RecSys, pp. 129–136).

Systems based on recommending the most popular items or picks of an expert (an actual human being) can do just as well, or better (Masnick, 2012). Much of the talk (and hype) about AI revolves systems like these, which dig into historical data in search of ever-elusive “patterns.” There are two fundamental problems with this focus. The first is practical and the second, and more fundamental, is strategic.

On the practical side, these systems require huge amounts of data to work. For instance, neural networks require masses of examples before they start to work well. In Artificial Intelligence Marketing and Predicting Consumer Choice we tried several neural networks on 176,000 data points. Apparently, this was not enough, and performance was poor. To get an idea of how much might work, one article reported success in identifying flowers when a network was trained on 800,000 images of flowers (Coldeway, 2016). Consider that each image likely consisted of thousands of data points. That is a massive amount of information.

The strategic consideration, while critical, often seems to be overlooked in all the excitement about new technologies. Even the best look at history can only prepare you for the challenges of today (if you are lucky), not the challenges of tomorrow. If you are not lucky, this will prepare you only for the challenges of yesterday.

Yet we have approaches that can do more, which have gained power by the inclusion of machine learning/AI methods. For instance, discrete choice modelling, if done correctly and carefully, allows us to see what would happen in a competitive marketplace if we reconfigure products or services and competitors respond. This method allows us to create real-time market simulations where products or services vary in thousands of ways, including new products entering and old products moving out of the competitive arena.

Similarly, use of an old standby, conjoint analysis, allows us to see how we can reconfigure communications or websites – or reconfigure complex service delivery – playing out thousands of variations and seeing their relative levels of appeal. It is machine learning methods/AI that extend conjoint to become useful in problems of this complexity.

In all of these applications, machine learning/AI assists in the background, doing the heavy analytical lifting and detecting patterns that we never could hope to find without it. We direct, doing the hard thinking about which changes we need to investigate and how to set up the problem. This is diametrically opposed to the highly popular dream of a machine that will define the issue and go out to solve it on its own. Perhaps all the hard work involved, and the knowledge required to get these approaches to work correctly, has led to them seeming less appealing.

You may have noticed that I favour the approach where we think and plan and the machines use their strongest powers to assist us. Yet, machines are getting more impressive and we could well be on the way to ceding our directive power to computers to a completely unprecedented level.

How much do you want to look forward and how much do you want a computer to tell you what you need to see looking backward?

It is up to you to decide the nature of your best strategic choices. At least for now.

Artificial Intelligence Marketing and Predicting Consumer Choice, by Steven Struhl, is available to purchase from Kogan Page now. GMA readers save 20% with discount code BMKAIM20  

Read also:

‘Artificial intelligence first’: how invisible intelligence creates magical customer experiences

How data drives 5 key steps to maximising account-based growth