Modelling Customer Relationships as Hidden Markov Chains

© 2020 The Authors. This article is licensed under a Creative Commons Attribution 4.0 License Abstract. Models in behavioural relationship marketing suggest that relations between the customer and the company change over time as a result of the continuous encounter. Some theoretical models have been put forward concerning relationship marketing, both from the standpoints of consumer behaviour and empirical modelling. In addition to these, this study proposes the hidden Markov model (HMM) as a potential tool for assessing customer relationships. Specifically, the HMM is submitted via the framework of a Markov chain model to classify customers relationship dynamics of a telecommunication service company by using an experimental data set. We develop and estimate an HMM to relate the unobservable relationship states to the observed buying behaviour of the customers giving an appropriate classification of the customers into the relationship states. By merely accounting for the functional and unobserved heterogeneity with a two-state hidden Markov model and taking estimation into account via an optimal estimation method, the empirical results not only demonstrate the value of the proposed model in assessing the dynamics of a customer relationship over time but also gives the optimal marketing-mixed strategies in different customer state.


INTRODUCTION
In interactive marketing, the lifetime value of a customer is very vital and useful, especially given the fact that we are in the age of relationship marketing [27]. An essential attribute of service quality is the direct interaction with customers and having customer-oriented behaviours [29]. Customer relationship management (CRM) is a set of process and enabling systems supporting a business strategy to build long term, profitable relationship with the customers [23]. For [6] CRM is a customer-focused business strategy that creates and adds value to the company and its customers. The author [9] presents the usefulness of CRM for some managerial problems, chiefly the budgeting of marketing expenditures for customer acquisition. CRM is also used to allocate spending across media (mail vs telephone vs television) and programs (gift vs special price); it furthermore informs decisions for retaining existing customers [18]. Author [19] avers that its use helps firms to achieve a strategic competitive advantage. It had developed as an approach based on maintaining positive relationships with customers, increasing customer loyalty and expanding customer lifetime value [2].
Despite the wide adoption of CRM in the business world, the academic community has been lagging in developing models that could help businesses analyze transaction data to assess customer relationships and put forward a support system for marketing decisions.
Lately, marketing modellers started to address this gap by developing models of customer lifetime value [15,28,31]. However, far less attention has been given to modelling the dynamics of customer relationships and the effect of relationship marketing actions on customer-brand relationships and the customer's choice behaviour.
In addition to the many available tools to assess customer relationships, one useful tool is the hidden Markov model (HMM). HMMs are widely used in science [22], engineering [29] and many other areas [26] and have been successfully applied in engineering problems such as speech recognition [30]. Advanced models such as higher-order HMM with applications in DNA sequences can be found in [7]. Authors [33] ISSN 2413-9009 Section "Technics" 5012 proposed a methodology based on Partially Observable Markov Decision Processes (POMDPs) with general state and action spaces, under a fractional discounted criterion, a model that combines the Hidden Markov Models with the Markov Decision Process by capturing both dynamics that depend on unobserved states and effects of decisions over time. However, not much work has been done in the field of marketing science.
In an interest in finding patterns in customer relationships and discovering hidden relationship states, this study focuses on the use of the HMM to capture the correct dynamic buying behaviour of customers to services provided by a telecommunication company in Nigeria, proceeding also to find their relationship state over some time. With the relationship states being a finite set which follows the markovian property, transitions between states are determined by the history of interactions between the customers and the brand.

Related Literature
Research in the area of relationship marketing has been emerging in the past decade, both from the consumer behaviour perspective [14] and the empirical modelling perspective [4,28,34]. Our focus will be on those relevant to this research.
Authors [17] applied the probabilistic method of Markov chain to the systematic study of brand switching and brand loyalty. After that this application developed slowly. Researchers tried to use it in other dimensions of CRM, such as analyzing and predicting unique customer behaviour [12,29]. Authors [37] applied a Markov chain model in quality function deployment to analyze customer requirements. Their proposed approach provides a decision-maker to investigate and then satisfy both present and future customers' needs. There probabilistic nature of Markov chain has a crucial impression on decision making [25]. Authors [20] proposed a data mining model that considers ecustomer activities through a discrete-time semi-Markov process. Researchers [21] use this modelling technique in a macro marketing decision level, applying the Markov decision process to forecast markets share and determine customer's decision pattern in the diverse circumstances that are common.
Authors [28] constructed and estimated a nonhomogenous HMM to model the transitions among hidden relationship states. They used HMM to overcome the problem of the unobserved state, as well as to describe a set of hidden conditions and changes between them; also, they translated these hidden states into observed customers' behaviours. HMM often encountered researchers with estimating the models' parameters. There are several estimation methods and algorithms, such as expectation-maximization (EM). Authors [36] used this algorithm to estimate HMM's parameters and utilized the Viterbi algorithm to find some path which was describing customers migration pattern in an online retailer. Also, [32] according to Bayesian rules obtained the conditional probability and calculated the Equation that was referred as likelihood function and then designed a classifier based on HMM for discovering which customer is loyal and which is not dedicated. Authors [10,24] used HMM as a portfolio optimization technique. Their basic idea is to describe the essential movement of the stock price using HMM and to calculate the optimal portfolio using HMM's recursive algorithms. Authors [37] constructed a specific HMM for web browsing, which could prospect whether the users have the intention to purchase in real-time. It can be useful to be ready for customers' needs in the buying process and predicting repetitive customer attitude [5,16,29].
gives the conditional probability distribution of each observation symbol within a given hidden state with the definition of denotes the th k observation symbol per state.
The initial state distribution For convenience, we used the compact notation

Hidden Markov Model of Customer Relationships
The HMM presented here is an individual-level model of buying behaviour. We consider a set of customers, each of whom is involved in repeated interactions of the service provided by a telecommunication company. Accordingly, we define a set of hidden relationship states, which differ for the strength of the relationship between the customer and the brand and the conditional likelihood of choice given the relationship state. A first-order random walk model characterizes the stochastic relationship-states model. The transitions between the Markovian states are probabilistically determined and are affected by relationship encounters [28].
The proposed HMM consists of three main components: (1) the initial state distribution (π), (2) a sequence of random walk transitions (Q) that is express in a probabilistic manner, the likelihood that the series of customer-brand interactions in the previous period were strong enough to transition the customer to an adjacent state, (3) a vector of state-dependent choice probabilities given the relationship states (m). The possibility of observing a sequence of choices is defined [28] by: where, it S is customer i's state at time t in a Markov process with N states, and it Y is customer i's choice at time t.
Following Equation (1) the three components of the HMM are defined as follows: 1) The initial state distribution -the probability that customer i in state s at the time one can be defined as: 2) The transitions -the probability that a customer transitions from the state . The transition matrix is defined as, Following the random walk transition, all terms that are two states or more away from the diagonal are set to zero. Each one of the matrix elements represents a probability of change to an adjacent state or a likelihood of staying in the current state.
3) The state-dependent choice shows the probability that the customer will choose the product at time t conditioned on her state is defined [28] as: Most times, the model parameters are obtained by using the EM algorithm, which results in a Section "Technics" 5014 nonlinear optimization problem [7,22,28]. Still, the EM algorithm does not work for the HMM here, so an optimal estimation method is used for obtaining the transition probabilities among the hidden states which guarantee convergence and global optimum.

Estimation procedure
Here, the work presents the procedure used to estimate the model described above. In choosing the estimation procedure, we focus on properly accounting for observed and unobserved heterogeneity. An optimal estimation procedure, according to [8], is used to estimate the model. The parameter estimation method is an HMM with two hidden states and two observable states.
The number of m hidden states and the stationary distribution of the hidden states is given by: The number of n observable state and the stationary distribution of the observable states is (pi1, pi2, . . . , pin).
Given an observed sequence of the observable states, the occurrences of each state in the series are calculated and hence the experimental distribution q. If the hidden stats are ignored, the visual states follow the one-step transition probability matrix: . is chosen to be the square of the L2norm. When . is determined to be L1-norm or the L ? -norm, the resulting optimization problem are transformed into a linear programming problem.
For the model having two hidden states (m = 2), 11 12 1   Then the optimal value of is given as follows: noting that Here, ...,  is the standard inner product on the vector space n R

Empirical Results and Discussion
Data Employed. The data records two services offered by a telecommunication company in Lagos State, Nigeria and represented by 'V' (voice) and 'D' (data). From the customer database, information is obtained on the expenditure distribution of 93 randomly chosen customers on these services for four months. The customers are classified into two groups based on responses observed. The data is split into two, which is the 'train data' to build an HMM and the 'test data' to validate the constructed HMM. Given the expenditure distribution of a customer observed from the customer's sequence of purchases, and the average expenditure distributions, the estimation procedure above is used to compute the transition probability α in the hidden states. The parameter estimation procedure is done using the R statistical software.
Through observing the responses of customers, customers are classified into two groups, among which 60 customers being loyal (Group 1) and the other 33 customers not-loyal (Group 2). For the train data, two-third of the customers are randomly selected from Group 1 and Group 2. This implies that for the train data, 40 customers are randomly selected (these customers are labelled in the first 40 customers in Table 1) from Group 1, and 22 customers are selected from Group 2. The remaining 20 customers from Group 1 and 11 customers from Group 2 are used for validating the constructed HMM. An HMM of two observable states (V and D) and two hidden forms (Group 1 and Group 2) is built. From the information of the customers in Group 1 and Group 2, the probability distributions of customers' purchases and the emission probability distributions for both groups are computed in Tables 1 and 2.  To find the corresponding optimal state given the observation sequence, the transition probability q is the probability distribution of the observation sequence of each customer.    Tables 5, 6 are the transition probabilities and the long-term behaviour of the hidden Markov chain.
The pattern of the one-step transitions of the customers based on their purchases with cutoff 0.67 separating the two related groups shows a higher percentage of loyal customers and a frequent movement from loyal customers to loyal customers.

Empirical Findings
Based on the analysis, the emission probability of the observations showed a 95% probability of a loyal customer making a purchase in service 'V' and a 49% probability of an unloyal customer purchasing service 'V' (see Table II). In measuring the loyalty of the customers to this service, the transition probability showed a 94% probability of a loyal customer remaining loyal and a 6% probability of a loyal customer becoming unloyal.
It also showed a 76% probability of an unloyal customer remaining unloyal and 24% of an unloyal customer becoming loyal (see Table 5). This led to a higher possibility of an unloyal customer becoming reliable than a loyal customer becoming unloyal. Loyal and unloyal customers were separated using cutoff value 0.67 which implied that a customer is loyal if 0.67

 
and is unloyal if 0.67   . A likely future state for a loyal or unloyal customer is predicted with an 80% possibility of a loyal customer remaining loyal and a 20% possibility of an unloyal customer remaining unloyal (see Table 6).

CONCLUSION
This study is based on modelling customer relationships as hidden Markov chains. The importance of good customer relationships to any company cannot be over-emphasized. Every business requires customers, but it is not enough to have customers, companies must be able to manage their customers and build good relationships with them. In this study, a Hidden Markov model (HMM) is utilized to study customer's relationship dynamics of a telecommunication service company using data on purchases made by these customers. The HMM developed in this research helped to relate the relationship-states Markov chain process to the observed buying behaviour by identifying customers purchases for two services provided and then obtained the hidden relationship states of customers through the observed