A Framework for Valuing the Quality of Customer Information

 

 

 

 

Gregory Hill

 

 

 

 

 

Submitted in total fulfilment of the requirements of the degree of Doctor of Philosophy

October 2009

Department of Information Systems

Faculty of Science

The University of Melbourne

 

 

Abstract

This thesis addresses a widespread, significant and persistent problem in Information Systems practice: under-investment in the quality of customer information. Many organisations require clear financial models in order to undertake investments in their information systems and related processes. However, there are no widely accepted approaches to rigorously articulating the costs and benefits of potential quality improvements to customer information. This can result in poor quality customer information which impacts on wider organisational goals.

To address this problem, I develop and evaluate a framework for producing financial models of the costs and benefits of customer information quality interventions. These models can be used to select and prioritise from multiple candidate interventions across various customer processes and information resources, and to build a business case for the organisation to make the investment.

The research process involved:

·         The adoption of Design Science as a suitable research approach, underpinned by a Critical Realist philosophy.

·         A review of scholarly research in the Information Systems sub-discipline of Information Quality focusing on measurement and valuation, along with topics from relevant reference disciplines in economics and applied mathematics.

·         A series of semi-structured context interviews with practitioners (including analysts, managers and executives) in a number of industries, examining specifically information quality measurement, valuation and investment.

·         A conceptual study using the knowledge from the reference disciplines to design a framework incorporating models, measures and methods to address these practitioner requirements.

·         A simulation study to evaluate and refine the framework by applying synthetic information quality deficiencies to real-world customer data sets and decision process in a controlled fashion.

·         An evaluation of the framework based on a number of published criteria recommended by scholars to establish that the framework is a purposeful, innovative and generic solution to the problem at hand.


 

Declaration

This is to certify that:

  1. the thesis comprises only my original work towards the PhD,
  2. due acknowledgement has been made in the text to all other material used,
  3. the thesis is less than 100,000 words in length, exclusive of tables, maps, bibliographies and appendices.

 

 

Gregory Hill

Acknowledgements

I wish to acknowledge:

·         The Australian Research Council for funding the research project,

·         Bill Nankervis at Telstra Corp. for additional funding, guidance and industry access,

·         my supervisor, Professor Graeme Shanks, for his powers of perseverance and persistence,

·         my mother, Elaine Hill, for her long-standing encouragement and support,

·         and finally, my partner, Marie Barnard, for her patience with me throughout this project.

 


 

Table of Contents

1             

Title. 1

Abstract 2

Declaration. 3

Acknowledgements. 4

Table of Contents. 5

List of Figures. 9

List of Tables. 10

Chapter 1 - Introduction. 12

1.1 Overview.. 12

1.2 Background and Motivation. 12

1.3 Outline of the Thesis. 13

1.4 Contributions of the Research. 14

Chapter 2 - Research Method and Design. 18

2.1 Summary. 18

2.2 Introduction to Design Science. 18

2.3 Motivation. 19

2.4 Goals of the Research Design. 20

2.5 Employing Design Science in Research. 21

2.5.1 Business Needs. 22

2.5.2 Processes. 22

2.5.3 Infrastructure and Applications. 23

2.5.4 Applicable Knowledge. 23

2.5.5 Develop/Build. 24

2.5.6 Justify/Evaluate. 25

2.6 Overall Research Design. 27

2.6.1 Philosophical Position. 28

2.6.2 Build/Develop Framework. 30

2.6.3 Justify/Evaluate Framework. 32

2.7 Assessment of Research Design. 32

2.8 Conclusion. 34

Chapter 3 - Literature Review.. 36

3.1 Summary. 36

3.2 Information Quality. 36

3.3 Existing IQ Frameworks. 38

3.3.1 AIMQ Framework. 38

3.3.2 Ontological Framework. 40

3.3.3 Semiotic Framework. 41

3.4 IQ Measurement 44

3.4.1 IQ Valuation. 46

3.5 Customer Relationship Management 47

3.5.1 CRM Business Context 48

3.5.2 CRM Processes. 48

3.5.3 Customer Value. 49

3.6 Decision Process Modelling. 50

3.6.1 Information Economics. 50

3.6.2 Information Theory. 51

3.6.3 Machine Learning. 51

3.7 Conclusion. 53

Chapter 4 - Context Interviews. 56

4.1 Summary. 56

4.2 Rationale. 56

4.2.1 Alternatives. 56

4.2.2 Selection. 57

4.3 Subject Recruitment 57

4.3.1 Sampling. 57

4.3.2 Demographics. 59

4.3.3 Limitations. 60

4.3.4 Summary of Recruitment 61

4.4 Data Collection Method. 61

4.4.1 General Approach. 61

4.4.2 Materials. 62

4.4.3 Summary of Data Collection. 65

4.5 Data Analysis Method. 65

4.5.1 Approach and Philosophical Basis. 66

4.5.2 Narrative Analysis. 68

4.5.3 Topic Analysis. 69

4.5.4 Proposition Induction. 70

4.5.5 Summary of Data Analysis. 71

4.6 Key Findings. 71

4.6.1 Evaluation. 71

4.6.2 Recognition. 73

4.6.3 Capitalisation. 74

4.6.4 Quantification. 76

4.6.5 The Context-Mechanism-Outcome Configuration. 80

4.6.6 Conclusion. 82

Chapter 5 - Conceptual Study. 84

5.1 Summary. 84

5.2 Practical Requirements. 84

5.2.1 Organisational Context 85

5.2.2 Purpose. 86

5.2.3 Outputs. 86

5.2.4 Process. 86

5.3 Theoretical Basis. 87

5.3.1 Semiotics. 88

5.3.2 Ontological Model 88

5.3.3 Information Theory. 90

5.3.4 Information Economics. 94

5.4 Components. 105

5.4.1 Communication. 105

5.4.2 Decision-making. 106

5.4.3 Impact 109

5.4.4 Interventions. 111

5.5 Usage. 114

5.5.1 Organisational Processes. 116

5.5.2 Decision-Making Functions. 116

5.5.3 Information System Representation. 118

5.5.4 Information Quality Interventions. 118

5.6 Conclusion. 121

Chapter 6 - Simulations. 124

6.1 Summary. 124

6.2 Philosophical Basis. 125

6.3 Scenarios. 127

6.3.1 Datasets. 128

6.3.2 Decision functions. 131

6.3.3 Noise process. 132

6.4 Experimental Process. 134

6.4.1 Technical Environment 134

6.4.2 Creating models. 135

6.4.3 Data Preparation. 137

6.4.4 Execution. 138

6.4.5 Derived Measures. 138

6.5 Results and derivations. 139

6.5.1 Effects of Noise on Errors. 139

6.5.2 Effects on Mistakes. 147

6.5.3 Effects on Interventions. 157

6.6 Application to Method. 160

6.7 Conclusion. 164

Chapter 7 - Research Evaluation. 166

7.1 Summary. 166

7.2 Evaluation in Design Science. 166

7.3 Presentation of Framework as Artefact 168

7.4 Assessment Guidelines. 174

7.4.1 Design as an Artefact 174

7.4.2 Problem Relevance. 174

7.4.3 Design Evaluation. 175

7.4.4 Research Contributions. 175

7.4.5 Research Rigour. 176

7.4.6 Design as a Search Process. 177

7.4.7 Communication as Research. 177

Chapter 8 - Conclusion. 180

8.1 Summary. 180

8.2 Research Findings. 180

8.3 Limitations and Further Research. 181

References. 184

Appendix 1. 194

List of Figures

Figure 1 Design Science Research Process Adapted from takeda (1990) 19

Figure 2 Design Science Research Model (Adapted from Hevner et al. 2004, p9). 22

Figure 3 IS Success Model of Delone and Mclean (DeLone and McLean 1992) 36

Figure 4 - PSP/IQ Matrix (Kahn et al. 2002) 39

Figure 5 Normative CMO Configuration. 80

Figure 6 Descriptive CMO Configuration. 81

Figure 7 Use of the Designed Artefact in Practice. 85

Figure 8 Ontological Model (a) perfect (b) flawed. 89

Figure 9 Simplified Source/Channel Model proposed by Shannon. 91

Figure 10 Channel as a Transition Matrix. 92

Figure 11 Augmented Ontological Model 98

Figure 12 (a) Perfect and (b) Imperfect Realisation. 99

Figure 13 Pay-off Matrix using the Cost-based approach. All units are dollars. 100

Figure 14 Costly Information Quality Defect. 102

Figure 15 Breakdown of Sources of Costly Mistakes. 102

Figure 16 Revised Augmented Ontological Model 104

Figure 18 Model if IQ Intervention. 112

Figure 19 Overview of Method. 116

Figure 20 ID3 Decision Tree for ADULT Dataset. 136

Figure 21 Error Rate (ε) vs Garbling Rate (g) 143

Figure 22 Effect of Garbling Rate on Fidelty. 146

Figure 23 Percent Cumulative Actionability for ADULT dataset. 155

Figure 24 Percent Cumulative Actionability for CRX dataset. 155

Figure 25 Percent Cumulative Actionability for GERMAN dataset. 156

Figure 26 Percent Cumulative Actionability for All datasets. 156

Figure 27 High-Level Constructs in the Framework. 169

Figure 28 The Augmented Ontological Model 169

Figure 29 Model of IQ Interventions. 170

Figure 30 Process Outline for value-based prioritisation of iq interventions. 172

 


 

List of Tables

 

Table 1 Possible Evaluation Methods in Design Science Research, adapted from (Hevner et al. 2004) 27

Table 2 ontological stratification in critical realism (Adapted from Bhaskar 1979) 29

Table 3 Guidelines for assessment of Design Science REsearch Adapted from(Hevner et al. 2004) 33

Table 4 Quality Category Information (Adapted from Price and Shanks 2005a) 42

Table 5 Adapted from Naumann and Rolker (2000) 44

Table 6 Subjects in Study by Strata. 59

Table 7 Initial Measure Sets. 64

Table 8 Final Measure Sets (new measures in italics) 64

Table 9 Normative CMO elements. 80

Table 10 Descriptive CMO elements. 81

Table 11 Example of Attribute Influence On a Decision. 117

Table 12 Outline of Method for Valuation. 122

Table 13 ADULT dataset. 129

Table 14 CRX Dataset. 130

Table 15 GERMAN Dataset. 131

Table 16 - Decision Model Performance by Algorithm and Dataset. 136

Table 17 gamma by Attribute and Decision Function. 140

Table 18 Predicted and Observed Error Rates for Three Attributes, a0, c0 and g0. 143

Table 19 Comparing Expected and Predicted Error Rates. 145

Table 20 alpha by attribute and decision function. 148

Table 21 Information Gains by Attribute and Decision Function. 151

Table 22 Correlation between Information Gain and Actionability, by Dataset and Decision Function. 151

Table 23 Information Gain Ratio by Attribute and Decision Function. 153

Table 24 Correlation between Information Gain Ratio and Actionability, by Dataset and Decision Function 153

Table 25 Rankings Comparison. 154

Table 26 Value Factors for Analysis of IQ Intervention. 161

Table 27 Illustration of an Actionability Matrix. 163





Contents
Next:
Chapter 1 - Introduction