T h e s i s 

8.1       Summary

The quality of information is a key challenge in designing, developing and maintaining any large-scale information system. In particular, customer information for use in enterprise systems supporting functions like Customer Relationship Management (CRM) systems requires significant organisational resources. Compared with other information systems projects, Information Quality (IQ) initiatives struggle to compete for these scarce organisational resources. This is due in part to the difficulty of articulating a quantitative cost/benefit analysis in financial terms.

The SIFT framework developed and evaluated in the preceding chapters offers analysts a set of constructs, measures and a method to help them assess, prioritise and present disparate IQ improvements as sound organisational investments. Further, this framework allows them to do so in an efficient manner with minimal disruption of operational systems.

The research process followed a Design Science approach, underpinned by a Critical Realist philosophy. Here, the framework was cast as an abstract artefact with utility as the goal. Motivated by a series of practitioner interviews, this artefact was developed through a conceptual study. This was followed by a quantitative investigation using computer simulations and mathematical derivations which provided further detail and empirical support. Finally, the framework was successfully evaluated against a leading set of criteria from the field of Design Science.

8.2       Research Findings

Within Information Systems research, the sub-field of Information Quality has seen a large number of conceptual and practitioner-oriented approaches. However, few efforts are directed at measuring the value created for organisations by investing in customer information quality improvements. Many of the frameworks are burdened by a lack of theoretical rigour, giving rise to unclear definitions, misleading measures and an inability to operationalise constructs that renders them impractical.

The large body of knowledge residing in Information Theory and Information Economics has had little bearing on Information Quality research. The Semiotic IQ Framework (Price and Shanks 2005a), organises a hierarchy of IQ concepts including the Ontological Model for IQ (Wand and Wang 1996) at the semantic level, which is isomorphic to Shannon and Weaver’s model of information (Shannon and Weaver 1949). This presented an opportunity to bridge knowledge from these different domains in order to tackle the difficult IQ investment problem.

A series of semi-structured interviews with practitioners was required to assess the current state of the art within industry and to capture the intended requirements of the framework. This involved interviewing 15 practitioners (analysts, managers and executives) with a range of backgrounds and experiences, for one to two hours about their experiences with IQ investment, measurement and business cases.

The key finding was that while IQ is recognised as an important enabler of value creation within organisations, there is a widespread inability to employ accepted value measurements in accordance with standard organisational practices. Many subjects argued that while they believe they know what is required to remedy specific IQ deficiencies and that doing so would be in their organisation’s interests, the lack of value measures means IQ initiatives cannot compete for and win funding against more traditional IS projects. As a result, this frustrated articulation of the benefits gives rise to widespread under-investment in IQ. The way many IQ initiatives secure funding and advance is from a mandate from a sufficiently senior sponsor. This can also lead to a perceived misallocation of the organisation’s resources.

This need for practitioners to support their business cases with financial arguments provided the central design goal of the framework. Drawing on the quantitative approaches from the earlier Literature Review (Chapter 3), the framework was conceived with customer-level decision-making at its core. This facilitated the use of a variety measurement approaches from the related fields of Decision Theory, Machine Learning and Information Theory. In this framework, CRM processes are re-cast as customer classifiers and their performance can be assessed similarly. More importantly, the effect of IQ deficiencies on classifier performance can be described quantitatively too.

To take the framework to a richer level and allow empirical validation, some real-world scenarios were found in the Data Mining literature. By recreating the target contexts (large-scale automated customer-level decision-making) and inducing the IQ deficiencies with a “garbling” noise process, the consequences of the deficiencies could be explored. In this way, the simulations allowed empirical verification of mathematical modelling of the noise process and its effect on observed errors.

Further, the simulations demonstrated that the proposed Influence metric (Information Gain Ratio, or IGR) is a very good proxy for actionability (the propensity of a representational error to translate into a decision mistake). This is important because IGR is a property of the decision-making function and can be assessed much more quickly and cheaply – and with minimal disruption – compared with the experimental requirements of directly measuring actionability.

Along with Stake, Fidelity and Traction, the Influence metric was embedded in a method for financial modelling of cash flows arising from the costs and benefits and candidate IQ interventions. The method employed an Actionability Matrix to guide the search of processes and attributes to determine efficiently the most valuable opportunities for improvement.

Finally, the framework – comprising the model, measures and method – was evaluated against the set of seven criteria proposed in MISQ (Hevner et al. 2004) by several leading Design Science scholars. The framework (as an abstract artefact) was found to be a purposeful, innovative and generic solution to an important, widespread and persistent problem.

8.3       Limitations and Further Research

The primary limitation of this research is the lack of empirical testing of the framework in its entirety in a realistic situation. Ideally, this would involve the application of the framework by the target users (business analysts within an organisation) to build a business case for customer IQ improvement across a range of customer processes. As discussed in Section 2.6.3, access to an organisation willing to support research like this would likely be difficult, owing to the large commitment in time and resources.

That said, the research evaluation undertaken as part of this project lowers the risk for prospective industry partners, making future collaboration more likely. Specifically, the development and testing of the use of IGR to speed up the search makes further field testing a more attractive proposition.

Undertaking further field trials would allow researchers to assess and understand:

·         The output. How acceptable are the kinds of financial models, valuations and recommendations produced by the framework? To what extent are decision-makers likely to give credence to these outputs, given the complexity and opacity of the underlying model? Does this change with familiarity? What are the determinants of acceptability for these decision-makers?

 

·         The process. What is the effectiveness and efficiency of actually using the framework? What are the determinants of successful use of the framework? What kinds of improvements could be made? Could it be made more generic or does it need to be more focused?

 

·         The context. What motivates organisations to adopt this framework? What background or preparation does an analyst need to use it successfully? How can the outputs be used by the organisation to manage IQ over time or across organisational boundaries?

All of these questions can only be tackled by deploying the framework in a real-world situation, perhaps as a field experiment (Neuman 2000). Practically, to do this kind of research would require an uncommon level of access with an industry partner. Given the immaturity of the framework plus the confidentiality and commercial constraints, an action research approach (eg. Burstein and Gregor 1999) to evaluating the artefact may be a better fit.

The second limitation (and hence opportunity for further research) is the noise process used in the simulations. To recap, a “garbling” noise process was used, which involves iterating through a set of records and swapping a particular field’s values with the value from another record selected at random. The garbling rate, g, was used to control the amount of noise introduced.

The effect of this noise process was modelled to a high level of precision by its mathematical derivation. However, in a practical sense, it represents a particularly “disastrous” noise event, where all traces of the correct value are lost. Such a garbling event might arise from a field value being erased or a database index getting “jumbled” or a data entry error by a clerk.

While this kind of event happens in practice and warrants study in its own right, other kinds of noise will have different properties. Specifically, some will retain some information about the original value. For instance, miscoding a telephone number by one digit is likely to result in a telephone number in the same geographical region. For IQ deficiencies arising from currency issues, the “stale” value is not entirely unrelated to the correct value. For example, with customer residential addresses, customers on the whole are likely to move to postcodes or regions with similar socio-economic conditions. Decisions made on the basis of such stale information may frequently be the same as with correct information. So the errors introduced by the stale address are not as complete or “disastrous” as the ones modelled by the garbling process, which represents the worst-case.

It may be that further research could yield an alternative noise process that captures some of the information-retaining characteristics of more realistic noise. Ideally, any new noise processes would still be amenable to the kind of empirical analysis undertaken here. If not, new methods could be developed for investigation, perhaps based on other areas of applied mathematics.