Business Intelligence A Managerial Perspective on Analytics, 3E Ramesh Sharda Dursun Delen Efraim Tu

Business Intelligence A Managerial Perspective on Analytics, 3E By Ramesh

Email: Richard@qwconsultancy.com

CHAPTER

1

An Overview of Business Intelligence, Analytics, and Decision Support

Learning Objectives for Chapter 1 ▪

Understand today’s turbulent business environment and describe how organizations survive and even excel in such an environment (solving problems and exploiting opportunities)

▪

Understand the need for computerized support of managerial decision making

▪

Describe the business intelligence (BI) methodology and concepts

▪

Understand the various types of analytics

CHAPTER OVERVIEW The business environment (climate) is constantly changing, and it is becoming more and more complex. Organizations, private and public, are under pressures that force them to respond quickly to changing conditions and to be innovative in the way they operate.

Such activities require organizations to be agile and to make frequent and quick strategic, tactical, and operational decisions, some of which are very complex. Making such decisions may require considerable amounts of relevant data, information, and knowledge. Processing these, in the framework of the needed decisions, must be done quickly, frequently in real time, and usually requires some computerized support.

This book is about using business analytics as computerized support for managerial decision making. It concentrates both on the theoretical and conceptual foundations of decision support, as well as on the commercial tools and techniques that are available. This introductory chapter provides more details of these topics as well as an overview of the book. This chapter has the following sections: 1 .

CHAPTER OUTLINE 1.1

OPENING VIGNETTE: MAGPIE SENSING EMPLOYS ANALYTICS TO MANAGE A VACCINE SUPPLY CHAIN EFFECTIVELY AND SAFELY  Questions for the Opening Vignette A. WHAT WE CAN LEARN FROM THIS VIGNETTE

1.2

CHANGING BUSINESS ENVIRONMENTS AND COMPUTERIZED DECISION SUPPORT A. THE BUSINESS PRESSURES–RESPONSES–SUPPORT MODEL 1.

The Business Environment 2.

Organizational Responses: Be Reactive, Anticipative, Adaptive, and Proactive 3.

Closing the Strategy Gap  Section 1.2 Review Questions

1.3

A FRAMEWORK FOR BUSINESS INTELLIGENCE (BI) A. DEFINITIONS OF BI B. A BRIEF HISTORY OF BI C. THE ARCHITECTURE OF BI D. THE ORIGINS AND DRIVERS OF BI ⧫ Application Case 1.1: Sabre Helps Its Clients Through Dashboards and Analytics E. A MULTIMEDIA EXERCISE IN BUSINESS INTELLIGENCE  Section 1.3 Review Questions

1.4

INTELLIGENCE CREATION, USE, AND BI GOVERNANCE A. A CYCLICAL PROCESS OF INTELLIGENCE CREATION AND USE B. INTELLIGENCE AND ESPIONAGE  Section 1.4 Review Questions

1.5

TRANSACTION PROCESSING VERSUS ANALYTIC PROCESSING  Section 1.5 Review Questions

1.6

SUCCESSFUL BI IMPLEMENTATION A. THE TYPICAL BI USER COMMUNITY B. APPROPRIATE PLANNING AND ALIGNMENT WITH THE BUSINESS STRATEGY C. REAL-TIME, ON-DEMAND BI IS ATTAINABLE D. DEVELOPING OR ACQUIRING BI SYSTEMS E. JUSTIFICATION AND COST-BENEFIT ANALYSIS F. SECURITY AND PROTECTION OF PRIVACY G. INTEGRATION OF SYSTEMS AND APPLICATIONS  Section 1.6 Review Questions 2 .

1.7

ANALYTICS OVERVIEW A. DESCRIPTIVE ANALYTICS ⧫ Application Case 1.2: Eliminating Inefficiencies at Seattle Children’s Hospital ⧫ Application Case 1.3: Analysis at the Speed of Thought B. PREDICTIVE ANALYTICS ⧫ Application Case 1.4: Moneyball: Analytics in Sports and Movies ⧫ Application Case 1.5: Analyzing Athletic Injuries C. PRESCRIPTIVE ANALYTICS ⧫ Application Case 1.6: Industrial and Commercial Bank of China (ICBC) Employs Models to Reconfigure Its Branch Network D. ANALYTICS APPLIED TO DIFFERENT DOMAINS E. ANALYTICS OR DATA SCIENCE?  Section 1.7 Review Questions

1.8

BRIEF INTRODUCTION TO BIG DATA ANALYTICS A. WHAT IS BIG DATA? ⧫ Application Case 1.7: Gilt Groupe’s Flash Sales Streamlined by Big Data Analytics  Section 1.8 Review Questions

1.9

PLAN OF THE BOOK

1.10

RESOURCES, LINKS, AND THE TERADATA UNIVERSITY NETWORK CONNECTION A. RESOURCES AND LINKS B. VENDORS, PRODUCTS, AND DEMOS C. PERIODICALS D. THE TERADATA UNIVERSITY NETWORK CONNECTION E. THE BOOK’S WEB SITE

Chapter Highlights Key Terms QUESTIONS FOR DISCUSSION Exercises Teradata University Network (TUN) and Other Hands-On Exercises End of Chapter Application Case: Nationwide Insurance Used BI to Enhance Customer Service  Questions for the Case References

3 .

TEACHING TIPS/ADDITIONAL INFORMATION

       

The purpose of any introductory chapter is to motivate students to be interested in the remainder of the course (and book). The real-life cases, beginning with Magpie Sensing and continuing with the others, will show students that business intelligence is not just an academic subject; it is something real companies use that makes a noticeable difference to their bottom line. So, try to relate the subject matter to these cases. For example, consider the types of actions managers take to counter pressures, especially the list of organizational responses. The opening case about Magpie illustrates several of the options available to health care companies, such as innovation, partnerships with others in the cold chain, and the use of IT to improve data access. The other cases in the chapter offer other examples of managerial actions taken in response to pressure. By referring back to this list when discussing other cases, you demonstrate the unity of the analytics field.

All this should show students that a new professional who understands how information systems can support decision making, and can help his or her employer obtain those benefits, has a bright career path. Since students in this course are typically within a year of graduation, that will get their attention!

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 1.1 Review Questions 1.

What information is provided by the descriptive analytics employed at Magpie Sensing?

The descriptive analytics monitor and report the properties of the cold storage system, including the set point of each thermostat, the typical range of temperatures in the system, and the duty cycle of each compressor. These values tell trained personnel whether each storage unit is properly configured for storing particular products.

2.

What type of support is provided by the predictive analytics employed at Magpie Sensing?

The predictive analytics use the descriptive data to alert users when a unit is not configured properly for storing the products. It also sends an alert when average temperature and compressor cycle runs signal that a temperature may go out of range (for example, after a power failure) or that there may have been a human error, such as failure to shut a door. These analytics tell users about a problem that may occur, giving them time to prevent the problem.

3.

How does prescriptive analytics help in business decision making?

4 .

Prescriptive analytics guides decision makers to the alternative with the greatest benefits. In Magpie’s cold chain system, prescriptive analytics uses data about storage unit performance to help buyers select the best storage units. And based on data about storage system efficiency and product sensitivity, prescriptive analytics guides decisions about where to distribute particular products in the supply chain.

4.

What are possible ways to report actionable information in real time to the concerned users of the system?

The system uses shippable wireless monitors. These continuously measure temperature, humidity, and location and transmit the data to the computer that analyzes the data.

5.

What other situations might need real-time monitoring applications?

Answers will vary, but students should consider other systems where current performance provides information about changes or decisions that would improve future performance. Examples include monitoring inventory levels at a manufacturing company to determine when to replenish stocks, monitoring sales in a store to identify when and how to adjust the mix of products, and monitoring patient status in a hospital to identify situations where different treatment is required or devices are malfunctioning. (If students have difficulty with this question, it suggests a need to devote some class time to discussing practical applications of data and analytics in business, government, and nonprofit sectors.)

Section 1.2 Review Questions 1.

List the components of and explain the Business Pressures–Responses–Support model.

The components of the pressure-response-support model are business pressures, companies’ responses to these pressures, and computerized support. The model suggests that responses are made to counter the pressures or to take advantage of opportunities, support facilitates monitoring the environment (e.g., for opportunities) and enhances the quality of the responses.

2.

What are some of the major factors in today’s business environment?

The major factors in today’s business environment are: •

Markets: strong competition, expansion of global markets, booming electronic markets on the Internet, innovative marketing methods, opportunities for outsourcing with IT support, and need for real time ondemand transactions

5 .

3.

•

Consumer demand: the desire for customization, quality, diversity of products, and speed of delivery; more powerful and less loyal customers.

•

Technology: more innovations, new products and services, increasing obsolescence rate, increasing information overload, social networking, Web 2.0 and beyond

•

Societal: growth of government regulations and deregulation; work force becoming more diversified, older, and composed of more women; prime concerns of homeland security and terrorist attacks; necessity of compliance with the Sarbanes-Oxley Act and other reporting-related legislation; increasing social responsibility of companies; greater emphasis on sustainability

What are some of the major response activities that organizations take?

Responses taken by organizations include the following reactive, anticipative, adaptive, and proactive activities: •

Employing strategic planning

•

Using new and innovative business models

•

Restructuring of business processes

•

Participating in business alliances

•

Improving corporate information systems

•

Improving partnership relationships

•

Encouraging innovation and creativity

•

Improving customer service and relationships

•

Moving to electronic commerce (e-commerce)

•

Moving to make-to-order production and on-demand manufacturing and services

•

Using new IT to improve communication, data access (discovery of information), and collaboration

•

Responding quickly to competitors’ actions (e.g., in pricing, promotions, new products and services)

•

Automating many tasks of white-collar employees 6 .

•

Automating certain decision processes especially those dealing with customers

•

Improving decision making by employing analytics

Since the question reads “some,” a correct answer need not list all of these.

Section 1.3 Review Questions 1.

Define BI. Business Intelligence (BI) is an umbrella term that combines architectures, tools, databases, analytical tools, applications, and methodologies. Its major objective is to enable interactive access (sometimes in real time) to data, enable manipulation of these data, and provide business managers and analysts the ability to conduct appropriate analysis.

2.

List and describe the major components of BI. BI systems have four major components: the data warehouse (with its source data), business analytics (a collection of tools for manipulating, mining, and analyzing the data in the data warehouse), business performance management (for monitoring and analyzing performance), and the user interface (e.g., a dashboard).

Section 1.4 Review Questions 1.

List the steps of intelligence creation and use.

See Figure 1.4 for the process and steps of intelligence creation and use.

2.

What is BI governance?

Some organizations refer to the project prioritization process as a form of BI governance. A major governance issue is who should serve as decision makers involved in prioritizing BI projects. The two critical partnerships required for BI governance are: (1) a partnership between functional area heads and/or product/service area leaders, and (2) a partnership between potential customers and providers.

3.

What is intelligence gathering?

Modern companies must ethically and legally organize themselves to glean as much as they can from their customers, their business environment, their stakeholders, their business processes, their competitors, and other such sources of potentially 7 .

valuable information. But collecting data is just the beginning. Vast amounts of that data need to be cataloged, tagged, analyzed, sorted, filtered, and must undergo a host of other operations to yield usable information that can impact decision making and improve the bottom line. The importance of these topics increases every day as companies track and accumulate more and more data.

Section 1.5 Review Questions 1.

Define OLTP. OLTP (online transaction processing) is a type of computer processing where the computer responds immediately to user requests. Each request is considered to be a transaction, which is a computerized record of a discrete event, such as the receipt of inventory or a customer order.

2.

Define OLAP. OLAP (online analytical processing) is processing for end-user ad hoc reports, queries, and analysis. Separating the OLTP from analysis and decision support provided by OLAP enables the benefits of BI that were described earlier and provides for competitive intelligence and advantage as described next.

Section 1.6 Review Questions 1.

Describe the major types of BI users.

BI may have a larger and more diversified user community. One of the most important aspects of a successful BI is that it must be of benefit to the enterprise as a whole. This implies that there are likely to be a host of users in the enterprise—many of whom should be involved from the outset of a DW investment decision. Not surprisingly, there are likely to be users who focus at the strategic level and those who are more oriented to the tactical level.

2.

List some of the implementation topics addressed by Gartner’s report.

Gartner’s framework decomposes planning and execution into business, organization, functionality, and infrastructure components. At the business and organizational levels, strategic and operational objectives must be defined while considering the available organizational skills to achieve those objectives. Issues of organizational culture surrounding BI initiatives and building enthusiasm for those initiatives and procedures for the intra-organizational sharing of BI best practices must be considered by upper management—with plans in place to prepare the organization for change.

3.

List some other success factors of BI. 8 .

If the company’s strategy is properly aligned with the reasons for DW and BI initiatives, and if the company’s IS organization is or can be made capable of playing its role in such a project, and if the requisite user community is in place and has the proper motivation, it is wise to start BI and establish a BI Competency Center (BICC) within the company. The center could serve some or all of the following functions.

•

The center can demonstrate how BI is clearly linked to strategy and execution of strategy.

•

A center can serve to encourage interaction between the potential business user communities and the IS organization.

•

The center can serve as a repository and disseminator of best BI practices between and among the different lines of business.

•

Standards of excellence in BI practices can be advocated and encouraged throughout the company.

•

The IS organization can learn a great deal through interaction with the user communities such as knowledge about the variety of types of analytical tools that are needed.

•

The business user community and IS organization can better understand why the data warehouse platform must be flexible enough to provide for changing business requirements.

•

It can help important stakeholders like high-level executives see how BI can play an important role.

Another important success factor of BI is its ability to facilitate a real-time, ondemand agile environment.

4.

Why is it difficult to justify BI applications?

Justifying BI applications is difficult because of the large number of intangible benefits. Both direct and intangible benefits need to be identified.

Section 1.7 Review Questions 1.

Define analytics.

The term replaces terminology referring to individual components of a decision support system with one broad word referring to business intelligence. More precisely, analytics is the process of developing actionable decisions or recommendations for actions based upon insights generated from historical data.

9 .

Students may also refer to the eight levels of analytics and this simpler descriptive language: “looking at all the data to understand what is happening, what will happen, and how to make the best of it.”

2.

What is descriptive analytics? What are the various tools that are employed in descriptive analytics?

Descriptive analytics refers to knowing what is happening in the organization and understanding some underlying trends and causes of such occurrences. Tools used in descriptive analytics include data warehouses and visualization applications.

3.

How is descriptive analytics different from traditional reporting?

Descriptive analytics gathers more data, often automatically. It makes results available in real time and allows reports to be customized.

4.

What is a data warehouse? How can data warehousing technology help to enable analytics?

A data warehouse, introduced in Section 1.7, is the component of a BI system that contains the source data. As described in this section, developing a data warehouse usually includes development of the data infrastructure for descriptive analytics—that is, consolidation of data sources and making relevant data available in a form that enables appropriate reporting and analysis. A data warehouse serves as the basis for developing appropriate reports, queries, alerts, and trends.

5.

What is predictive analytics? How can organizations employ predictive analytics?

Predictive analytics is the use of statistical techniques and data mining to determine what is likely to happen in the future. Businesses use predictive analytics to forecast whether customers are likely to switch to a competitor, what customers are likely to buy, how likely customers are to respond to a promotion, and whether a customer is creditworthy. Sports teams have used predictive analytics to identify the players most likely to contribute to a team’s success.

6.

What is prescriptive analytics? What kind of problems can be solved by prescriptive analytics?

Prescriptive analytics is a set of techniques that use descriptive data and forecasts to identify the decisions most likely to result in the best performance. Usually, an organization uses prescriptive analytics to identify the decisions or actions that will optimize the performance of a system. Organizations have used prescriptive

10 .

analytics to set prices, create production plans, and identify the best locations for facilities such as bank branches.

7.

Define modeling from the analytics perspective.

As Application Case 1.6 illustrates, analytics uses descriptive data to create models of how people, equipment, or other variables operate in the real world.

These models can be used in predictive and prescriptive analytics to develop forecasts, recommendations, and decisions.

8.

Is it a good idea to follow a hierarchy of descriptive and predictive analytics before applying prescriptive analytics?

As noted in the analysis of Application Case 1.5, it is important in any analytics project to understand the business domain and current state of the business problem. This requires analysis of historical data, or descriptive analytics.

Although the chapter does not discuss a hierarchy of analytics, students may observe that testing a model with predictive analytics could logically improve prescriptive use of the model.

9.

How can analytics aid in objective decision making?

As noted in the analysis of Application Case 1.4, problem solving in organizations has tended to be subjective, and decision makers tend to rely on familiar processes. The result is that future decisions are no better than past decisions.

Analytics builds on historical data and takes into account changing conditions to arrive at fact-based solutions that decision makers might not have considered.

Section 1.8 Review Questions 1.

What is Big Data analytics?

The term Big Data refers to data that cannot be stored in a single storage unit.

Typically, the data is arriving in many different forms, be they structured, unstructured, or in a stream. Big Data analytics is analytics on a large enough scale, with fast enough processing, to handle this kind of data.

2.

What are the sources of Big Data?

Major sources include clickstreams from Web sites, postings on social media, and data from traffic, sensors, and the weather.

11 .

3.

What are the characteristics of Big Data?

Today Big Data refers to almost any kind of large data that has the characteristics of volume, velocity, and variety. Examples include data about Web searches, such as the billions of Web pages searched by Google; data about financial trading, which operates in the order of microseconds; and data about consumer opinions measured from postings in social media.

4.

What processing technique is applied to process Big Data?

One computer, even a powerful one, could not handle the scale of Big Data. The solution is to push computation to the data, using the MapReduce programming paradigm.

Section 1.9 Review Questions (This section has no review questions.)

Section 1.10 Review Questions (This section has no review questions.)

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 1.1: Sabre Helps Its Clients Through Dashboards and Analytics

1.

What is traditional reporting? How is it used in the organization?

The traditional reporting process is a manual process of collecting and aggregating financial and other information. Organizations have used this timeconsuming process as a way to obtain information for making decisions.

However, the resulting presentations may be flat, slow to develop, and difficult to apply to specific situations.

12 .

2.

How can analytics be used to transform traditional reporting?

Analytics can enable real-time decision support and deliver information to a userfriendly dashboard. Users of a dashboard such as the one provided by Sabre’s Enterprise Travel Data Warehouse can see at a glance a 360-degree view of the company’s overall health generated from various data sources. Many stakeholders in the organization can request data needed for particular types of decisions, and the graphical user interface makes the information easily understandable.

3.

How can interactive reporting assist organizations in decision making?

When a system incorporates interactive drill-down capabilities, users can select the data they need for evaluating a specific kind of performance and making decisions in a particular function or situation. For example, the airlines using the Sabre system can focus on data about sales performance (ticketing, seats sold, etc.) and operational performance (flight movement, inventory, etc.). This flexibility encourages decision makers to use data in support of their decisions.

Application Case 1.2: Eliminating Inefficiencies at Seattle Children’s Hospital

1.

Who are the users of the tool?

The business intelligence application at Seattle Children’s Hospital is used by data analysts, business managers, financial analysts, clinicians, doctors, and researchers.

2.

What is a dashboard?

A dashboard is a user interface that displays information in some type of visual system. For example, it shows current performance measured against standards.

3.

How does visualization help in decision making?

13 .

Visualizations help users gain insights into existing data. Exploring data visually can help users identify the root causes of problems and provide a basis for working toward solutions.

4.

What are the significant results achieved by the use of Tableau?

Tableau’s business intelligence application lets users create monthly and daily dashboards that have helped users significantly improve day-to-day decision making. Visualizing data about patient wait-times helped the hospital improve overall waiting time and improve the availability of beds. The hospital also saved about $3 million from the supply chain and has made processes more efficient.

Application Case 1.3: Analysis at the Speed of Thought

1.

What are the desired functionalities of a reporting tool?

A reporting tool should generate reports that make visualizations easily discernible. It should help users make sense out of the reports, identify problem areas, and contribute to improvements. It also should be able to handle the volume of data required and prepare reports within the necessary time frame.

2.

What advantages were derived by using a reporting tool in this case?

At Kaleida Health, Tableau’s reporting tool handles more data faster than its traditional reporting tools did. Also, the analytic capability is deeper. The reports provide detailed data for analyzing efficiency and comparing the performance of Kaleida’s hospitals with other hospitals nationwide.

Application Case 1.4: Moneyball: Analytics in Sports and Movies

1.

How is predictive analytics applied in Moneyball?

The Oakland Athletics hired an assistant general manager with a background in economics and computer expertise. He followed a Sabermetric approach, which uses baseball statistics to analyze the game. He built a prediction model that selected players based on their on-base percentage, which measures how often a 14 .

batter reaches base for any reason other than fielding error, fielder’s choice, dropped or uncaught third strike, fielder’s obstruction, or catcher’s interference.

This strategy built a team that won 20 consecutive games and set a league record.

2.

What is the decision between subjective and objective approaches in decision making?

Subjective decision making relies on experience and intuition; in this case, the team’s scouts applied their experience to select players. An objective approach identifies measures associated with success to create a predictive model for decision making based on data.

Application Case 1.5: Analyzing Athletic Injuries

1.

What types of analytics are applied in the injury analysis?

The analytics used data about the type of injury, action taken, healing start and end dates, players’ position, activity, onset, and game location. The data were used to classify healing time into five periods and to associate healing time with players’ positions, severity of injury, and treatment offered. That provided information for creating neural network models using player and injury data to predict healing time in terms of the five categories.

2.

How do visualizations aid in understanding the data and delivering insights into the data?

Visualizations provide a great tool for gaining the initial insights into data, which can be further refined based on expert opinions. Visualizations also aid in generating ideas for obscured business problems, which can be pursued in building predictive models.

3.

What is a classification problem?

Classification is a technique used in developing predictive analytical applications.

In this example, various kinds of data, such as severity of injury and healing time, were classified for purposes of making decisions about how to handle injuries and which players might be available to play in the future.

15 .

4.

What can be derived by performing sequence analysis?

Drawing sequence rules can predict the relationship among types of data—in this case, the relationship among the injuries and the various body parts afflicted with injuries.

Application Case 1.6: Industrial and Commercial Bank of China (ICBC) Employs Models to Reconfigure Its Branch Network

1.

How can analytical techniques help organizations to retain competitive advantage?

In this case, analytical techniques use geographic and customer behavioral segmentation to isolate the profitability of customers and employ optimization techniques to locate the branches that deliver high profitability in each geographic segment. Encourage students to apply this example to the basics of competition in business: Organizations that know their customers better can better target their goods and services. They can operate more efficiently. They can sell the most profitable products to the most profitable customers. In these ways, they can outperform the competition.

2.

How can descriptive and predictive analytics help in pursuing prescriptive analytics?

Descriptive analytics tell the organization what is happening, and predictive analytics tell what is likely to happen. When an organization has a good view of what is happening and what is likely to happen, it can also employ other techniques—prescriptive analytics—to make the best decisions under the circumstances. According to one view, these types of analytics are somewhat independent steps, with one type of analytics leading to another. In another view, the types of analytics overlap somewhat.

3.

What kinds of prescriptive analytic techniques are employed in the case study?

At ICBC, a branch reconfiguration system uses a market potential calculation model, a branch network optimization model, and a branch site evaluation model.

The branch network optimization model uses mixed integer programming to 16 .

locate branches where they cover the areas with the largest market potential.

Using the model’s recommendations has increased deposits in the branches by $21.2 billion.

4.

Are the prescriptive models once built good forever?

No, because conditions change, and new knowledge can improve a model. In the ICBC example, the branch reconfiguration tool has been improved through an iterative process.

Application Case 1.7: Gilt Groupe’s Flash Sales Streamlined by Big Data Analytics

1.

What makes this case study an example of Big Data analytics?

Big Data is data that cannot be stored in a single storage unit and typically is arriving in many different forms or in a stream. Although the case does not describe the volume of data that Gilt Groupe is analyzing, it does say the company is receiving a stream of data because it is measuring every click on the company’s website, as well as references to Gilt on Twitter. Also, the software the company uses is based on Teradata Aster’s technology solution that includes Big Data analytics technologies.

2.

What types of decisions does Gilt Groupe have to make?

Gilt Groupe must decide what to order—both the selection of brands and styles and the quantity of each item—to minimize inventory costs. It makes personalized recommendations, so it needs to decide what items to recommend to each customer. It needs to decide on the timing of e-mails that communicate the offers.

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION

17 .

  

1. Survey the literature from the past six months to find one application each of DSS, BI, and analytics. Summarize the applications on one page and submit it with the exact sources.

There is no single correct answer to this question. Answers will depend on when the question is posed and on the student’s online or library search strategy. The correctness of an answer can be evaluated by examining the sources cited in it or submitted with it.

2.

Distinguish BI from DSS. • • • • •

BI uses a data warehouse, whereas DSS can use any data source (including a data warehouse).

Most DSS are built to support decision making directly, whereas most BI systems are built to provide information which it is believed will lead to improved decision making.

BI has a strategy/executive orientation whereas DSS are usually oriented toward analysts.

BI systems tend to be developed with commercially available tools, whereas DSS tend to use more custom programming to deal with problems that may be unstructured.

DSS methodologies and tools originated largely in academia, whereas BI arose largely from the software industry. Many BI tools, such as data mining and predictive analysis, have come to be considered DSS tools as well.

3. Compare and contrast predictive analytics with predictive and descriptive analytics.

Use examples.

Predictive analytics is the use of statistical techniques and data mining to determine what is likely to happen in the future. For example, an airline might use predictive analytics to forecast the impact on sales and profits if it raises baggage fees by $10. It applies information from descriptive analytics, applying historical or real-time data to know what is happening in the organization and understand some underlying trends and causes of such occurrences. In the airline example, descriptive analytics would include data about ticket prices, baggage fees, ticket sales, baggage volume, and so on, applied to find relationships among these variables. Predictive analytics may be applied to prescriptive analytics, which is a set of techniques that use descriptive data and forecasts to identify the decisions most likely to result in the best performance.

For example, predictive analytics could forecast the impact on profits of different baggage fees. It might show, for example, that raising baggage fees by $5 will lead to the greatest profits after the airline takes into account fee revenues, ticket sales, the amount of baggage carried, and the cost to transport the baggage. (Students may use different examples, as long as they illustrate the definitions.)

4.

What is BI governance?

18 .

See Review question 2 of Section 1.4.

5.

Discuss the major issues in implementing BI. Major issues in implementing BI are: • •

Properly appreciating the different classes of potential users of the BI applications.

Properly aligning BI with the business strategy.

•

Developing BI applications that meet users’ needs for real-time, ondemand capabilities.

•

Determining whether to develop or acquire BI systems, and how to do so.

•

Justifying the BI investment using cost-benefit analysis.

•

Insuring the security and privacy protection

•

Integrating BI applications with organizational systems, databases, and ecommerce.

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS 1.

 

Why did Nationwide need an enterprise-wide data warehouse?

With more than 100 business units offering a variety of products, Nationwide experienced duplication of effort in gathering data, analyzing it, and generating reports. Data-processing environments were widely dissimilar, and there was extreme data redundancy, resulting in higher expenses. Mergers and acquisitions only added to the difficulty and cost. A single, authoritative data warehouse would apply best practices and provide clean, consistent, and complete data.

2.

How did integrated data drive the business value?

Integrated data about customers improved marketing campaigns and better targeted communications, which improved customer satisfaction and retention, as well as contributing to a gain in sales. Integrated financial data made financial reporting faster and more efficient and added tools for better risk assessment and decision support. Integrated data following mergers fostered smoother integration of businesses. Integrated data for reporting gave agents easy access to reports within seconds rather than days, significantly improving productivity.

3.

What forms of analytics are employed at Nationwide?

19 .

It uses the three basic types of business analytics. The company’s data warehouse supports descriptive analytics for all functions. For example, the data describes customer behavior, financial performance, and policy information. It uses predictive analytics in the Customer Knowledge Store to identify the kinds of customer interaction that are important for customers at different points in their lives. At the level of prescriptive analytics, the Financial Performance Management system applies financial data to a decision support system. (Students may identify additional examples of these three categories.) Optionally, students may also refer to analytics applied to different domains, such as marketing analytics, financial analytics, and even insurance analytics.

4.

With integrated data available in an enterprise data warehouse, what other applications could Nationwide potentially develop?

This would be a good question for class discussion, as answers will vary, and students can apply any work experience they have, including internships, as well as the cases in this chapter, to identify areas of business where analytics adds value. The case described customer relationships and financial reporting. Other areas of decision making for an insurance business would include the development and pricing of products, regulatory compliance, hiring, risk management, and the location of facilities. Accept any reasonable answers.

20 .

CHAPTER

2

Data Warehousing

Learning Objectives for Chapter 2 ▪

Understand the basic definitions and concepts of data warehouses

▪

Understand data warehousing architectures

▪

Describe the processes used in developing and managing data warehouses

▪

Explain data warehousing operations

▪

Explain the role of data warehouses in decision support

▪

Explain data integration and the extraction, transformation, and load (ETL) processes

▪

Describe real-time (active) data warehousing

▪

Understand data warehouse administration and security issues

CHAPTER OVERVIEW The concept of data warehousing has been around since the late 1980s. This chapter provides the foundation for an important type of database, called a data warehouse, which is primarily used for decision support and provides improved analytical capabilities. We discuss data warehousing in the following sections:

CHAPTER OUTLINE 2.1

OPENING VIGNETTE: ISLE OF CAPRI CASINOS IS WINNING WITH ENTERPRISE DATA WAREHOUSE 1 .

A.

 Questions for the Opening Vignette WHAT WE CAN LEARN FROM THIS VIGNETTE

2.2

DATA WAREHOUSING DEFINITIONS AND CONCEPTS A. WHAT IS A DATA WAREHOUSE? B. A HISTORICAL PERSPECTIVE TO DATA WAREHOUSING C. CHARACTERISTICS OF DATA WAREHOUSING D. DATA MARTS E. OPERATIONAL DATA STORES F. ENTERPRISE DATA WAREHOUSES (EDW) ⧫ Application Case 2.1: A Better Data Plan: Well-Established TELCOs Leverage Data Warehousing and Analytics to Stay on Top in a Competitive Industry G. METADATA  Section 2.2 Review Questions

2.3

DATA WAREHOUSING PROCESS OVERVIEW ⧫ Application Case 2.2: Data Warehousing Helps MultiCare Save More Lives  Section 2.3 Review Questions

2.4

DATA WAREHOUSING ARCHITECTURES A. ALTERNATIVE DATA WAREHOUSING ARCHITECTURES B. WHICH ARCHITECTURE IS THE BEST?  Section 2.4 Review Questions

2.5

DATA INTEGRATION AND THE EXTRACTION, TRANSFORMATION, AND LOAD (ETL) PROCESSES A. DATA INTEGRATION ⧫ Application Case 2.3: BP Lubricants Achieves BIGS Success B. EXTRACTION, TRANSFORMATION, AND LOAD  Section 2.5 Review Questions

2.6

DATA WAREHOUSE DEVELOPMENT ⧫ Application Case 2.4: Things Go Better with Coke’s Data Warehouse A. DATA WAREHOUSE DEVELOPMENT APPROACHES 1.

The Inmon Model: The EDW Approach 2.

The Kimball Model: The Data Mart Approach 3.

Which Model Is Best? ⧫ Application Case 2.5: Starwood Hotels & Resorts Manages Hotel Profitability with Data Warehousing B. ADDITIONAL DATA WAREHOUSE DEVELOPMENT CONSIDERATIONS ⧫ Technology Insights 2.1: Hosted Data Warehouses 2 .

C. D. E. F.

REPRESENTATION OF DATA IN DATA WAREHOUSE ANALYSIS OF DATA IN DATA WAREHOUSE OLAP VERSUS OLTP OLAP OPERATIONS 1.

Variations of OLAP ⧫ Technology Insights 2.2: Hands-On Data Warehousing with MicroStrategy  Section 2.6 Review Questions

2.7

DATA WAREHOUSING IMPLEMENTATION ISSUES ⧫ Application Case 2.6: EDW Helps Connect State Agencies in Michigan A. MASSIVE DATA WAREHOUSES AND SCALABILITY  Section 2.7 Review Questions

2.8

REAL-TIME DATA WAREHOUSING ⧫ Application Case 2.7: Egg Plc Fries the Competition in Near Real Time ⧫ Technology Insights 2.3: The Real-Time Realities of Active Data Warehousing  Section 2.8 Review Questions

2.9

DATA WAREHOUSE ADMINISTRATION, SECURITY ISSUES, AND FUTURE TRENDS ⧫ Technology Insights 2.4: Ambeo Delivers Proven DataAccess Auditing Solution A. THE FUTURE OF DATA WAREHOUSING  Section 2.9 Review Questions

2.10

RESOURCES, LINKS, AND THE TERADATA UNIVERSITY NETWORK CONNECTION A. RESOURCES AND LINKS B. CASES C. VENDORS, PRODUCTS, AND DEMOS D. PERIODICALS E. ADDITIONAL REFERENCES F. THE TERADATA UNIVERSITY NETWORK (TUN) CONNECTION Chapter Highlights Key Terms Questions for Discussion Exercises Teradata University and Other Hands-On Exercises Team Assignments and Role-Playing Projects Internet Exercises ⧫ End of Chapter Application Case: Continental Airlines Flies High with Its Real-Time Data Warehouse 3 .



Questions for the Case

References

TEACHING TIPS/ADDITIONAL INFORMATION

       

Expect to spend some time on this chapter, as understanding its content is crucial and many of the concepts are not intuitive to students whose major computer exposure has been at the personal level. You should prepare yourself with some technical examples of how things work, such as screen shots from some of the tools mentioned in the chapter, as most of the technical discussion here is at a more conceptual level. The business discussion is tangible, but the technical part is less so. It will be up to you to connect it to the students.

Section 2.2 defines several fundamental concepts. Students must understand that these are not definitions for the sake of definitions (every textbook has some of those, this one is no exception) but are important to anyone working in the field. The characteristics of a data warehouse in its extended definition, for example, are given not because some researcher thought they’d be theoretically nice, but, more specifically, because they’ve turned out to be important in practice. So, it’s important to really know what each of them is about. (The concept of an operational data store should already be familiar to students, though perhaps not by that name. They should know what metadata are from their use with general databases, too.)

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 2.1 Review Questions 1.

Why is it important for Isle to have an EDW? In the gaming industry, companies distinguish themselves based on their customer relationships. An enterprise data warehouse (EDW) gathers and provides the data needed to tell Isle of Capri what customers respond to, so the casinos can adapt their offerings. The information provided by the EDW lets Isle deepen its understanding of customers, so it can efficiently give them more of the kinds of entertainment they are looking for.

2.

What were the business challenges or opportunities that Isle was facing?

Isle of Capri Casinos is one of the largest publicly traded gaming companies in the United States, but the gaming business is competitive. Other casinos offer essentially the same games, so Isle must find ways to make its entertainment and hospitality atmosphere one that exceeds customer expectations. Before Isle implemented the EDW, casino managers had to wait to review monthly data until the second week of the following month. The time lag made it difficult for casinos 4 .

to identify what actions were appealing to customers in time to respond. Adding to these business challenges, Hurricane Katrina set back initial efforts to set up a data warehouse at the southeastern company.

3.

What was the process Isle followed to realize EDW? Comment on the potential challenges Isle might have had going through the process of EDW development.

Isle of Capri brought in two expert suppliers. Teradata provided the core solution; IBM Cognos provided expertise in business intelligence. Isle hired a management team that understood how EDW could support decision making at Isle. That team would be able to help Isle’s managers with the challenge of understanding how they can frame queries and follow-up questions to figure out ways to improve the business. Most likely, the potential of using detailed, up-to-the-minute data would be unfamiliar to many of these managers.

4.

What were the benefits of implementing an EDW at Isle? Can you think of other potential benefits that were not listed in the case?

The implementation of EDW brought several benefits related to the timeliness and detail of the data that became available. Instead of five week-old monthly reports, managers can now study a variety of daily, weekly, and monthly reports.

The reports segment data by particular properties and customer groups, so managers can zero in on particular problems and successes, easily making comparisons among properties. Managers can submit queries about data sets and receive information within minutes. In this way, managers can find out how particular promotions are affecting customer behavior at particular casinos. The EDW also connects data about casino activity with data about customers’ use of hotels and efforts by Isle’s hosts. This, too, helps the company target promotions and offer customers incentives they value. Even decisions as detailed as where to locate slot machines can be adjusted to boost profits based on data from the EDW.

5.

Why do you think large enterprises like Isle in the gaming industry can succeed without having a capable data warehouse/business intelligence infrastructure?

In the past, businesses in the gaming industry could succeed without a capable data warehouse/business intelligence infrastructure because their managers knew as much about customers as their competitors’ management knew. They were all testing ideas for promotion or entertainment and responding to customer behavior at about the same pace. However, when one company such as Isle begins to respond to daily, property-by-property data, choosing the marketing and entertainment options that deliver exactly what most profitably lures customers, competitors without that capability will soon begin to suffer.

Section 2.2 Review Questions 1.

What is a data warehouse?

5 .

A data warehouse is defined in this section as “a pool of data produced to support decision making.” This focuses on the essentials, leaving out characteristics that may vary from one DW to another but are not essential to the basic concept.

The same paragraph gives another definition: “a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management’s decisionmaking process.” This definition adds more specifics, but in every case appropriately: it is hard, if not impossible, to conceive of a data warehouse that would not be subject-oriented, integrated, etc.

2.

How does a data warehouse differ from a database?

Technically a data warehouse is a database, albeit with certain characteristics to facilitate its role in decision support. Specifically, however, it is (see previous question) an “integrated, time-variant, nonvolatile, subject-oriented repository of detail and summary data used for decision support and business analytics within an organization.” These characteristics, which are discussed further in the section just after the definition, are not necessarily true of databases in general—though each could apply individually to a given one.

As a practical matter most databases are highly normalized, in part to avoid update anomalies. Data warehouses are highly denormalized for performance reasons. This is acceptable because their content is never updated, just added to.

Historical data are static.

3.

What is an ODS? Operational Data Store is the database from which a business operates on an ongoing basis.

4.

Differentiate among a data mart, an ODS, and an EDW. An ODS (Operational Data Store) is the database from which a business operates on an ongoing basis.

Both an EDW and a data mart are data warehouses. An EDW (Enterprise Data Warehouse) is an all-encompassing DW that covers all subject areas of interest to the entire organization. A data mart is a smaller DW designed around one problem, organizational function, topic, or other suitable focus area.

5.

Explain the importance of metadata.

Metadata, “data about data,” are the means through which applications and users access the content of a data warehouse, through which its security is managed, and through which organizational management manages, in the true sense of the 6 .

word, its information assets. Most database management systems would be unable to function without at least some metadata. Indeed, the use of metadata, which enable data access through names and logical relationships rather than physical locations, is fundamental to the very concept of a DBMS. Metadata are essential to any database, not just a data warehouse. (See answer to Review Question 2 of this section above.)

Section 2.3 Review Questions 1.

Describe the data warehousing process.

The data warehousing process consists of the following steps: 1.

Data are imported from various internal and external sources 2.

Data are cleansed and organized consistently with the organization’s needs 3. a.

Data are loaded into the enterprise data warehouse, or b.

Data are loaded into data marts.

4. a.

If desired, data marts are created as subsets of the EDW, or b.

The data marts are consolidated into the EDW 5.

Analyses are performed as needed

2.

Describe the major components of a data warehouse.

• • • • • •

3.

Data sources. Data are sourced from operational systems and possibly from external data sources.

Data extraction and transformation. Data are extracted and properly transformed using custom-written or commercial software called ETL. Data loading. Data are loaded into a staging area, where they are transformed and cleansed. The data are then ready to load into the data warehouse.

Comprehensive database. This is the EDW that supports decision analysis by providing relevant summarized and detailed information.

Metadata. Metadata are maintained for access by IT personnel and users.

Metadata include rules for organizing data summaries that are easy to index and search.

Middleware tools. Middleware tools enable access to the data warehouse from a variety of front-end applications.

Identify and discuss the role of middleware tools.

Middleware tools enable access to the data warehouse. Power users such as analysts may write their own SQL queries. Others may access data through a managed query environment. There are many front-end applications that business users can use to interact with data stored in the data repositories, including data 7 .

mining, OLAP, reporting tools, and data visualization tools. All these have their own data access requirements. Those may not match with how a given data warehouse must be accessed. Middleware translates between the two.

Section 2.4 Review Questions 1.

What are the key similarities and differences between a two-tiered architecture and a three-tiered architecture?

Both provide the same user visibility through a client system that accesses a DSS/BI application remotely. The difference is behind the scenes and is invisible to the user: in a two-tiered architecture, the application and data warehouse reside on the same machine; in a three-tiered architecture, they are on separate machines.

2.

How has the Web influenced data warehouse design?

Primarily by making Web-based data warehousing possible.

3.

List the alternative data warehousing architectures discussed in this section.

• • • • •

4.

Independent data marts architecture Data mart bus architecture with linked dimensional data marts Hub-and-spoke architecture (corporate information factory) Centralized data warehouse architecture Federated architecture

What issues should be considered when deciding which architecture to use in developing a data warehouse? List the 10 most important factors.

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

Information interdependence between organizational units Upper management’s information needs Urgency of need for a data warehouse Nature of end-user tasks Constraints on resources Strategic view of the data warehouse prior to implementation Compatibility with existing systems Perceived ability of the in-house IT staff Technical issues Social/political factors

(This list from the text, while clearly intended by the authors as the answer to this review question, does not explicitly say that these are the ten most important factors. Students may choose others.)

5.

Which data warehousing architecture is the best? Why? 8 .

See Table 2.1 Average Assessment Scores for the Success of the Architectures.

What is interesting is the similarity of the averages for the bus, hub-and-spoke, and centralized architectures. The differences are sufficiently small that no claims can be made for a particular architecture’s superiority over the others, at least based on a simple comparison of these success measures.

Section 2.5 Review Questions 1.

Describe data integration.

Data integration is an umbrella term that covers three processes that combine to move data from multiple sources into a data warehouse: accessing the data, combining different views of the data, and capturing changes to the data.

2.

Describe the three steps of the ETL process.

Extraction: selecting data from one or more sources and reading the selected data.

Transformation: converting data from their original form to whatever form the DW needs. This step often also includes cleansing of the data to remove as many errors as possible.

Load: putting the converted (transformed) data into the DW.

3.

Why is the ETL process so important for data warehousing efforts?

Since ETL is the process through which data are loaded into a data warehouse, a DW could not exist without it. The ETL process also contributes to the quality of the data in a DW.

Section 2.6 Review Questions 1.

List the benefits of data warehouses.

Direct benefits include: • Allowing end users to perform extensive analysis in numerous ways.

• A consolidated view of corporate data (i.e., a single version of the truth).

• Better and more timely information. A data warehouse permits information processing to be offloaded from costly operational systems onto low-cost servers; therefore, end-user information requests can be processed more quickly.

9 .

• •

Enhanced system performance. A data warehouse frees production processing because some operational system reporting requirements are moved to DSS. Simplification of data access.

Indirect benefits arise when end users take advantage of these direct benefits.

2.

List several criteria for selecting a data warehouse vendor, and describe why they are important.

Six important criteria are: financial strength, ERP linkages, qualified consultants, market share, industry experience, and established partnerships. These are important to indicate that a vendor is likely to be in business for the long term, to have the support capabilities its customers need, and to provide products that interoperate with other products the potential user has or may obtain.

One could add others, such as product functionality (Does it do what we need?), vendor strategic vision (Does their direction make sense for our future plans and/or is it consistent with industry trends?) and quality of customer references (What do their existing customers think of them?).

3.

What is OLAP and how does it differ from OLTP? Data stored in a data warehouse can be analyzed using techniques referred to as OLAP. OLAP is one of the most commonly used data analysis techniques in data warehouses. OLAP is an approach to quickly answer ad hoc questions that require data analysis.

OLTP is concerned with the capture and storage of data. OLAP is concerned with the analysis of that data.

4.

What is a cube? What do drill down, roll up, and slice and dice mean?

The main operational structure in OLAP is based on a concept called cube. A cube in OLAP is a multidimensional data structure (actual or virtual) that allows fast analysis of data.

Using OLAP, an analyst can navigate through the database and screen for a particular subset of the data (and its progression over time) by changing the data’s orientations and defining analytical calculations. These types of user-initiated navigation of data through the specification of slices (via rotations) and drill down/up (via aggregation and disaggregation) are sometimes called “slice and dice.” Commonly used OLAP operations include slice and dice, drill down, roll up, and pivot.

10 .

5.

•

Slice: A slice is a subset of a multidimensional array (usually a twodimensional representation) corresponding to a single value set for one (or more) of the dimensions not in the subset.

•

Dice: The dice operation is a slice on more than two dimensions of a data cube.

•

Drill Down/Up: Drilling down or up is a specific OLAP technique whereby the user navigates among levels of data ranging from the most summarized (up) to the most detailed (down).

What are ROLAP, MOLAP, and HOLAP? How do they differ from OLAP? ROLAP stands for Relational Online Analytical Processing. ROLAP is an alternative to the MOLAP (Multidimensional OLAP) technology. While both ROLAP and MOLAP analytic tools are designed to allow analysis of data through the use of a multidimensional data model, ROLAP differs in that it does not require the pre-computation and storage of information. Instead, ROLAP tools access the data in a relational database and generate SQL queries to calculate information at the appropriate level when an end user requests it. MOLAP is an alternative to the ROLAP technology. MOLAP differs from ROLAP significantly in that it requires the pre-computation and storage of information in the cube— the operation known as preprocessing. MOLAP stores this data in an optimized multidimensional array storage, rather than in a relational database.

HOLAP (Hybrid Online Analytical Processing) is a combination of ROLAP and MOLAP. HOLAP allows storing part of the data in a MOLAP store and another part of the data in a ROLAP store. The degree of control that the cube designer has over this partitioning varies from product to product.

All of these are variations of OLAP.

Section 2.7 Review Questions 1.

What are the major DW implementation tasks that can be performed in parallel?

Reeves (2009) and Solomon (2005) provided some guidelines regarding the critical questions that must be asked, some risks that should be weighted, and some processes that can be followed to help ensure a successful data warehouse implementation. They compiled a list of 11 major tasks that could be performed in parallel: Establishment of service-level agreements and data-refresh requirements

11 .

Identification of data sources and their governance policies Data quality planning Data model design ETL tool selection Relational database software and platform selection Data transport Data conversion Reconciliation process Purge and archive planning End-user support 2.

List and discuss the most pronounced DW implementation guidelines.

•

• •

•

3.

Senior management must support development of the data warehouse. The DW needs a project champion at a high position in the organization chart.

Benefits of a DW project may be difficult to measure, so management support makes it more likely the project will receive funding.

Web-based data warehouses may need special security requirements.

These ensure that only authorized users have access to the data.

Users should participate in the development process. Their participation is essential for data modeling and access modeling. User participation ensures that the DW includes the needed data and that decision makers can retrieve the data they need.

DW implementation requires certain skills from members of the development team: in-depth knowledge of database technology and the development tools used.

When developing a successful data warehouse, what are the most important risks and issues to consider and potentially avoid?

•

Starting with the wrong sponsorship chain

•

Setting expectations that you cannot meet

•

Engaging in politically naive behavior

•

Loading the data warehouse with information just because it is available

12 .

4.

•

Believing that data warehousing database design is the same as transactional database design

•

Choosing a data warehouse manager who is technology oriented rather than user oriented

•

Focusing on traditional internal record-oriented data and ignoring the value of external data and of text, images, and, perhaps, sound and video

•

Delivering data with overlapping and confusing definitions

•

Believing promises of performance, capacity, and scalability

•

Believing that your problems are over when the data warehouse is up and running

•

Focusing on ad hoc data mining and periodic reporting instead of alerts

What is scalability? How does it apply to DW? Scalability refers to the degree to which a system can adjust to changes in demand without major additional changes or investments. DW scalability issues are the amount of data in the warehouse, how quickly the warehouse is expected to grow, the number of concurrent users, and the complexity of user queries. A data warehouse must scale both horizontally and vertically. The warehouse will grow as a function of data growth and the need to expand the warehouse to support new business functionality. Data growth may be a result of the addition of current cycle data (e.g., this month’s results) and/or historical data.

Section 2.8 Review Questions 1.

What is an RDW? A real-time data warehouse, in which decision-making data are updated on an ongoing basis as business transactions occur; same as an active data warehouse (ADW).

2.

List the benefits of an RDW. The RDW extends the benefits of data warehousing, in general, down into tactical, and perhaps operational, decision making. It empowers people who interact directly with customers and suppliers by providing them with information to make decisions. It can then be extended to customers and suppliers themselves, thus affecting almost all aspects of customer service, SCM, logistics, and beyond.

13 .

It can also facilitate e-business activities, as when sales outlets such as overstock.com (cited in the text) use historical data to price new close-outs.

3.

What are the major differences between a traditional data warehouse and an RDW? 1.

2.

3.

4.

5.

6.

7.

4.

A traditional data warehouse (TDW) is used for strategic decisions (and sometimes tactical); an RDW for strategic and tactical (sometimes operational) ones.

The results of using a TDW can be hard to measure; results of using an RDW are measured by operational data.

Acceptable TDW refresh rates range from daily to monthly; RDW data must be up to the minute.

TDW summaries are often appropriate; RDWs must supply detailed data.

Small user community at upper organizational levels means a TDW supports few concurrent users; an RDW must support many, perhaps over a thousand.

TDWs typically use restrictive reporting to confirm or check patterns, often predefined summary tables; RDWs need flexible, ad hoc reporting.

TDW user community generally consists of power users, knowledge workers, managers, other internal users; RDWs are used by operational staff, call centers, perhaps external users.

List some of the drivers for RDW. • • •

•

A business often cannot afford to wait a whole day for its operational data to load into the data warehouse for analysis.

Traditional data warehouses have captured snapshots of an organization’s fixed states instead of incremental real-time data showing every state change and almost analogous patterns over time.

With a traditional hub-and-spoke architecture, retaining the metadata in sync is difficult. It is also costly to develop, maintain, and secure many systems as opposed to one huge data warehouse so that data are centralized for BI/BA tools.

In cases of huge nightly batch loads, the necessary ETL setup and processing power for large nightly data warehouse loading might be very high, and the processes might take too long. An EAI with real-time data collection can reduce or eliminate the nightly batch processes.

Section 2.9 Review Questions 1.

What steps can an organization take to ensure the security and confidentiality of customer data in its data warehouse?

Effective security in a data warehouse should focus on four main areas: 14 .

Step 1. Establishing effective corporate and security policies and procedures. An effective security policy should start at the top and be communicated to everyone in the organization.

Step 2. Implementing logical security procedures and techniques to restrict access.

This includes user authentication, access controls, and encryption.

Step 3. Limiting physical access to the data center environment.

Step 4. Establishing an effective internal control review process for security and privacy.

2.

What skills should a DWA possess? Why? • • • •

3.

Familiarity with high-performance hardware, software, and networking technologies, since the data warehouse is based on those Solid business insight, to understand the purpose of the DW and its business justification Familiarity with business decision-making processes to understand how the DW will be used Excellent communication skills, to communicate with the rest of the organization

What recent technologies may shape the future of data warehousing? Why? Following are some of the recently popularized concepts and technologies that will play a significant role in defining the future of data warehousing.

Sourcing: Acquisition of data from diverse and dispersed sources •

Web, social media, and Big Data

•

Open source software

•

SaaS (software as a service) “The Extended ASP Model”

•

Cloud computing

Infrastructure: Architectural—hardware and software—enhancements •

Columnar (a new way to score and access data in the database)

•

Real-time data warehousing

•

Data warehouse appliances (all-in-one solutions to DW)

•

Data management technologies and practices

15 .

•

In-database processing technology (putting the algorithms where the data is)

•

In-memory storage technology (moving the data in the memory for faster processing)

•

New database management systems

•

Advanced analytics

As the world of business becomes more global and complex, the need for business intelligence and data warehousing tools also becomes more prominent. The fast improving information technology tools and techniques seem to be moving in the right direction to address the needs of the future business intelligence systems.

Section 2.10 Review Questions (This section has no review questions.)

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 2.1: A Better Data Plan: Well-Established TELCOs Leverage Data Warehousing and Analytics to Stay on Top in a Competitive Industry 1.

What are the main challenges for TELCOs? To stay competitive, TELCOs must continuously refine everything from customer service to plan pricing. The major challenges faced by both entrenched and new companies in this industry include: retaining customers, decreasing costs, finetuning pricing models, improving customer satisfaction, acquiring new customers, and understanding the role of social media in customer loyalty.

2.

How can data warehousing and data analytics help TELCOs in overcoming their challenges?

Highly targeted data analytics play an ever more critical role in helping carriers secure or improve their standing in an increasingly competitive marketplace.

Argentina’s Telefónica de Argentina used analytics for its “traceability project,” which tracked the factors involved in customer churn, a big problem among phone service carriers. France’s Bouygues Telecom used BI technologies to facilitate cost reduction through automation via its Teradata-based marketing operations management system, which automates marketing/communications collateral production. Pakistan’s Mobilink uses BI to help acquire customers and grow their subscriber network, largely aided by social networking.

16 .

3.

Why do you think TELCOs are well suited to take full advantage of data analytics?

TELCOs control the telecommunications infrastructure, and acquire much usage data as a result. They have the technical expertise to create, deploy, and refine plans to address their business challenges. The industry and mobile technology have expanded and improved over the years, which provides a strong foundation on which to build intelligent solutions. The data analytics solutions that have been created to meet these challenges have also improved drastically over the past few years, placing TELCOs in a good position to capitalize on their technological advantages.

Application Case 2.2: Data Warehousing Helps Multi-Care Save More Lives 1.

What do you think is the role of data warehousing in healthcare systems?

This question seeks opinions, so accept any reasonable answer. The case focuses on the role of data warehousing in improving patient care, which can literally save lives. Data on patient needs, hospital practices, and patient outcomes can help health care systems identify the practices that lead to the best patient outcomes.

Of course, organizations also can use data warehousing to improve efficiency and profits.

2.

How did MultiCare use data warehousing to improve health outcomes?

MultiCare organized and simplified data from multiple sources across the continuum of care, so the Adaptive Data Warehouse became a single source that unified decision makers across functions. It addressed the serious problem of septicemia by defining a standard of care for septic patients, identify patients with conditions signaling they were likely to take a sudden turn for the worse, and efficiently implement the standard of care. Building on the first-year success of the effort, MultiCare intended to expand the application of data warehousing in similar ways to treat heart failure, manage the performance of the emergency department, and improve inpatient throughput.

Application Case 2.3: BP Lubricants Achieve BIGS Success 1.

What is BIGS? BIGS is the Business Intelligence and Global Standards program, a strategic initiative for management information and business intelligence. Its purpose is to deliver globally consistent and transparent management information across functions.

17 .

2.

What were the challenges, the proposed solution, and the obtained results with BIGS? The challenge facing BP Lubricants was that it had been involved in merger activity, which meant it had data held in disparate source systems. To be more agile and prepare for growth, it wanted a unified system. The proposed solution was Kalido, an adaptive enterprise data warehousing solution. The system integrates and stores information from multiple source systems to provide consolidated views for marketing, sales, and finance. The obtained results are that BIGS helps the business identify a multitude of business opportunities to improve profits and lower costs. Typical benefits include improved consistency and transparency of business data, faster and more flexible reporting, and greater ability to respond intelligently to new business opportunities.

Application Case 2.4: Things Go Better with Coke’s Data Warehouse 1.

How did Coca-Cola in Japan use data warehousing to improve its business processes?

The Hokuriku Coca-Cola Bottling Co. (HCCBC) hired Teradata Corp. to implement a data warehouse and analytical software to collect historical data and near-real-time data wirelessly transmitted from each vending machine. The data tells the company what is selling, and it sends alerts when a product sells out, a customer is short-changed, or a machine is malfunctioning.

2.

What were the results of their enterprise active data warehouse implementation?

In a pilot test of the system, the accurate demand forecasts and rapid problem identification led to an immediate 10 percent increase in sales. More accurate machine servicing contributed to a 42 percent decline in costs while improving the efficiency of the salespeople who service the machines. Based on the results, HCCBC intended to expand the use of the active data warehouse to cover all the vending machines in Japan.

Application Case 2.5: Starwood Hotels & Resorts Manages Hotel Profitability with Data Warehousing 1.

How big and complex are the business operations of Starwood Hotels & Resorts?

The business operations are very large and complex. Starwood is one of the world’s leading hotel and leisure companies, with 1,112 properties in nearly 100 countries. The company has 154,000 employees at properties that it owns and manages. It also operates a customer loyalty program.

18 .

2.

How did Starwood Hotels & Resorts use data warehousing for better profitability?

With data warehousing, Starwood Hotels can prepare operational reports in just four to six hours. With real-time feeds, transactions can be posted immediately to the data warehouse, and users can access the changes in just 5 to 10 minutes, or 288 times faster than before. The accelerated access to data improves profits because hotel managers can make sure rooms are available for premier customers and improve the ability to manage room occupancy rates. Marketing campaigns also have become more efficient as managers analyze results in days or weeks instead of months.

3.

What were the challenge, the proposed solution, and the obtained results?

The challenge was that the company had grown significantly, especially in the Asia/Pacific region. This greatly increased the need for business critical information about the hotels and their customers. Obtaining reports from the central reservation system could take up to 18 hours, and with delays in obtaining data, managers had difficulty adjusting rates to increase revenues. The proposed solution was an Oracle Exadata system that provides near-real-time information.

As a result of Starwood implementing the system, managers have accelerated access to data so they can do same-day or next-day analyses, as well as additional reporting tools. The company now operates more efficiently and profitably, and customer service has improved.

Application Case 2.6: EDW Helps Connect State Agencies in Michigan 1.

Why would a state invest in a large and expensive IT infrastructure (such as an EDW)?

Like a business, a state government can operate more efficiently and make better decisions if it has access to current data. State officials can use the EDW to improve efficiency and better serve the state’s residents.

2.

What are the size and complexity of EDW used by state agencies in Michigan?

Michigan’s EDW has almost 10,000 users in five major departments, 20 agencies, and more than 100 bureaus. Just in the Department of Human Services, the EDW contributes to nearly every function, including accurate delivery of and accounting for benefits to almost 2.5 million clients receiving public assistance.

3.

What were the challenges, the proposed solution, and the obtained results of the EDW? The challenge is not directly stated, but students should be aware that state governments struggle to balance budgets and satisfy a wide variety of needs. They also should recognize that a state operates many kinds of departments, so it likely 19 .

could benefit from a unified source of information. The proposed solution was an enterprise (statewide) electronic data warehouse linking employees across departments. The obtained results include financial benefits worth $1 million per business day. Savings come from operational efficiencies, avoidance of sanctions, improved client outcomes, integrity benefits awarded to well-performing programs, and recovery of inappropriate benefits payments. The data warehouse has yielded a 15:1 cost-effectiveness ratio and improvements in the accurate delivery and accounting of benefits.

Application Case 2.7: Egg Plc Fries the Competition in Near Real Time 1.

What kind of business is Egg plc in? What is the competitive landscape?

Egg plc is an online bank that provides banking, insurance, investments, and mortgages to more than 3.6 million customers through its Internet site. Although the case does not discuss the competition, it mentions that Egg was once the world’s largest online bank. Students may note that the competitive landscape for online banking has changed considerably over the past few years, with most banks now offering at least some services online. Also accept any other reasonable description of the competitive landscape.

2.

How did Egg plc use near-real-time data warehousing for competitive advantage?

Near-real-time data access gave employees and customers access to data. Within the company, the data helped with the construction of sales and marketing campaigns. It allowed faster decision making about specific customers and customer classes. An interesting basis for classroom discussion might be the fact that Egg did not maintain a competitive advantage for long, as it has gone out of business. Setting up a data warehouse can give the organization an initial advantage over companies slower to adopt the technology, but how might a company maintain that advantage, using analytics?

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION 1.

  

Compare data integration and ETL. How are they related?

Data integration consists of three processes that integrate data from multiple sources into a data warehouse: accessing the data, combining different views of the data, and capturing changes to the data. It makes data available to ETL tools and, through the three processes of ETL, to the analysis tools of the data warehousing environment.

20 .

2.

What is a data warehouse and what are its benefits? Why is Web accessibility important with a data warehouse?

A data warehouse can be defined as “a pool of data produced to support decision making.” This focuses on the essentials, leaving out characteristics that may vary from one DW to another but are not essential to the basic concept.

The same paragraph gives another definition: “a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management’s decisionmaking process.” This definition adds more specifics, but in every case appropriately: it is hard, if not impossible, to conceive of a data warehouse that would not be subject-oriented, integrated, etc.

The benefits of a data warehouse are that it provides decision-making information, organized in a way that facilitates the types of access required for that purpose and supported by a wide range of software designed to work with it.

Web accessibility of a data warehouse is important because many analysis applications are Web-based, because users often access data over the Web (or over an intranet using the same tools) and because data from the Web may feed the DW.

3.

A data mart can replace a data warehouse or complement it. Compare and discuss these options.

For a data mart to replace a data warehouse, it must make the DW unnecessary.

This would mean that all the analyses for which the DW would be used can instead be satisfied by a DM (or perhaps a combination of several DMs). If this is so, it can be much less expensive, in terms of development and computer resources, to use multiple DMs (let alone one DM!) instead of an overall DW. In other situations, a data mart can be used for some analyses which would in its absence use the DW, but not all of them. For those, the smaller DM is more efficient—quite possibly, enough more efficient as to justify the cost of having a DM in addition to a DW. Here the DM complements the DW.

4.

Discuss the major drivers and benefits of data warehousing to end users.

Major drivers include: • Increased competition and pace of business, leading to increased need for good decisions quickly • Successful pioneering experiences with data warehouses, leading to their wider user acceptance

21 .

• • • •

5.

Decreasing hardware costs, making terabyte databases with masses of historical data economically feasible for more firms Increased availability of software to manage a large data warehouse Increased availability of analysis tools making DWs potentially more useful Increased computer literacy of decision makers, making them more likely to use these tools

List the differences and/or similarities between the roles of a database administrator and a data warehouse administrator.

Since a data warehouse is a specific type of database designed for a specific application area, a data warehouse administrator has all the roles of a database administrator—plus others. One new role is advising on decision support uses of the DW, for which a DWA needs to understand decision-making processes.

Beyond that, the issue is more a need for additional skills in the same roles as a DBA—e.g., understanding high-performance hardware to deal with the large size of a DW—than it is one of additional roles. (See Review Question 2, Section 5.8, for list of skills.)

6.

Describe how data integration can lead to higher levels of data quality.

A question involving the word “higher” (or any other comparative, for that matter) requires asking “higher than what?” In this case, we can take it to mean “higher than we would have for the same data, but without a formal data integration process.”

Without a data integration process to combine data in a planned and structured manner, data might be combined incorrectly. That could lead to misunderstood data (a measurement in meters taken as being in feet) and to inconsistent data (data from one source applying to calendar months, data from another to fourweek or five-week fiscal months). These are aspects of low-quality data which can be avoided, or at least reduced, by data integration.

7.

Compare the Kimball and Inmon approaches toward data warehouse development. Identify when each one is most effective.

Inmon’s approach starts with an enterprise data warehouse, creating data marts as subsets if appropriate. It is most effective when there is a recognized need for an EDW, an executive “champion” of the project, and a willingness to invest in a data warehousing infrastructure before it will show results.

Kimball’s approach starts with data marts, consolidating them into an EDW later if appropriate. It is most effective when it is desired to provide a “proof of

22 .

concept” implementation before embarking on a full-scale EDW project or when a well-defined area with the greatest benefits can be identified.

8.

Discuss security concerns involved in building a data warehouse.

Security and privacy concerns are important in building a data warehouse: 1.

Laws and regulations, in the U.S. and elsewhere, require certain safeguards on databases that contain the type of information typically found in a DW. 2.

The large amount of valuable corporate data in a data warehouse can make it an attractive target.

3.

The need to allow a wide variety of unplanned queries in a DW makes it impractical to restrict end user access to specific carefully constrained screens, one way to limit potential violations.

9.

Investigate current data warehouse development implementation through offshoring. Write a report about it. In class, debate the issue in terms of the benefits and costs, as well as social factors.

Open-ended answer to the report; it is impossible to predict what the debate will bring.

A student’s position on this issue is related to his/her feelings on the relationship of national economies to the global economy. It can be argued that offshoring improves the global economy while potentially harming one or more of the national economies involved—such as the student’s own. U.S. students may see primarily the damage they perceive it does to their national economy (and to their own career prospects), but students in India may take a different view. The economic, political, and philosophical issues can be pursued well beyond what is practical in a DSS course.

If you feel students are too nationalistic on this issue, you can ask them if they feel the same way about a Massachusetts or California bank processing checks in Alabama to reduce labor costs. (This example uses U.S. territories, but similar issues exist in any country large enough to have regional economic differences.)

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS 1.

 

Describe the benefits of implementing the Continental Go Forward strategy.

This strategy consisted of a number of interrelated, concurrent actions. The first version of the overall strategy had the benefit of restoring Continental (CO) to profitability and giving it first-place rankings by many airline industry metrics.

The second phase of the strategy led to savings of $41 million and a reduction of 23 .

$7 million in fraud in the first year alone. Its revenue increased by over $500 million in six years. A data warehouse played a critical role in the second phase.

2.

Explain why it is important for an airline to use a real-time data warehouse.

It’s important for an airline to use a real-time data warehouse because many airline decisions cannot be made with week-old, or even day-old, data.

An example is frequent flyer award availability on a given flight. Airlines limit these so as not to give away too many seats that would otherwise be sold. Award seat allocation is usually automated. Travelers can check availability online.

American Airlines (and probably others) offers expanded award availability to the 30,000 or so people who fly at least 100,000 miles per year with them. When one of these wants an award seat that is not available online, he or she can request it by phone. The agent must decide whether or not to make it available. If the flight is selling slowly, the traveler gets the seat, even if the computer hasn’t allocated it for award. If it is likely to sell out, the seat isn’t offered, even to this select group.

To make this decision, telephone agents (and the Yield Management staff, which agents can consult) need up-to-the-minute, or at least up-to-the-hour, information.

3.

Examine the sample system output screen at http://www.teradata.com/casestudies/Continental-Airlines-Case-Study-eb4349. Describe how it can assist the user in identifying problems and opportunities.

This screen appears to show the relationship among three independent variables and one dependent variable. The independent age are the age of a caller (column), the time the person tends to call (color of rectangle) and the number of complaints by that category of caller (height of rectangle). The dependent variable is profitability, shown by the width of the rectangle. The chart shows that weekend callers, while generating neither the highest nor the lowest number of complaints, are the airline’s most profitable customers. This information can be used to find out why some other groups have more complaints, to give higher priority to weekend callers, and for many other purposes.

4.

Identify the major differences between the traditional data warehouse and a realtime data warehouse, as was implemented at Continental.

A traditional data warehouse moves data from operational databases to the DW on a scheduled basis, typically daily or weekly. This provides consistent data for analyses performed during one update cycle, but does not make current information available for decisions that require it. A real-time DW, as was implemented at CO, moves data into the DW on an hourly or even more frequent basis.

5.

What strategic advantage can Continental derive from the real-time system as opposed to a traditional information system?

24 .

By having real-time data available through its data warehouse, CO can make decisions using up-to-date information. While data warehousing applications, which focus on long-term decisions, aren’t affected much by the last hour’s, day’s, or even week’s data, lower-level short-term decisions are. As the use of the DW is extended to these decisions and down in the organization, current data become necessary. By having real-time (or near-real-time) data in the system, CO obtains a strategic advantage by making better decisions.

25 .

CHAPTER

3

Business Reporting, Visual Analytics, and Business Performance Management

Learning Objectives for Chapter 3 ▪

Define business reporting and understand its historical evolution

▪

Recognize the need for and the power of business reporting

▪

Understand the importance of data/information visualization

▪

Learn different types of visualization techniques

▪

Appreciate the value that visual analytics brings to BI/BA

▪

Know the capabilities and limitations of dashboards

▪

Understand the nature of business performance management (BPM)

▪

Learn the closed-loop BPM methodology

▪

Describe the basic elements of balanced scorecards

CHAPTER OVERVIEW

              

A report is a communication artifact prepared with the specific intention of relaying information in a presentable form. If it concerns business matters, then it is called a business report. Business reporting is an essential part of the business intelligence movement toward improving managerial decision making. Nowadays, these reports are more visually oriented, often using colors and graphical icons that collectively look like a 1 .

dashboard to enhance the information content. Business reporting and business performance management (BPM) are both enablers of business intelligence and analytics.

As a decision support tool, BPM is more than just a reporting technology. It is an integrated set of processes, methodologies, metrics, and applications designed to drive the overall financial and operational performance of an enterprise. It helps enterprises translate their strategies and objectives into plans, monitor performance against those plans, analyze variations between actual results and planned results, and adjust their objectives and actions in response to this analysis.

This chapter starts with examining the need for and the power of business reporting. With the emergence of analytics, business reporting evolved into dashboards and visual analytics, which, compared to traditional descriptive reporting, are much more predictive and prescriptive. Coverage of dashboards and visual analytics is followed by a comprehensive introduction to BPM. As you will see and appreciate, BPM and visual analytics have a symbiotic relationship (over scorecards and dashboards) where they benefit from each other’s strengths.

CHAPTER OUTLINE 3.1

OPENING VIGNETTE: SELF-SERVICE REPORTING ENVIRONMENT SAVES MILLIONS FOR CORPORATE CUSTOMERS  Questions for the Opening Vignette A. WHAT WE CAN LEARN FROM THIS VIGNETTE

3.2

BUSINESS REPORTING DEFINITIONS AND CONCEPTS A. WHAT IS A BUSINESS REPORT? ⧫ Application Case 3.1: Delta Lloyd Group Assures Accuracy and Efficiency in Financial Reporting 1.

Metric Measurement Reports 2.

Dashboard-Type Reports 3.

Balanced Scorecard-Type Reports B. COMPONENTS OF BUSINESS REPORTING SYSTEMS ⧫ Application Case 3.2: Flood of Paper Ends at FEMA  Section 3.2 Review Questions

3.3

DATA AND INFORMATION VISUALIZATION ⧫ Application Case 3.3: Tableau Saves Blastrac Thousands of Dollars with Simplified Information Sharing A. A BRIEF HISTORY OF DATA VISUALIZATION ⧫ Application Case 3.4: TIBCO Spotfire Provides DanaFarber Cancer Institute with Unprecedented Insight into Cancer Vaccine Clinical Trials  Section 3.3 Review Questions

3.4

DIFFERENT TYPES OF CHARTS AND GRAPHS A. BASIC CHARTS AND GRAPHS 2 .

B.

1.

Line Chart 2.

Bar Chart 3.

Pie Chart 4.

Scatter Plot 5.

Bubble Chart SPECIALIZED CHARTS AND GRAPHS 1.

Histogram 2.

Gantt Chart 3.

Pert Chart 4.

Geographic Map 5.

Bullet 6.

Heat Map 7.

Highlight Table 8.

Tree Map  Section 3.4 Review Questions

3.5

THE EMERGENCE OF DATA VISUALIZATION AND VISUAL ANALYTICS ⧫ Technology Insights 3.1: Gartner Magic Quadrant for Business Intelligence and Analytics Platforms A. VISUAL ANALYTICS ⧫ Technology Insights 3.2: Telling Great Stories with Data and Visualization B. HIGH-POWERED VISUAL ANALYTICS ENVIRONMENTS  Section 3.5 Review Questions

3.6

PERFORMANCE DASHBOARDS ⧫ Application Case 3.5: Dallas Cowboys Score Big with Tableau and Teknion A. DASHBOARD DESIGN ⧫ Application Case 3.6: Saudi Telecom Company Excels with Information Visualization B. WHAT TO LOOK FOR IN A DASHBOARD C. BEST PRACTICES IN DASHBOARD DESIGN D. BENCHMARK KEY PERFORMANCE INDICATORS WITH INDUSTRY STANDARDS E. WRAP THE DASHBOARD METRICS WITH CONTEXTUAL METADATA F. VALIDATE THE DASHBOARD DESIGN BY A USABILITY SPECIALIST G. PRIORITIZE AND RANK ALERTS/EXCEPTIONS STREAMED TO THE DASHBOARD H. ENRICH DASHBOARD WITH BUSINESS-USER COMMENTS I. PRESENT INFORMATION IN THREE DIFFERENT LEVELS J. PICK THE RIGHT VISUAL CONSTRUCT USING DASHBOARD DESIGN PRINCIPLES 3 .

K.

PROVIDE FOR GUIDED ANALYTICS  Section 3.6 Review Questions

3.7

BUSINESS PERFORMANCE MANAGEMENT A. CLOSED-LOOP BPM CYCLE ⧫ Application Case 3.7: IBM Cognos Express Helps Mace for Faster and Better Business Reporting  Section 3.7 Review Questions

3.8

PERFORMANCE MANAGEMENT A. KEY PERFORMANCE INDICATOR (KPI) B. PERFORMANCE MEASUREMENT SYSTEM  Section 3.8 Review Questions

3.9

BALANCED SCORECARDS A. THE FOUR PERSPECTIVES 1.

The Customer Perspective 2.

The Financial Perspective 3.

The Learning and Growth Perspective 4.

The Internal Business Process Perspective B. THE MEANING OF BALANCE IN BSC C. DASHBOARDS VERSUS SCORECARDS  Section 3.9 Review Questions

3.10

SIX SIGMA AS A PERFORMANCE MEASUREMENT SYSTEM A. THE DMAIC PERFORMANCE MODEL B. BALANCED SCORECARD VERSUS SIX SIGMA C. EFFECTIVE PERFORMANCE MANAGEMENT ⧫ Application Case 3.8: Expedia.com’s Customer Satisfaction Scorecard  Section 3.10 Review Questions

Chapter Highlights Key Terms QUESTIONS FOR DISCUSSION Exercises Teradata University and Other Hands-On Exercises Team Assignments and Role-Playing Projects ⧫ End of Chapter Application Case: Smart Business Reporting Helps Healthcare Providers Deliver Better Care  Questions for the Case References

4 .

TEACHING TIPS/ADDITIONAL INFORMATION

       

This chapter is different from those that have gone before or will come in the future. It discusses largely management issues, with secondary attention (though not no attention!) to the technologies used to support them. Management students will take to such things as the BPM processes or Six Sigma like ducks to water; in fact, they may have already studied some of them in other courses. Technically oriented students may wonder why you’re wasting their time.

The fact is that a great deal of BI work today is focused on BPM and the other areas covered in this chapter. They are today’s version of what was once called an “executive information system,” highly evolved but conceptually similar. Other than the technology having changed, so has the way these systems are used. The old EIS was pretty much independent of everything else that went on in the company; it could be used, or not used, according to the whims of each executive. Today’s BPM software is integrated into the planning structure of the organization. Using it is not optional. As regards your students: understanding that structure isn’t optional either if they are to work with this kind of software.

If you happen to have mostly a technical background yourself, you may find some of the concepts in this chapter a bit alien and difficult to teach. If it’s practical in your situation, you may be able to prevail on one of your more business-oriented colleagues to take a class or two at this point.

From a pedagogic point of view, this chapter is more unified than some of the others because it covers a subject area which is itself inherently more unified. You won’t have to add as many transitions from one topic to the next; they’re already there. Students will probably find that it flows better, too. Compared to the earlier chapters, you (and they) will probably also find that the Questions for Discussion at the end involve more hard work!

Section 3.2, the first section after the opening vignette, introduces business reporting concepts. The role of reporting is shown in the overall context of information technology, business functions, and managerial decision making. Three main categories of reports are described (metric management, dashboard-type, and balanced scorecard).

You may find that this breakdown is different from what you’ve read in previous texts.

For example, reporting systems have often been delineated between static, ad hoc (custom), and exception reports. But in this chapter, the notion of “static” reports is deemphasized, and reports are generally thought of as highly interactive and graphical entities.

Section 3.3 gives a history of data and information visualization. Students will be interested to see how ancient the ideas behind visualization are. Spend some time talking about Minard’s graphic of Napoleon’s Russian campaign (Figure 3.3). Students can see how color coding, sizes of visual entities, and geographic overlays can combine to form a rich information visualization. Later, when discussing different types of charts and graphs from section 3.4, you can compare Minard’s visualization with the GapMinder example of Figure 3.4. Here, size also plays a role (compare the size of Minard’s band depicting army size with the size of bubbles depicting country populations). Another thing to do in 5 .

this lecture would be to actually go to gapminder.com and watch the motion chart in action.

Section 3.5 discusses visual analytics, and distinguishes this topic from simple information visualization. Note that a sizable emphasis of visual analytics (as opposed to information visualization) is on its utility for prescriptive analytics (not just descriptive).

Keep in mind that prescriptive analytics topics aren’t discussed in depth until chapter 4, so students might not have enough context to truly appreciate the topics of this section until later. Another theme in this section is the importance of storytelling. Business reporting should tell compelling stories.

Section 3.6 explores dashboards, which leads naturally into section 3.7’s discussion of BPM. The basic BPM methodology breaks down into four processes, which are discussed in 3.7. Once students understand the processes, the rest of the chapter looks at specific aspects of them in more depth. In many cases these discussions are connected back to the four processes of BPM. In others, you might want to make the connection explicit yourself. Section 3.8 introduces performance measurement. This is followed in Sections 3.9 and 3.10 with discussions of Balanced Scorecard (BSC) and Six Sigma, respectively. It’s useful to compare these two and emphasize the strategic vs. operational nature of the two performance management approaches.

There are eight application cases in this chapter, and all bring some real-world examples that you can use to reinforce the concepts covered in this chapter. Many of the applications involve utilizing multiple reporting and performance management approaches and technologies.

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 3.1 Review Questions 1.

What does Travel and Transport, Inc., do?

Travel and Transport, Inc., is the sixth largest travel management company in the United States, with more than 700 employee-owners located nationwide. The company has extensive experience in multiple verticals, including travel management, loyalty solutions programs, meeting and incentive planning, and leisure travel services.

2.

Describe the complexity and the competitive nature of the business environment in which Travel and Transport, Inc., functions.

The travel industry is a very crowded marketplace. With travel arrangements often made on an ad hoc basis, customers find it difficult to analyze costs or instate optimal purchase agreements.

3.

What were the main business challenges?

In the field of employee travel services, the ability to effectively communicate a value proposition to existing and potential customers is critical to winning and retaining business. With travel arrangements often made on an ad hoc basis, 6 .

customers find it difficult to analyze costs or instate optimal purchase agreements.

Travel and Transport wanted to overcome these challenges by implementing an integrated reporting and analysis system to enhance relationships with existing clients, while providing the kind of value-added services that would attract new prospects.

4.

What was the solution? How was it implemented?

Travel and Transport implemented Information Builders’ WebFOCUS business intelligence (BI) platform (called eTTek Review) as the foundation of a dynamic customer self-service BI environment. This dashboard-driven expensemanagement application helps more than 800 external clients, and more than 200 internal employees, to plan, track, analyze, and budget their travel expenses more efficiently, and generate more precise forecasts. The approach was to create an online system that allows clients to access information directly, so they won’t have to rely on the IT department to run reports for them.

5.

Why do you think a multi-vendor, multi-tool solution was implemented?

Travel and Transport decided to combine multiple vendor tools, including Adobe Flex, WebFOCUS, DB2, and Global Distribution Systems (GDS). A combination of vendors and tools was required for their solution; each vendor brings a unique technological contribution to the mix. Adobe helps with the reporting features, WEBFOCUS contributes online BI capabilities, DB2 serves as the database backend, and GDS brings the domain expertise (i.e., travel agency knowledge) into the system.

6.

List and comment on at least three main benefits of the implemented system. Can you think of other potential benefits that are not mentioned in the case?

The implemented system benefits customers by enabling them to plan, track, analyze, and budget their travel expenses more efficiently. Customers are getting the best pricing from airlines, hotels, and car rental companies. For employees, including customer service representatives, the system helps with forecasting and quickly responding to client needs. Metrics derived from actionable data are key to helping Travel and Transport negotiate with their suppliers, enforce their travel policy, and save their company money.

Section 3.2 Review Questions 1.

What is a report? What are reports used for?

A report is any communication artifact prepared with the specific intention of conveying information in a presentable form to whoever needs it, whenever and wherever they may need it. It is usually a document that contains information (usually driven from data and personal experiences) organized in a narrative, 7 .

graphic, and/or tabular form, prepared periodically (recurring) or on an asrequired (ad hoc) basis, referring to specific time periods, events, occurrences, or subjects.

2.

What is a business report? What are the main characteristics of a good business report?

A business report is a written document that contains information regarding business matters. Business reporting (also called enterprise reporting) is an essential part of the larger drive toward improved managerial decision making and organizational knowledge management. The foundation of these reports is various sources of data coming from both inside and outside the organization.

Creation of these reports involves ETL (extract, transform, and load) procedures in coordination with a data warehouse and then using one or more reporting tools.

While reports can be distributed in print form or via e-mail, they are typically accessed via a corporate intranet. Primary characteristics of a good business report include clarity, brevity, completeness, and correctness.

3.

Describe the cyclic process of management and comment on the role of business reports.

The cyclic process of management, as illustrated in Figure 3.1, involves these steps: data acquisition leads to information generation which leads to decision making which leads to business process management. Perhaps the most critical task in this cyclic process is the reporting (i.e., information generation)— converting data from different sources into actionable information.

4.

List and describe the three major categories of business reports.

There are a wide variety of business reports, which for managerial purposes can be grouped into three major categories: metric management reports, dashboardtype reports, and balanced scorecard-type reports. Metric management reports involve outcome-oriented metrics based on service level agreements and/or key performance indicators. Dashboard-type reports present a range of performance indicators on one page, with both static/predefined elements and customizable widgets and views. Balanced scorecard reports present an integrated view of a company’s health and include financial, customer, business process, and learning/growth perspectives.

5.

What are the main components of a business reporting system?

A business reporting system includes several components. One is the online transaction processing system (ERP, POS, etc.) that records transactions. A second is a data supply that takes recorded events and transactions and delivers them to the reporting system. Next comes an ETL component that ensures quality and performs necessary transformations prior to loading the data into a data store.

Then there is the data storage itself (such as a data warehouse). Business logic converts the data into the reporting outputs. Publication distributes or hosts the 8 .

reports for end users. And finally assurance provides a quality control check on the reports and their dissemination.

Section 3.3 Review Questions 1.

What is data visualization? Why is it needed?

Data visualization, perhaps more appropriately called “information visualization,” is the use of visual representations to explore, make sense of, and communicate data. It is closely related to the fields of information graphics, scientific visualization, and statistical graphics. What is portrayed in visualizations is the information (aggregations, summarizations, and contextualization) and not the data. Companies and individuals increasingly rely on data to make good decisions. Because data is so voluminous, there is a need for visual tools that help people understand it.

2.

What are the historical roots of data visualization?

Predecessors to data visualization date back to the second century AD. Today’s most popular visual forms date back a few centuries. Geographical exploration, mathematics, and popularized history spurred the creation of early maps, graphs, and timelines as far back as the 1600s. The now familiar line and bar charts date back to the late 1700s. Charles Joseph Minard used visualizations to graphically portray the losses suffered by Napoleon’s army in the Russian campaign of 1812.

The 1900s saw the rise of a more formal, empirical attitude toward visualization, which tended to focus on aspects such as color, value scales, and labeling. In the 2000s the Internet has emerged as a new medium for visualization, and added interactivity to previously static graphics.

3.

Carefully analyze Charles Joseph Minard’s graphical portrayal of Napoleon’s march. Identify and comment on all of the information dimensions captured in this ancient diagram.

In this graphic Minard managed to simultaneously represent several data dimensions, including the size of the army, direction of movement, geographic locations, outside temperature, etc. He did this in an artistic and informative manner. The background of the image is a map depicting the location of battles.

There is a thick lighter band that shows the size of Napoleon’s army at each position, and a dark lower one that depicts the retreat. A line at the bottom depicts temperatures at each position in time and space.

4.

Who is Edward Tufte? Why do you think we should know about his work?

Edward Tufte is a statistician whose website chronicles many historical data visualizations, including Minard’s graphic of Napoleon’s defeat. His work can bring insights into how to follow best practices for information visualization.

9 .

5.

What do you think is the “next big thing” in data visualization?

The future of data/information visualization is very hard to predict. We can only extrapolate from what has already been invented: more three-dimensional visualization, more immersive experience with multidimensional data in a virtual reality environment, and holographic visualization of information. There is a pretty good chance that we will see something that we have never seen in the information visualization realm invented before the end of this decade.

Section 3.4 Review Questions 1.

Why do you think there are many different types of charts and graphs?

Different types of charts are appropriate for conveying different types of information. Line graphs are good for time-series data. Bar charts are good for depicting nominal or numerical data that can be easily categorized. Pie charts should be used for depicting proportions. Scatter plots and bubble charts are good for illustrating relationships between two or three variables (bubble charts add a dimension via the size of the dot). Histograms are like bar charts, except they depict frequency distributions. Gantt charts and PERT charts are good at illustrating project timelines and task dependencies. Geographic maps, of course, show geographic information. Bullet graphs show progress toward a goal. Heat maps and highlight tables illustrate the comparison of continuous values across two categories using color. Tree maps are good for showing hierarchical information. Even though these charts and graphs cover a major part of what is commonly used in information visualization, they by no means cover it all.

Nowadays, one can find many other specialized graphs and charts that serve a specific purpose.

2.

What are the main differences among line, bar, and pie charts? When should you use one over the others?

Line graphs are good for time-series data. Bar charts are good for depicting nominal or numerical data that can be easily categorized. Pie charts should be used for depicting proportions. You shouldn’t use pie charts if the number of categories is very large.

3.

Why would you use a geographic map? What other types of charts can be combined with a geographic map?

Geographic maps are useful when the data set includes any kind of location data, including addresses, postal codes, state names or abbreviations, country names, latitude/longitude, or some type of custom geographic encoding. Maps can be used in conjunction with other charts and graphs. For instance, one can use maps to show distribution of customer service requests by product type (depicted in pie charts) by geographic locations.

10 .

4.

Find two more kinds of charts that are not covered in this section and comment on their usability.

A concept map is a diagram that shows relationships between concepts, usually showing specific ideas and information as boxes and using arrows to connect them. Concept maps are often used by designers and engineers to organize ideas.

Another type of chart is an organization chart (or org-chart). This is a hierarchical, tree-structured chart that shows how an organization is structured and how its parts and jobs are related.

A motion chart is like a bubble chart in that it depicts data on dimensions of the xaxis, y-axis, size, and color of bubbles. In addition, however, it is also animated, so that bubbles move and resize themselves over time. The gapminder example from Figure 3.4 is an example of a motion chart.

Section 3.5 Review Questions 1.

What are the reasons for the recent emergence of visual analytics?

The growth of visual analytics correlates with the growth of analytics in general.

More BI and analytics vendors are becoming aware that their customers require quick and preferably interactive visualizations, not just for their normal reporting systems, but also to illustrate predictive and prescriptive decision-making information. Many of the information visualization vendors are adding the capabilities to call themselves visual analytics solution providers. Conversely, analytics solution providers such as SAS are embedding their analytics capabilities into a high-performance data visualization environment that they call visual analytics.

2.

Look at Gartner’s Magic Quadrant for Business Intelligence and Analytics Platforms. What do you see? Discuss and justify your observations.

What we can see from Gartner’s Magic Quadrant is that the vast majority of the “challengers” in the visual analytics space are considered to be “niche players,” while all of the “leaders” are “visionaries.” Most of the leaders are either relatively recently founded information visualization companies (e.g., Tableau Software, QlikTech, or Tibco Spotfire) or are well-established, large analytics companies (e.g., SAS, IBM, Microsoft, SAP, or MicroStrategy) that are increasingly focusing their efforts in information visualization and visual analytics. This chart, and the Gartner report from which it comes, shows that data discovery/visualization has become a mainstream architecture, perhaps surpassing the OLAP/semantic offerings of traditional BI vendors. This is further evidenced by the change of name of the magic quadrant from “BI” to “BI and Analytics.”

3.

What is the difference between information visualization and visual analytics?

11 .

Visual analytics is the combination of visualization and predictive analytics.

While information visualization is aimed at answering “what happened” and “what is happening” and is closely associated with business intelligence (routine reports, scorecards, and dashboards), visual analytics is aimed at answering “why is it happening,” “what is more likely to happen,” and is usually associated with business analytics (forecasting, segmentation, and correlation analysis).

4.

Why should storytelling be a part of your reporting and data visualization?

The central idea of business reporting is to tell a story. Everyone who has data to analyze has stories to tell, whether it’s diagnosing the reasons for manufacturing defects, selling a new idea in a way that captures the imagination of your target audience, or informing colleagues about a particular customer service improvement program. Stories bring life to data and facts. They can help you make sense and order out of a disparate collection of facts. They make it easier to remember key points and can paint a vivid picture of what the future can look like. Stories also create interactivity—people put themselves into stories and can relate to the situation. People will be much more engaged and receptive if information is presented to them in a story format.

5.

What is a high-powered visual analytics environment? Why do we need it?

A high-powered visualization environment is one in which high-performance, inmemory solutions are applied to exploring massive amounts of data in a very short time (almost instantaneously). Due to the increasing demand for visual analytics coupled with fast-growing data volumes, there is an ever-growing need to invest in highly efficient visualization systems. SAS Visual Analytics is an example of such an environment. These systems help to empower larger numbers of users, solve complex problems more quickly, and improve collaboration and information sharing. By enabling end-users, IT staff are freed up. In addition, these tools allow for growth at a self-determined pace.

Section 3.6 Review Questions 1. What is a performance dashboard? Why are they so popular for BI software tools?

Performance dashboards provide visual displays of important information that is consolidated and arranged on a single screen so that information can be digested at a single glance and easily drilled in and further explored. They are common components of most, if not all, performance management systems, performance measurement systems, BPM software suites, and BI platforms. Dashboards pack a lot of information into a single screen, which is one reason for their popularity.

12 .

2.

What are the graphical widgets commonly used in dashboards? Why? Dashboards can include many kinds of visual widgets, including charts, performance bars, sparklines, gauges, meters, stoplights, geographic maps, etc.

These help to highlight, at a glance, the data and exceptions that require action. A picture tells a thousand words, and through the use of many graphical widgets, a dashboard can convey a wealth of information to decision makers in a short time.

3.

List and describe the three layers of information portrayed on dashboards.

The three layers of information found in most dashboards are: 1. Monitoring. Graphical, abstracted data to monitor key performance metrics.

2. Analysis. Summarized dimensional data to analyze the root cause of problems.

3. Management. Detailed operational data that identify what actions to take to resolve a problem.

4.

What are the common characteristics of dashboards and other information visuals?

All well-designed dashboards share some common characteristics. They use visual components (e.g., charts, performance bars, sparklines, gauges, meters, stoplights) to highlight, at a glance, the data and exceptions that require action.

They are transparent to the user, meaning that they require minimal training and are extremely easy to use. They combine data from a variety of systems into a single, summarized, unified view of the business. They enable drill-down or drillthrough to underlying data sources or reports, providing more detail about the underlying comparative and evaluative context. They present a dynamic, realworld view with timely data refreshes, enabling the end user to stay up to date with any recent changes in the business. And they require little, if any, customized coding to implement, deploy, and maintain.

5.

What are the best practices in dashboard design?

• • • • • • •

Benchmark key performance indicators with industry standards.

Wrap the dashboard metrics with contextual metadata (e.g. data source, data currency, refresh schedule).

Prioritize and rank alerts/exceptions streamed to the dashboard.

Enrich the dashboard with business users’ comments.

Present information in three different levels (visual dashboard, static report, and self-service cube).

Pick the right visual construct using dashboard design principles.

Provide for guided analytics.

13 .

Section 3.7 Review Questions 1.

What is business performance management? How does it relate to BI? Business performance management (BPM) refers to the business processes, methodologies, metrics, and technologies used by enterprises to measure, monitor, and manage business performance. It is also known as corporate performance management (CPM), enterprise performance management (EPM), and strategic enterprise management (SEM). It can be considered to be a type of BI tool/technique. The most significant differentiator of BPM from any other BI tools and practices is its strategy focus. BPM encompasses a closed-loop set of processes that link strategy to execution in order to optimize business performance.

2.

What are the three key components of a BPM system?

According to Colbert (2009), a BPM encompasses three key components. The first is a set of integrated, closed-loop management and analytic processes (supported by technology) that addresses financial as well as operational activities. The second involves tools for businesses to define strategic goals and then measure and manage performance against those goals. And the third component involves a core set of processes, including financial and operational planning, consolidation and reporting, modeling, analysis, and monitoring of key performance indicators (KPIs), linked to organizational strategy.

3.

List and briefly describe the four phases of the BPM cycle.

The BPM cycle contains four main phases. First is to strategize. This involves answering the question, “Where do we want to go?”, and involves a high-level, long-term plan. Missions, visions, and objectives are key components of this phase. The second phase is to plan, which answers the question, “How do we get there?” Key elements here are a detailed operational plan and a financial plan including budget. The next phase is to monitor and analyze, which answers the question, “How are we doing?” Here is where KPIs, dashboards, reporting, and analytics are helpful. Finally come action and adjustment, based on comparing our analysis results against our plans. Sometimes this means changing the way we operate, and sometimes it means adjusting our strategy.

4.

Why is strategy the most important part of a BPM implementation?

Strategy is the art and the science of crafting decisions that help businesses achieve their goals. More specifically, it is the process of identifying and stating the organization’s mission, vision, and objectives. Business strategy provides an overall direction to the enterprise, which is why it is so important.

14 .

Section 3.8 Review Questions 1.

What is a performance management system? Why do we need one?

The purpose of a performance management system is to (a) identify and articulate the strategic mission, goals and objectives of an organization, and (b) assist managers in tracking the implementations of business strategy by comparing actual results against these strategic goals and objectives. The latter task is accomplished by a performance measurement system, which can be considered a subset of the overall performance management system. A performance measurement system typically comprises systematic methods of setting business goals together with periodic feedback reports that indicate progress against goals.

This is a key and necessary element of the BPM process.

2.

What are the distinguishing features of KPIs? The key features described in the book are strategy, targets, ranges, encodings, time frames, and benchmarks. KPIs embody strategic objectives and measure performance against specific targets, based on specified ranges of values.

Encodings provide visual cues (e.g., color) to indicate how close or far from a target we are on a particular metric. Benchmarks provide something to compare against.

3.

List and briefly define the four most commonly cited operational areas for KPIs. • • • •

4.

Customer performance. Metrics for customer satisfaction, speed and accuracy of issue resolution, and customer retention.

Service performance. Metrics for service-call resolution rates, service renewal rates, service-level agreements, delivery performance, and return rates.

Sales operations. New pipeline accounts, sales meetings secured, conversion of inquiries to leads, and average call closure time.

Sales plan/forecast. Metrics for price-to-purchase accuracy, purchase order-tofulfillment ratio, quantity earned, forecast-to-plan ratio, and total closed contracts.

What is a performance measurement system? How does it work?

A performance measurement system is one component of a performance management system. The most popular performance measurement systems in use are some variant of Kaplan and Norton’s balanced scorecard (BSC).

Section 3.9 Review Questions 1.

What is a balanced scorecard? Where did it come from?

Balanced Scorecard was first developed by Kaplan and Norton in their 1992 Harvard Business Review article, “The Balanced Scorecard: Measures That Drive 15 .

Performance.” It is a performance management system whose key feature is that it does not rely solely on financial measures of success. Over the past few years, BSC has become a generic term that is used to represent virtually every type of scorecard application and implementation, but it is intended to emphasize a strategic focus.

2.

What are the four perspectives that BSC suggests to view organizational performance?

The four perspectives are: customer, financial, internal business processes, and learning and growth. If customers are not satisfied, they will eventually find other suppliers that will meet their needs. Poor performance from this perspective is thus a leading indicator of future decline, even though the current financial picture may look good. Timely and accurate funding data will always be a priority, and managers will do whatever is necessary to provide it. This should include risk analysis. In the current climate of rapid technological change, it is becoming necessary for knowledge workers to be in a continuous learning and growing mode. Metrics based on this perspective allow the managers to know how well their internal business processes and functions are running, and whether the outcomes of these processes (i.e., products and services) meet and exceed the customer requirements (the mission).

3.

Why do we need to define separate objectives, measures, targets, and initiatives for each of these four BSC perspectives?

BSC is designed to overcome the limitations of systems that are financially focused. An organization’s vision and strategy should recognize the interrelation between financial and nonfinancial objectives, measures, targets, and initiatives.

Therefore, nonfinancial objectives form a simple causal chain with “learning and growth” driving “internal business process” change, which produces “customer” outcomes that are responsible for reaching a company’s “financial” objectives.

4.

What is the meaning of and motivation for balance in BSC? In BSC, the term “balance” arises because the combined set of measures is supposed to encompass indicators that are financial and nonfinancial, leading and lagging, internal and external, quantitative and qualitative, and both short term and long term.

5.

What are the differences and commonalities between dashboards and scorecards?

These terms are often used interchangeably, and they share many common features. The main difference is that scorecards are used by executives, managers, and staff to monitor strategic alignment and success with strategic objectives and targets. By contrast, dashboards are used at the operational and tactical levels.

Managers, supervisors, and operators use operational dashboards to monitor detailed operational performance on a weekly, daily, or even hourly basis.

16 .

Section 3.10 Review Questions 1.

What is Six Sigma? How is it used as a performance measurement system?

Most companies use Six Sigma as a process improvement methodology that enables them to scrutinize their processes, pinpoint problems, and apply remedies.

It's not used much as a performance management or measurement methodology.

As a performance tool, it is aimed at reducing the number of defects in a business process to as close to zero DPMO (defects per million opportunities) as possible.

2.

What is DMAIC? List and briefly describe the steps involved in DMAIC. DMAIC is a closed loop performance improvement model that involves the following steps: define, measure, analyze, improve, and control. First, you define the goals, objectives, and boundaries of the improvement activity. Next, you measure the existing system, in order to monitor its performance against the goals.

Then, you analyze the system to identify ways to eliminate the gap between the current performance of the system or process and the desired goal. This leads to improvement, which involves initiating actions to reduce these gaps. Finally, control involves modifying compensation and incentive systems, policies, procedures, manufacturing resource planning, budgets, operation instructions, or other management systems.

3.

Compare BSC and Six Sigma as two competing performance measurement systems.

BSC is focused on improving overall strategy, whereas Six Sigma is focused on improving processes. BSC gives a longer-term view of the organization, whereas Six Sigma gives a snapshot at a particular point in time of its operational effectiveness. BSC focuses on long-term growth, whereas Six Sigma emphasizes current profitability. These are a few of the differences between the two. Some companies choose to give a more holistic performance assessment by combining elements of both approaches.

4.

What are the ingredients for an effective performance management system?

Effective performance management/measurement should focus on key factors. It should mix past, present, and future. Also, it should balance the needs of shareholders, employees, partners, suppliers, and other stakeholders. Performance measures should start at the top and flow to the bottom, and should involve targets that are based on research and reality rather than arbitrary.

17 .

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 3.1: Delta Lloyd Group Assures Accuracy and Efficiency in Financial Reporting 1.

How did Delta Lloyd Group improve accuracy and efficiency in financial reporting?

They switched from a manual, spreadsheet-based process to utilizing IBM Cognos for their financial reporting.

2.

What were the challenges, the proposed solution, and the obtained results?

The most challenging part of the reporting process is the “last mile”—the stage at which the consolidated figures are cited, formatted, and described to form the final text of the report. They used to do this with Microsoft Excel and manual processes, which is very time consuming and error-prone. Accuracy is essential in financial reporting, since errors could lead to penalties, reputational damage, and even a negative impact on the company’s stock price. The solution was to implement IBM Cognos Financial Statement Reporting (FSR). This allowed the workload to be divided up, so that staff were able complete the report with 25% less overtime. It also allowed more flexibility; staff members could update figures automatically across the whole report which reduced the scope for manual error inherent in spreadsheet-based processes and activities.

3.

Why is it important for Delta Lloyd Group to comply with industry regulations?

Since Delta Lloyd Group is publicly listed on the NYSE Euronext Amsterdam, it is obliged to produce annual and half-year reports. Their subsidiaries must also produce reports to fulfill local legal requirements for banking and insurance purposes. They must provide reports to meet international requirements, such as the IFRS (International Financial Reporting Standards).

Application Case 3.2: Flood of Paper Ends at FEMA 1.

What is FEMA and what does it do?

The Federal Emergency Management Agency (FEMA) is a U.S. federal agency that coordinates disaster response when the President declares a national disaster.

2.

What are the main challenges that FEMA faces?

18 .

When a disaster occurs, FEMA is inundated with a huge amount of paperwork to sift through in order to administer the National Flood Insurance Program (NFIP).

Sifting through this paperwork is very cumbersome and labor-intensive.

3.

How did FEMA improve its inefficient reporting practices?

The solution was to implement a system based on WebFOCUS software from Information Builders. As a result, FEMA staff can now browse insurance data posted on NFIP’s BureauNet intranet site, select just the information they want to see, and get an onscreen report or download the data as a spreadsheet. This also allows them to create custom reports without having to go through their IT provider, CSC. The first major test of this system was Tropical Storm Allison, and BureauNet worked very well. It also has been able to scale up to handle increased demand.

Application Case 3.3: Tableau Saves Blastrac Thousands of Dollars with Simplified Information Sharing 1.

How did Blastrac achieve significant cost saving in reporting and information sharing?

They switched from a manual, Excel-based approach of reporting to using Tableau Software for data visualization and reporting.

2.

What were the challenge, the proposed solution, and the obtained results?

Blastrac depends on the creation and distribution of reports across the organization to make business decisions, but had no consistent reporting method in place. This caused them to spend a lot of time extracting data from the multiple ERP systems, loading it into several Excel spreadsheets, creating filtering capabilities, and establishing predefined pivot tables. This led to many errors. An enterprise BI system would have been too costly. So, they deployed two data visualization tools from Tableau Software (Tableau Desktop to create reports and Tableau Reader to view and filter them). As a result, only one hour per week is now required to create reports. Also, updates to these reports happen automatically through Tableau. This has resulted in a 95 percent increase in productivity.

Application Case 3.4: TIBCO Spotfire Provides Dana-Farber Cancer Institute with Unprecedented Insight into Cancer Vaccine Clinical Trials 1.

How did Dana-Farber Cancer Institute use TIBCO Spotfire to enhance information reporting and visualization?

19 .

TIBCO Spotfire was used to conduct data exploration and discovery regarding data from clinical trials that had been recorded in an XML-based data registry.

This automated process was much more effective and efficient than manual, spreadsheet-based data analysis.

2.

What were the challenges, the proposed solution, and the obtained results?

Dana Farber needs to study data from many sources in order to understand the relationships underlying the data and identify trends and patterns from clinical vaccine trials. This data was obtained from 645 clinical trials related to cancer vaccines. The data was extracted in XML from the ClinicalTrials.gov website.

There is great inconsistency and redundancy in this data registry. It was too difficult and time consuming to analyze information from the multiple data sources separately using spreadsheets. The solution was to use TIBCO Spotfire DXP (Spotfire) computational and visual analysis tool for data exploration and discovery. This solution provides the Dana-Farber and the research community at large with a better understanding of the cancer vaccine clinical trials landscape and enables rapid insight into the hotspots of cancer vaccine activity, as well as into the identification of neglected cancers.

Application Case 3.5: Dallas Cowboys Score Big with Tableau and Teknion 1.

How did the Dallas Cowboys use information visualization?

They incorporated Tableau and Teknion to assist with visualizing and understanding their merchandising activities, involving the complete supply chain from manufacture to end customer.

2.

What were the challenges, the proposed solution, and the obtained results?

The Dallas Cowboys Merchandising Division needed more visibility into their data so they could run more profitably. Microsoft was selected as the baseline platform that had been selected was not sufficient for the task. Tableau and Teknion together provided real-time reporting and dashboard capabilities that provided the necessary visualization functionality to meet and exceed the Cowboys’ requirements. Now, for the first time, the Dallas Cowboys are able to monitor their complete merchandising activities from manufacture to end customer and see not only what is happening across the life cycle, but drill down even further into why it is happening.

Application Case 3.6: Saudi Telecom Company Excels with Information Visualization

20 .

1.

Why do you think telecommunication companies are among the prime users of information visualization tools?

Telecommunications companies typically have millions of customers, contacting them constantly for billing, payment, network usage, and support. All of this information has to be monitored in a timely fashion.

2.

How did Saudi Telecom use information visualization?

STC used visualization to monitor information regarding network statistics, service analytics, and customer calls. Part of this requires prioritizing and contextualizing the data, identifying the relevant metrics, and viewing KPIs.

3.

What were their challenges, the proposed solution, and the obtained results?

STC needed to identify the relevant metrics, properly visualize them, and provide them to the right people, often with time-sensitive information. But executives didn’t have the ability to see key performance indicators in a timely fashion. They would have to contact the technical teams to get status reports, which often came too late and to be of benefit for preventing problems before they happen. The solution was to contract with Dundas business intelligence consultants to refine the telecommunication dashboards and improve their functionality. This led to engagement on an enterprise-wide, mission-critical project to transform their data center and create a more proactive monitoring environment. Dundas’ information visualization tools allowed STC to see trends and correct issues before they became problems. This resulted in a decrease in the amount of service tickets by 55 percent.

Application Case 3.7: IBM Cognos Express Helps Mace for Faster and Better Business Reporting 1.

What was the reporting challenge Mace was facing? Do you think this is an unusual challenge specific to Mace?

Many of Mace’s international projects are contracted on a fixed-price basis, and their success depends on the company’s ability to control costs and maintain profitability. Until recently, the only way for senior managers to gain a full understanding of international operations was a monthly report, based on a complex spreadsheet which drew data from numerous different accounting systems in subsidiaries around the world. This spreadsheet had about 40 tabs and hundreds of cross-links, which meant it was very easy to introduce errors, and the lack of a standardized approach was affecting accuracy and consistency.

21 .

This is not a unique problem for Mace. Many businesses face similar issues, especially those with an international component who contract on fixed-price basis like Mace.

2.

What was the approach for a potential solution?

The solution was to use IBM Cognos Express as the foundation of their international reporting platform; they were already using Cognos for their domestic business. The implementation was completed within six months.

3.

What were the results obtained in the short term, and what were the future plans?

The resulting system produces much more useful and dynamic reports, including drill-down capabilities. Reporting is automated and easily customizable. Mace is now in a better position to monitor performance and maintain profitability while ensuring that their projects are delivered on time and within budget. In the future, Mace is working to unite all its UK and international subsidiaries into this single financial reporting and budgeting system. The company also plans to continue working with IBM to explore other possible benefits of Cognos for facilitating standardization and financial control.

Application Case 3.8: Expedia.com’s Customer Satisfaction Scorecard 1.

Who are the customers for Expedia.com? Why is customer satisfaction a very important part of their business?

Expedia, Inc., is the parent company to some of the world’s leading travel companies (hotels.com, hotwire.com, expedia.com). Therefore, it acts as an online travel agent for consumers who want to travel. Because Expedia.com is an online business, the customer’s shopping experience is critical to Expedia’s revenues.

The online shopping experience can make or break an online business. Obviously, the travel experience is also critical to Expedia’s success.

2.

How did Expedia.com improve customer satisfaction with scorecards?

Scorecards give Expedia much better insight into customer satisfaction by providing staff, managers and executives instant visualization and reporting of patterns and trends. Through KPIs, the scorecard also allows for comparison of the customer patterns and trends to Expedia’s corporate goals and objectives.

3.

What were the challenges, the proposed solution, and the obtained results?

Because the customer experience is critical, all customer issues need to be tracked, monitored, and resolved as quickly as possible. But Expedia had no uniform way of measuring satisfaction, of analyzing the drivers of satisfaction, or 22 .

of determining the impact of satisfaction on the company’s profitability or overall business objectives. However, there was plenty of data to work with. It took a business analyst two to three weeks every month to pull and aggregate the data, leaving virtually no time for analysis. Expedia’s solution was to start by identifying key drivers of customer satisfaction. From these drivers, the customer satisfaction group constructed a scorecard application. They came up with 10 to 12 objectives that linked directly to Expedia’s corporate initiatives, which translated into more than 200 KPIs. This involved three steps: deciding how to measure satisfaction, setting performance targets, and putting the data into context. All of this was applied to their data warehouse (which they called DSS Factory). As a result, managers and executives have a transparent view of how well actions are aligning with the strategy, with the ability to drill down into the data underlying any of the trends or patterns observed. This would have taken weeks to do previously. Other business units are also benefitting.

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION 1.

  

What are the best practices in business reporting? How can we make our reports stand out?

Business reporting is about good story-telling. Here are some best practices for telling compelling stories with your business report: Think of your analysis as a story—use a story structure. Be authentic—your story will flow. Be visual—think of yourself as a film editor. Make it easy for your audience and you. Invite and direct discussion.

Business reports are often presented in the form of a dashboard or scorecard.

Following are some best practices of dashboard design: • Benchmark key performance indicators with industry standards.

• Wrap the dashboard metrics with contextual metadata (e.g. data source, data currency, refresh schedule).

• Prioritize and rank alerts/exceptions streamed to the dashboard.

• Enrich the dashboard with business users’ comments.

• Present information in three different levels (visual dashboard, static report, and self-service cube).

• Pick the right visual construct using dashboard design principles.

• Provide for guided analytics.

2.

Why has information visualization become a centerpiece in business intelligence and analytics? Is there a difference between information visualization and visual analytics?

A picture tells a thousand words. Hence, information visualization is key to quickly providing understanding of complex information. This has been true for hundreds of years, as evidenced by Minard’s graphical depiction of Napoleon’s 23 .

defeat. In an increasingly complex information age, visualization is even more important. Since BI and analytics are all about making sense of high volume, fastpaced, in interrelated data, it is natural that information visualization would be a centerpiece. BI companies are rapidly adding information visualization and visual analytics into their products and services.

Information visualization and visual analytics are related but distinct concepts.

Visual analytics is the combination of visualization and predictive analytics.

While information visualization is aimed at answering “what happened” and “what is happening” and is closely associated with business intelligence (routine reports, scorecards, and dashboards), visual analytics is aimed at answering “why is it happening,” “what is more likely to happen,” and is usually associated with business analytics (forecasting, segmentation, and correlation analysis).

3.

Do you think performance dashboards are here to stay? Or are they about to be outdated? What do you think will be the next big wave in business intelligence and analytics?

Like all technology, performance dashboards will evolve and morph over time.

Dashboards are not a new concept. Their roots can be traced at least to the EIS of the.1980s. Today, dashboards are ubiquitous. Currently they are used as reporting tools, largely for the purpose of evaluating a company’s performance, and are often tied with business performance methodologies like Balanced Scorecard or Six Sigma. The specific visualizations in a dashboard (charts, graphs, and maps) will evolve over time, and the technology platforms may change. For example, with virtual reality, holographic and immersive technologies, the user experience may change considerably. But the purpose of a dashboard to monitor company performance and respond to warnings and events will continue. (Answers will vary by student.)

4.

SAP uses the term strategic enterprise management (SEM), Cognos uses the term corporate performance management (CPM), and Hyperion uses the term business performance management (BPM). Are they referring to the same basic ideas?

Provide evidence to support your answer.

In the business and trade literature, business performance management (BPM) has a number of names, including corporate performance management (CPM), enterprise performance management (EPM), and strategic enterprise management (SEM). CPM was coined by the market analyst firm Gartner (gartner.com). EPM is a term associated with Oracle’s (oracle.com) offering by the same name. SEM is the term that SAP (sap.com) uses. All of these terms refer to the same general idea. They all encompass a closed-loop set of processes that link strategy to execution in order to optimize business performance.

5.

BPM encompasses five basic processes: strategize, plan, monitor, act, and adjust.

Select one of these processes and discuss the types of software tools and applications that are available to support it. Figure 3.10 provides some hints. Also, 24 .

refer to Bain & Company’s list of management tools for assistance (bain.com/management_tools/home.asp).

(Students will select a single process. Below is one paragraph for each of the first three processes; Act/Adjust is usually something that involves human decision making as a result of the previous three phases.)

Strategize: One type of software for the strategize process involves applications that assist with the development of strategy maps. These applications are often components of a BSC software. Strategy map software enables users to build diagrams that tie objectives to different perspectives (for example, the BSC perspectives of customer, finance, process, and learning/growth). There are many products on the market that assist with strategy map formulation, including SmartDraw, Corporater, and StrategyKpi. Plan: The planning process involves creating budgets, project plans, forecasts, models, initiatives, and targets. Budgeting software can be as simple as Excel spreadsheets, or can involve accounting packages such as Quicken or Quickbooks or Peachtree. Here is a link to an article describing various options for budgeting software: http://www.forbes.com/sites/moneywisewomen/2012/01/03/budgetingsoftware-options/. Planning software will include project management tools such as Microsoft Project. Forecasting tools typically involve predictive analytics, and there are many products that combine budgeting and forecasting together, as described in this article: http://www.softwareadvice.com/accounting/budgetingforecasting-software-comparison/.

Monitor/Analyze: Monitoring and analyzing typically involves dashboards, scorecards, and visual analytics tools. Dashboards provide graphical visuals of various types. Gartner’s Magic Quadrant shows several companies that either specialize in dashboards and information visualization (e.g., Tableau Software, QlikTech, and Tibco Spotfire) or incorporate these tools into broader BI and analytics offerings (e.g., SAS, IBM, Microsoft, SAP, and MicroStrategy) 6.

Select a public company of interest. Using the company’s 2013 annual report, create three strategic financial objectives for 2014. For each objective, specify a strategic goal or target. The goals should be consistent with the company’s 2013 financial performance.

(This question will obviously vary significantly per student. A good approach to answering this question is to do what Figure 3.12 does, by combining a strategy map with a balanced scorecard.)

25 .

7.

Netflix’s strategy of moving to online video downloads has been widely discussed in a number of articles that can be found online. What are the basic objectives of Netflix’s strategy now? What are some of the major assumptions underlying the strategy? Given what you know about discovery-driven planning, do these assumptions seem reasonable?

(Student answers will vary. The following answer is based primarily on the following sources: https://lpelin.expressions.syr.edu/trf483/files/2013/05/final_strategic_report.pdf, http://ir.netflix.com/long-term-view.cfm, http://www.cnbc.com/id/100905886, and http://www.strategy-business.com/article/cs00003?pg=all) Netflix started as a mail-order DVD rental business, in which customers would pay for each DVD they rented. This was sort of like the Blockbuster model, except that it was done remotely. Netflix did this without a brick-and-mortar store. Instead, customers would order videos online and receive them in the characteristic red envelope. Also, Netflix eschewed fines and late fees, which gave them a jump up in customer loyalty.

As Web technologies evolved and ISP speeds improved, Netflix shifted its primary delivery to online streaming. The DVD side of the business has declined significantly, and in fact they attempted to split that off into a separate company called Qwikster, a move that irritated many of its customers and even led to a lawsuit.

Another major shift with Netflix was to begin developing their own content. The award-winning series House of Cards is an example. The delivery of this show was also innovative; rather than a week-by-week delivery, they made a whole season available at once, giving customers the ability to view the show at their own pace.

Netflix’s 2013 strategy document includes a SWOT analysis, which sheds light on the underlying assumptions driving their top decision makers’ thinking. The major challenges they face are (a) high cost and debt for acquiring content from providers, (b) significant losses in their physical media delivery (DVD mailings), (c) potentially higher ISP costs, and (d) competitors such as Amazon and Hulu.

By contrast, their strengths and opportunities include (a) strong brand identity, (b) success with their early attempts at providing original content, and (c) a rising stock price.

Netflix has taken advantage of Web analytics to track customer behavior patterns.

In this way they are using discovery-driven planning. To illustrate, here is a quote from their 2014 strategic plan: “Another area of focus is personalized recommendations and merchandising, which drives what content we feature on a member’s initial screen. Google search is an example of a ranking system, where 26 .

results are automatically computed to show Google’s estimate of the most relevant answer to the query. For Netflix, the member’s opening screen is the personalized ranking of what we think will be most relevant for that specific member selected from our diverse catalog. By analyzing terabytes of data from every recent click, view, repeat view, early abandon, page views and other data, we are able to generate a personalized display with the content most likely to please. Our aim is to keep inventing and tuning algorithms to generate higher satisfaction, viewing, and retention.”

Clearly, Netflix is embracing analytics in order to refine their marketing efforts as well as their overall business strategy. This seems to help them stay agile and competitive as top players in the entertainment industry.

8.

In recent years, the Beyond Budgeting Round Table (BBRT; bbrt.org) has called into question traditional budgeting practices. A number of articles on the Web discuss the BBRT’s position. In the BBRT’s view, what is wrong with today’s budgeting practices? What does the BBRT recommend as a substitute?

BBRT’s “beyond budgeting” is an idea that budgeting is an inherently flawed process and that companies should move beyond budgeting. Companies should instead employ a range of techniques, such as rolling forecasts, to replace traditional budgeting. In this sense, “beyond budgeting” is considered to be more adaptive and less centralized than traditional budgeting.

BBRT advocates claim that this helps alleviate several drawbacks of traditional budgeting, which are time consuming, expensive, and rarely focus on strategy.

Traditional budgets limit flexibility and often discourage change, and focus too much on reducing costs rather than creating value.

With BBRT, “budget” does not refer solely to money expenditures, but to the entire performance management process. In this sense it shares a common philosophy with BSC. The BBRT approach involves setting clear priorities and boundaries, with empowered teams that take responsibility for value creating systems.

(NOTE: This information comes from http://www.cimaglobal.com/Documents/ImportedDocuments/cid_tg_beyond_bud geting_oct07.pdf)

9.

Distinguish between performance management and performance measurement.

There is a difference between a performance measurement system and a performance management system. The latter encompasses the former. That is, any performance management system has a performance measurement system, but not the other way around. If you were to ask, most companies today would claim that they have a performance measurement system but not necessarily a performance 27 .

management system, even though a performance measurement system has very little, if any, use without the overarching structure of the performance management system. Probably the best-known and most widely used performance management system is the balanced scorecard (BSC).

The purpose of a performance measurement system is to assist managers in tracking the implementations of business strategy by comparing actual results against strategic goals and objectives. A performance measurement system typically comprises systematic methods of setting business goals together with periodic feedback reports that indicate progress against goals. This is a key and necessary element of the BPM process.

10.

Create a measure for some strategic objective of interest (you can use one of the objectives formulated in discussion question 6). For the selected measurement, complete the measurement template found in Table W3.2.1 in the online file for this chapter.

(This question will vary significantly per student. A good approach to answering this question is to do what Figure 3.12 does, by combining a strategy map with a balanced scorecard. Here is an example.)

Such an objective for a company entering a brand new market might be a market share increase from 20 to 30 percent. For that objective, a possible measurement template might look like this:

Measurement Name Purpose

Property What is it called, and is it understandable?

Why is it being used, and what actions is it encouraging?

Relates to

What strategies does it support, and what other measures is it linked to?

Metric/formula

What is the actual measurement?

Target level(s) Frequency

What is the desirable level of performance?

How often is the measurement made and reported?

Where do the data come from?

Source of data Who measures

Who acts on the data (owner)

What are the name and function of the person responsible for collecting, collating, and analyzing the data?

Who—name and function—is responsible for ensuring the performance levels?

28 .

Explanation Market share increase; yes Used to measure improvement in our competitive position in our industry; encourages sales Supports strategy of expanding market penetration to obtain new customers while market is still in early growth stages; linked to shipments Ratio of sales revenue to total market size using public industry data 30 percent Monthly Our data: accounting; industry data: IDC John Doe, Business Analyst

Jane Roe, VP of Sales

Measurement What do they do

Property How will the owner use the data, and what actions can he or she take to improve performance?

Efficiency

Are all the data that are being collected used?

Are there easier ways to collect the data?

Is there any duplication among the measures?

Can the amount of data being collected be reduced through sampling?

Can the frequency of collection or reporting be reduced?

Effectiveness

Can the measure be eliminated because it is no longer being used?

Are there new tools that can be used to produce the measure in new ways?

Are all key processes adequately measured?

Are proper process measures in place to keep performance within acceptable limits?

Is everyone getting the information they need?

Does everyone understand the measures so they can interpret the results?

Are users taking action as a result of the measures?

Can the measures be easily used?

Are the measures sufficiently timely and accurate?

Does everyone have access to the measures they need?

Is the relative impact of different measures clear to users?

Have the measures been changed to reflect changes in processes?

29 .

Explanation If the data do not show an increase in market share tending toward at least 30 percent, she will use this as evidence of a need to develop new sales programs to achieve it They will be No, as far as we know No No; it must be collected for other reasons in any case No; monthly collection frequency is required for other reasons No; it is being used No Yes Yes Yes Yes Yes Yes Yes Yes Yes N/A, processes have not changed

11.

Create a strategy for a hypothetical company, using the four perspectives of the BSC. Express the strategy as a series of strategic objectives. Produce a strategy map depicting the linkages among the objectives.

(This answer will vary significantly depending on the hypothetical company and industry that the student selects. The key here is for students to discuss the company in terms of customer, financial, process, and learning/growth perspectives. For each of these perspectives, students should come up with objectives, linked together in a strategy map. These objectives can then be translated into measures and targets, and finally action plans. Figure 3.12 is a useful guide for students to follow.)

One example of a hypothetical company is Microsoft’s Adventure Works sample, found at http://technet.microsoft.com/en-us/library/ms124825(v=sql.100).aspx.

Adventure Works is a fictional bicycle manufacturing company. In their case, the primary business process will be the manufacture and sale of bicycles, and many measures can be incorporated, including measures of waste and scrap. Adventure Works sells primarily to retail stores and secondarily to individual customers through its online store (they do not have brick-and-mortar stores of their own).

Questions of customer retention therefore have two sub-perspectives (retail stores and individuals). Many factors can be brought to bear, including sales/marketing, supply chain and purchasing, and human resource concerns. Adventure Works is a very useful case for a BI class because it is used extensively in SqlServer’s sample database and data warehouse, as well as including many online tutorials for a wide range of data integration, OLAP, and data mining topics that are relevant for this textbook.

12.

Select two companies that you are familiar with. What terms do they use to describe their BPM initiatives and software suites? Compare and contrast their offerings in terms of BPM applications and functionality.

The DMAIC model (the performance improvement model used in the Six Sigma methodology) and BPM are both closed-loop models. However, their purposes are different: BPM is for overall monitoring of an organization, while the DMAIC model is focused on improving performance of a process or area. Their emphasis and the resulting processes therefore differ. BPM has four processes (strategize, plan, monitor, act and adjust) while the DMAIC model has five (define, measure, analyze, improve, control). Specific differences between them include: • • •

BPM takes a more strategic approach. It begins with strategizing. DMAIC takes strategic objectives as given, and continues from there.

The Plan process of BPM corresponds to the Define process of the DMAIC model.

The DMAIC model, as fits its purpose, goes into more detail in the later (implementation) stages of the improvement process, which BPM lumps under “act and adjust.”

30 .

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS 1.

 

What is Premier? What does it do?

Premier is a healthcare services company that serves more than 2,600 U.S. hospitals and 84,000-plus other healthcare sites. It exists to help its members improve the cost and quality of the care they provide the communities they serve.

Premier also assists its members to prepare for and stay ahead of health reform, including accountable care and other new models of care delivery and reimbursement.

2.

What were the main challenges for Premier to achieve its vision?

Premier had developed a series of “siloed” applications, making it difficult for members to connect different data sources and metrics and see the “big picture” of how to drive healthcare transformation. These platforms and associated software systems also lacked the scalability required to support the massive transaction volumes that were needed. The company’s existing technical infrastructure could not support their needs for better integration.

3.

What was the solution provided by IBM and other partners?

Premier “re-platformed” their IT architecture to enable the sharing and analysis of data from its thousands of member organizations. The new data architecture and infrastructure uses IBM software and hardware, including a data model called IBM Healthcare Provider Data Warehouse (HCPDW), based on IBM DB2 pureScale. The organization’s IT division is building its new infrastructure from the ground up. This includes replacing its existing x86 servers from a variety of hardware vendors with IBM POWER7 processor-based systems. Data from IBM DB2 database software is loaded to an IBM Netezza data warehouse so that members can conduct advanced and rapid analytics.

The new platform is designed to deliver trusted information in the right context at the right time to users based on their roles. Using the new platform, Premier members will be able to use the portal to access the integrated system for various clinical, business, and compliance-related applications. This helps healthcare organizations provide their staff with accurate and timely information to support evidence-based, patient-centric, and accountable care.

4.

What were the results? Can you think of other benefits coming from such an integrated system?

Potential benefits for saving lives, helping people enjoy healthier lives, and reducing healthcare costs are enormous. In one Premier project, 157 participating hospitals saved an estimated 24,800 lives while reducing healthcare spending by 31 .

$2.85 billion. Over the next 5 years, Premier plans to provide its members with many new applications in support of healthcare reform and other legislative requirements. As capabilities are added, the SaaS model will enable the platform to support its commitment to keep all 2,600 hospital members on the same page, and even expand its user community.

32 .

CHAPTER

4

Data Mining

Learning Objectives for Chapter 4 ▪

Describe data mining as an enabling technology for business analytics

▪

Understand the objectives and benefits of data mining

▪

Become familiar with the wide range of applications of data mining

▪

Learn the standardized data mining processes

▪

Understand the steps involved in data preprocessing for data mining

▪

Learn different methods and algorithms of data mining

▪

Build awareness of the existing data mining software tools

▪

Understand the privacy issues, pitfalls, and myths of data mining

CHAPTER OVERVIEW Generally speaking, data mining is a way to develop intelligence (i.e., actionable information or knowledge) from data that an organization collects, organizes, and stores.

A wide range of data mining techniques are being used by organizations to gain a better understanding of their customers and their own operations and to solve complex organizational problems.

In this chapter, we study data mining as an enabling technology for business analytics, learn about the standard processes of conducting data mining projects, understand and build expertise in the use of major data mining techniques, develop awareness of the existing software tools, and explore privacy issues, and common myths and pitfalls that are often associated with data mining. .

1 .

CHAPTER OUTLINE 4.1

OPENING VIGNETTE: CABELA’S REELS IN MORE CUSTOMERS WITH ADVANCED ANALYTICS AND DATA MINING  Questions for the Opening Vignette A. WHAT WE CAN LEARN FROM THIS VIGNETTE

4.2

DATA MINING CONCEPTS AND APPLICATIONS ⧫ Application Case 4.1: Smarter Insurance: Infinity P&C Improves Customer Service and Combats Fraud with Predictive Analytics A. DEFINITIONS, CHARACTERISTICS, AND BENEFITS ⧫ Technology Insights 4.1: A Simple Taxonomy of Data B. HOW DATA MINING WORKS ⧫ Application Case 4.2: Harnessing Analytics to Combat Crime: Predictive Analytics Helps Memphis Police Department Pinpoint Crime and Focus Police Resources 1.

Prediction 2.

Classification 3.

Clustering 4.

Associations 5.

Visualization and Time-Series Forecasting C. DATA MINING VERSUS STATISTICS  Section 4.2 Review Questions

4.3

DATA MINING APPLICATIONS ⧫ Application Case 4.3: A Mine on Terrorist Funding  Section 4.3 Review Questions

4.4

DATA MINING PROCESS A. STEP 1: BUSINESS UNDERSTANDING B. STEP 2: DATA UNDERSTANDING C. STEP 3: DATA PREPARATION D. STEP 4: MODEL BUILDING ⧫ Application Case 4.4: Data Mining in Cancer Research E. STEP 5: TESTING AND EVALUATION F. STEP 6: DEPLOYMENT G. OTHER DATA MINING STANDARDIZED PROCESSES AND METHODOLOGIES  Section 4.4 Review Questions

4.5

DATA MINING METHODS A. CLASSIFICATION

2 .

B.

C.

D.

ESTIMATING THE TRUE ACCURACY OF CLASSIFICATION MODELS 1.

Simple Split 2. k-Fold Cross-Validation 3.

Additional Classification Assessment Methodologies 4.

Classification Techniques 5.

Decision Trees ⧫ Application Case 4.5: 2degrees Gets a 1275 Percent Boost in Churn Identification CLUSTER ANALYSIS FOR DATA MINING 1.

Determining the Optimal Number of Clusters 2.

Analysis Methods 3.

K-Means Clustering Algorithm ASSOCIATION RULE MINING 1.

Apriori Algorithm  Section 4.5 Review Questions

4.6

DATA MINING SOFTWARE TOOLS ⧫ Application Case 4.6: Data Mining Goes to Hollywood: Predicting Financial Success of Movies  Section 4.6 Review Questions

4.7

DATA MINING PRIVACY ISSUES, MYTHS, AND BLUNDERS A. DATA MINING AND PRIVACY ISSUES ⧫ Application Case 4.7: Predicting Customer Buying Patterns—The Target Story B. DATA MINING MYTHS AND BLUNDERS  Section 4.7 Review Questions

Chapter Highlights Key Terms QUESTIONS FOR DISCUSSION Exercises Teradata University Network (TUN) and Other Hands-On Exercises Team Assignments and Role-Playing Projects Internet Exercises ⧫ End of Chapter Application Case: Macys.com Enhances Its Customers’ Shopping Experience with Analytics References

TEACHING TIPS/ADDITIONAL INFORMATION

       

Data mining was mentioned briefly in previous chapters. Students have 3 .

encountered the term several times but will only have a general idea of what it means. By the time they finish this chapter, students will have a comprehensive understanding of key data mining topics and applications.

The order of sections in this chapter follows a logical sequence to make teaching this topic easy. After the opening vignette, Section 4.2 discusses the general concepts.

Section 4.3 covers a wide range of applications. Section 4.4 focuses on the data mining process. Section 4.5 describes the most popular data mining methods and explains their representative techniques. Section 4.6 covers software vendors, commercial tools, and open source tools for modeling and data visualization. Finally, section 4.7 covers privacy concerns.

This chapter contains seven application cases that can help the material come alive for students. The Internet exercises at the end are also helpful, especially the ones that send students to vendor sites for case studies and white papers. It can be a good class or group project for students to go to several different vendor sites and compare what they find on each.

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 4.1 Review Questions 1.

Why should retailers, especially omni-channel retailers, pay extra attention to advanced analytics and data mining?

Utilizing large and information-rich transactional and customer data (that they collect on a daily basis) to optimize their business processes is not a choice for large-scale retailers anymore, but a necessity to stay competitive.

2.

What are the top challenges for multi-channel retailers? Can you think of other industry segments that face similar problems?

The retail industry is amongst the most challenging because of the change that they have to deal with constantly. Understanding customer needs, wants, likes, and dislikes is an ongoing challenge. As the volume and complexity of data increase, so does the time spent on preparing and analyzing it.

Prior to the integration of SAS and Teradata, data for modeling and scoring customers was stored in a data mart. This process required a large amount of time to construct, bringing together disparate data sources and keeping statisticians from working on analytics.

4 .

3.

What are the sources of data that retailers such as Cabela’s use for their data mining projects?

Cabela’s uses large and information-rich transactional and customer data (that they collect on a daily basis). In addition, through Web mining they track clickstream patterns of customers shopping online.

4.

What does it mean to have “a single view of the customer”? How can it be accomplished?

Having a single view of the customer means treating the customer as a single entity across whichever channels the customer utilizes. Shopping channels include brick-and-mortar, television, catalog, and e-commerce (through computers and mobile devices). Achieving this single view helps to better focus marketing efforts and drive increased sales.

5.

What type of analytics help did Cabela’s get from their efforts? Can you think of any other potential benefits of analytics for large-scale retailers like Cabela’s?

Cabela’s has long relied on SAS statistics and data mining tools to help analyze the data it gathers from sales transactions, market research, and demographic data associated with its large database of customers. Using SAS data mining tools, Cabela’s analysts create predictive models to optimize customer selection for all customer contacts. Cabela’s uses these prediction scores to maximize marketing spending across channels and within each customer’s personal contact strategy.

These efforts have allowed Cabela’s to continue its growth in a profitable manner.

In addition, dismantling the information silos, and integration of SAS and Teradata, enabled them to create “a holistic view of the customer.” Since this works so well for the sales/customer side of the business, it could also work in other areas as well. Supply chain is one example. Analytics could help produce a “holistic view of the vendor” as well.

6.

What was the reason for Cabela’s to bring together SAS and Teradata, the two leading vendors in the analytics marketplace?

Cabela’s was already using both for different elements of their business. Each of the two systems was producing actionable analysis of data. But by being separate, too much time was required to construct data marts, bringing together disparate data sources and keeping statisticians from working on analytics. Now, with the integration of the two systems, statisticians can leverage the power of SAS using the Teradata warehouse as one source of information.

7.

What is in-database analytics, and why would you need it?

In-database analytics refers to the practice of applying analytics directly to a database or data warehouse rather than the traditional practice of first transforming into the analytics application’s data format. The time it takes to

5 .

transform production data into a data warehouse format can be very long. Indatabase analytics eliminates this need.

Section 4.2 Review Questions 1.

Define data mining. Why are there many different names and definitions for data mining?

Data mining is the process through which previously unknown patterns in data were discovered. Another definition would be “a process that uses statistical, mathematical, and artificial learning techniques to extract and identify useful information and subsequent knowledge from large sets of data.” This includes most types of automated data analysis. A third definition: Data mining is the process of finding mathematical patterns from (usually) large sets of data; these can be rules, affinities, correlations, trends, or prediction models.

Data mining has many definitions because it’s been stretched beyond those limits by some software vendors to include most forms of data analysis in order to increase sales using the popularity of data mining.

2.

What recent factors have increased the popularity of data mining?

Following are some of the most pronounced reasons:

3.

•

More intense competition at the global scale driven by customers’ everchanging needs and wants in an increasingly saturated marketplace.

•

General recognition of the untapped value hidden in large data sources.

•

Consolidation and integration of database records, which enables a single view of customers, vendors, transactions, etc.

•

Consolidation of databases and other data repositories into a single location in the form of a data warehouse.

•

The exponential increase in data processing and storage technologies.

•

Significant reduction in the cost of hardware and software for data storage and processing.

•

Movement toward the de-massification (conversion of information resources into nonphysical form) of business practices.

Is data mining a new discipline? Explain.

Although the term data mining is relatively new, the ideas behind it are not. Many of the techniques used in data mining have their roots in traditional statistical 6 .

analysis and artificial intelligence work done since the early part of the 1980s.

New or increased use of data mining applications makes it seem like data mining is a new discipline.

In general, data mining seeks to identify four major types of patterns: Associations, Predictions, Clusters and Sequential relationships. These types of patterns have been manually extracted from data by humans for centuries, but the increasing volume of data in modern times has created a need for more automatic approaches. As datasets have grown in size and complexity, direct manual data analysis has increasingly been augmented with indirect, automatic data processing tools that use sophisticated methodologies, methods, and algorithms. The manifestation of such evolution of automated and semiautomated means of processing large datasets is now commonly referred to as data mining.

4.

What are some major data mining methods and algorithms?

Generally speaking, data mining tasks can be classified into three main categories: prediction, association, and clustering. Based on the way in which the patterns are extracted from the historical data, the learning algorithms of data mining methods can be classified as either supervised or unsupervised. With supervised learning algorithms, the training data includes both the descriptive attributes (i.e., independent variables or decision variables) as well as the class attribute (i.e., output variable or result variable). In contrast, with unsupervised learning the training data includes only the descriptive attributes. Figure 4.3 (p. 157) shows a simple taxonomy for data mining tasks, along with the learning methods, and popular algorithms for each of the data mining tasks.

5.

What are the key differences between the major data mining methods?

Prediction: the act of telling about the future. It differs from simple guessing by taking into account the experiences, opinions, and other relevant information in conducting the task of foretelling. A term that is commonly associated with prediction is forecasting. Even though many believe that these two terms are synonymous, there is a subtle but critical difference between the two. Whereas prediction is largely experience and opinion based, forecasting is data and model based. That is, in order of increasing reliability, one might list the relevant terms as guessing, predicting, and forecasting, respectively. In data mining terminology, prediction and forecasting are used synonymously, and the term prediction is used as the common representation of the act.

Classification: analyzing the historical behavior of groups of entities with similar characteristics, to predict the future behavior of a new entity from its similarity to those groups Clustering: finding groups of entities with similar characteristics Association: establishing relationships among items that occur together 7 .

Sequence discovery: finding time-based associations Visualization: presenting results obtained through one or more of the other methods Regression: a statistical estimation technique based on fitting a curve defined by a mathematical equation of known type but unknown parameters to existing data Forecasting: estimating a future data value based on past data values.

Section 4.3 Review Questions 1.

What are the major application areas for data mining?

Applications are listed near the beginning of this section (pp. 160-161): CRM, banking, retailing and logistics, manufacturing and production, brokerage and securities trading, insurance, computer hardware and software, government and defense, travel, healthcare, medicine, entertainment, homeland security and law enforcement, and sports.

2.

Identify at least five specific applications of data mining and list five common characteristics of these applications.

This question expands on the prior question by asking for common characteristics.

Several such applications and their characteristics are listed on pp. 160-161.

3.

What do you think is the most prominent application area for data mining? Why? Students’ answers will differ depending on which of the applications (most likely banking, retailing and logistics, manufacturing and production, government, healthcare, medicine, or homeland security) they think is most in need of greater certainty. Their reasons for selection should relate to the application area’s need for better certainty and the ability to pay for the investments in data mining.

4.

Can you think of other application areas for data mining not discussed in this section? Explain.

Students should be able to identify an area that can benefit from greater prediction or certainty. Answers will vary depending on their creativity.

8 .

Section 4.4 Review Questions 1.

What are the major data mining processes?

Similar to other information systems initiatives, a data mining project must follow a systematic project management process to be successful. Several data mining processes have been proposed: CRISP-DM, SEMMA, and KDD.

2.

Why do you think the early phases (understanding of the business and understanding of the data) take the longest in data mining projects?

Students should explain that the early steps are the most unstructured phases because they involve learning. Those phases (learning/understanding) cannot be automated. Extra time and effort are needed upfront because any mistake in understanding the business or data will most likely result in a failed BI project.

3.

List and briefly define the phases in the CRISP-DM process.

CRISP-DM provides a systematic and orderly way to conduct data mining projects. This process has six steps. First, an understanding of the data and an understanding of the business issues to be addressed are developed concurrently.

Next, data are prepared for modeling; are modeled; model results are evaluated; and the models can be employed for regular use.

4.

What are the main data preprocessing steps? Briefly describe each step and provide relevant examples.

Data preprocessing is essential to any successful data mining study. Good data leads to good information; good information leads to good decisions. Data preprocessing includes four main steps (listed in Table 4.1 on page 167): data consolidation: access, collect, select and filter data data cleaning: handle missing data, reduce noise, fix errors data transformation: normalize the data, aggregate data, construct new attributes data reduction: reduce number of attributes and records; balance skewed data

5.

How does CRISP-DM differ from SEMMA? The main difference between CRISP-DM and SEMMA is that CRISP-DM takes a more comprehensive approach—including understanding of the business and the relevant data—to data mining projects, whereas SEMMA implicitly assumes that the data mining project’s goals and objectives along with the appropriate data sources have been identified and understood.

9 .

Section 4.5 Review Questions 1.

Identify at least three of the main data mining methods.

Classification learns patterns from past data (a set of information—traits, variables, features—on characteristics of the previously labeled items, objects, or events) in order to place new instances (with unknown labels) into their respective groups or classes. The objective of classification is to analyze the historical data stored in a database and automatically generate a model that can predict future behavior.

Cluster analysis is an exploratory data analysis tool for solving classification problems. The objective is to sort cases (e.g., people, things, events) into groups, or clusters, so that the degree of association is strong among members of the same cluster and weak among members of different clusters.

Association rule mining is a popular data mining method that is commonly used as an example to explain what data mining is and what it can do to a technologically less savvy audience. Association rule mining aims to find interesting relationships (affinities) between variables (items) in large databases.

2.

Give examples of situations in which classification would be an appropriate data mining technique. Give examples of situations in which regression would be an appropriate data mining technique.

Students’ answers will differ, but should be based on the following issues.

Classification is for prediction that can be based on historical data and relationships, such as predicting the weather, product demand, or a student’s success in a university. If what is being predicted is a class label (e.g., “sunny,” “rainy,” or “cloudy”) the prediction problem is called a classification, whereas if it is a numeric value (e.g., temperature such as 68°F), the prediction problem is called a regression.

3.

List and briefly define at least two classification techniques.

•

Decision tree analysis. Decision tree analysis (a machine-learning technique) is arguably the most popular classification technique in the data mining arena.

•

Statistical analysis. Statistical classification techniques include logistic regression and discriminant analysis, both of which make the assumptions that the relationships between the input and output variables are linear in nature, the data is normally distributed, and the variables are not correlated and are independent of each other.

•

Case-based reasoning. This approach uses historical cases to recognize commonalities in order to assign a new case into the most probable category.

10 .

4.

5.

•

Bayesian classifiers. This approach uses probability theory to build classification models based on the past occurrences that are capable of placing a new instance into a most probable class (or category).

•

Genetic algorithms. The use of the analogy of natural evolution to build directed search-based mechanisms to classify data samples.

•

Rough sets. This method takes into account the partial membership of class labels to predefined categories in building models (collection of rules) for classification problems.

What are some of the criteria for comparing and selecting the best classification technique?

•

The amount and availability of historical data

•

The types of data, categorical, interval, ration, etc.

•

What is being predicted—class or numeric value

•

The purpose or objective

Briefly describe the general algorithm used in decision trees.

A general algorithm for building a decision tree is as follows: 1. Create a root node and assign all of the training data to it.

2. Select the best splitting attribute.

3. Add a branch to the root node for each value of the split. Split the data into mutually exclusive (non-overlapping) subsets along the lines of the specific split and mode to the branches.

4. Repeat steps 2 and 3 for each and every leaf node until the stopping criteria is reached (e.g., the node is dominated by a single class label).

6.

Define Gini index. What does it measure?

The Gini index and information gain (entropy) are two popular ways to determine branching choices in a decision tree. The Gini index measures the purity of a sample. If everything in a sample belongs to one class, the Gini index value is zero.

7.

Give examples of situations in which cluster analysis would be an appropriate data mining technique.

Cluster algorithms are used when the data records do not have predefined class identifiers (i.e., it is not known to what class a particular record belongs).

11 .

8.

What is the major difference between cluster analysis and classification?

Classification methods learn from previous examples containing inputs and the resulting class labels, and once properly trained they are able to classify future cases. Clustering partitions pattern records into natural segments or clusters.

9.

What are some of the methods for cluster analysis?

The most commonly used clustering algorithms are k-means and self-organizing maps.

10.

Give examples of situations in which association would be an appropriate data mining technique.

Association rule mining is appropriate to use when the objective is to discover two or more items (or events or concepts) that go together. Students’ answers will differ.

Section 4.6 Review Questions 1.

What are the most popular commercial data mining tools?

Examples of these vendors include IBM (IBM SPSS Modeler), SAS (Enterprise Miner), StatSoft (Statistica Data Miner), KXEN (Infinite Insight), Salford (CART, MARS, TreeNet, RandomForest), Angoss (KnowledgeSTUDIO, KnowledgeSeeker), and Megaputer (PolyAnalyst). Most of the more popular tools are developed by the largest statistical software companies (SPSS, SAS, and StatSoft).

2.

Why do you think the most popular tools are developed by statistics companies?

Data mining techniques involve the use of statistical analysis and modeling. So it’s a natural extension of their business offerings.

3.

What are the most popular free data mining tools?

Probably the most popular free and open source data mining tool is Weka. Others include RapidMiner and Microsoft’s SQL Server.

4.

What are the main differences between commercial and free data mining software tools?

The main difference between commercial tools, such as Enterprise Miner and Statistica, and free tools, such as Weka and RapidMiner, is computational efficiency. The same data mining task involving a rather large dataset may take a

12 .

whole lot longer to complete with the free software, and in some cases it may not even be feasible (i.e., crashing due to the inefficient use of computer memory).

5.

What would be your top five selection criteria for a data mining tool? Explain.

Students’ answers will differ. Criteria they are likely to mention include cost, user-interface, ease-of-use, computational efficiency, hardware compatibility, type of business problem, vendor support, and vendor reputation.

Section 4.7 Review Questions 1.

What are the privacy issues in data mining?

Data that is collected, stored, and analyzed in data mining often contains information about real people. This includes identification, demographic, financial, personal, and behavioral information. Most of these data can be accessed through some third-party data providers. In order to maintain the privacy and protection of individuals’ rights, data mining professionals have ethical (and often legal) obligations.

2.

How do you think the discussion between privacy and data mining will progress?

Why? As technology advances and more information about people becomes easier to get, the privacy debate will adjust accordingly. People’s expectations about privacy will become tempered by their desires for the benefits of data mining, from individualized customer service to higher security. As with all issues of social import, the privacy issue will include social discourse, legal and legislative decisions, and corporate decisions. The fact that companies often choose to selfregulate (e.g., by ensuring their data is de-identified) implies that we may as a society be able to find a happy medium between privacy and data mining.

(Answers will vary by student.)

3.

What are the most common myths about data mining?

•

Data mining provides instant, crystal-ball predictions.

•

Data mining is not yet viable for business applications.

•

Data mining requires a separate, dedicated database.

•

Only those with advanced degrees can do data mining.

•

Data mining is only for large firms that have lots of customer data.

13 .

4.

What do you think are the reasons for these myths about data mining?

Students’ answers will differ. Some answers might relate to fear of analytics, fear of the unknown, or fear of looking dumb.

5.

What are the most common data mining mistakes/blunders? How can they be minimized and/or eliminated?

•

Selecting the wrong problem for data mining

•

Ignoring what your sponsor thinks data mining is and what it really can and cannot do

•

Leaving insufficient time for data preparation. It takes more effort than one often expects

•

Looking only at aggregated results and not at individual records

•

Being sloppy about keeping track of the mining procedure and results

•

Ignoring suspicious findings and quickly moving on

•

Running mining algorithms repeatedly and blindly. (It is important to think hard enough about the next stage of data analysis. Data mining is a very hands-on activity.)

•

Believing everything you are told about data

•

Believing everything you are told about your own data mining analysis

•

Measuring your results differently from the way your sponsor measures them

Ways to minimize these risks are basically the reverse of these items.

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 4.1: Smarter Insurance: Infinity P&C Improves Customer Service and Combats Fraud with Predictive Analytics 1.

How did Infinity P&C improve customer service with data mining?

One out of five claims is fraudulent. Rather than putting all five customers through an investigatory process, SPSS helps Infinity ‘fast-track’ four of them and close their cases within a matter of days. This results in much happier customers, 14 .

contributes to a more efficient workflow with improved cycle times, and improves retention due to an overall better claims experience.

2.

What were the challenges, the proposed solution, and the obtained results?

Infinity P&C faces a significant challenge in recognizing fraudulent claims. Fraud represents a $20 billion exposure to the insurance industry and in certain venues could be an element in around 40 percent of claims.

The solution involved using IBM SPSS predictive analytics tools. Based on “red flag” claims, they used this to develop rules for rating and identifying potential frauds. A key benefit of the IBM SPSS system is its ability to continually analyze and score these claims, which helps ensure that we get the claim to the right adjuster at the right time.

As a result of implementing the IBM SPSS analytics tools, Infinity P&C has doubled the accuracy of its fraud identification, contributing to a return on investment of 403 percent per a Nucleus Research study. With SPSS, Infinity P&C has reduced SIU referral time from an average of 45–60 days to approximately 1–3 days. Predictive analytics also helped with subrogation, the process of collecting damages from the at-fault driver’s insurance company.

3.

What was their implementation strategy? Why is it important to produce results as early as possible in data mining studies?

The Special Investigative Unit (SIU) started with low hanging fruit, applying data mining for constructing discovery rules from “red flag” claims. They then leveraged the company’s credit-rating approach toward coming up with a similar scoring mechanism of claims (higher scores indicate greater chance of fraud).

They added rules to the SPSS rule-base to help flag potential subrogation claims.

This incremental approach to implementation works well with a tool like SPSS.

Application Case 4.2: Harnessing Analytics to Combat Crime: Predictive Analytics Helps Memphis Police Department Pinpoint Crime and Focus Police Resources 1.

How did the Memphis Police Department use data mining to better combat crime?

They started mining MPD’s crime data banks to help zero in on where and when criminals were hitting hardest. This began as a pilot program called Operation Blue CRUSH, or Crime Reduction Utilizing Statistical History. Their use of data mining enabled them to focus police resources intelligently by putting them in the right place, on the right day, at the right time. Today, the MPD is continuing to explore new ways to exploit statistical analysis in its crime-fighting mission.

2.

What were the challenges, the proposed solution, and the obtained results?

15 .

Crime across the metro area was surging, and city leaders were growing impatient. The solution, a project called Blue CRUSH, involves IBM SPSS Modeler, which enables officers to unlock the intelligence hidden in the department’s huge digital library of crime records and police reports going back nearly a decade. This has put a serious dent in Memphis area crime. Since the program was launched, the number of Part One crimes—a category of serious offenses including homicide, rape, aggravated assault, auto theft, and larceny— has plummeted, dropping 27 percent from 2006 to 2010.

Application Case 4.3: A Mine on Terrorist Funding 1.

How can data mining be used to fight terrorism? Comment on what else can be done beyond what is covered in this short application case.

The application case discusses use of data mining to detect money laundering and other forms of terrorist financing. Other applications could be to track the behavior and movement of potential terrorists, as well as text mining emails, blogs, and social media threads.

2.

Do you think data mining, while essential for fighting terrorist cells, also jeopardizes individuals’ rights of privacy?

Yes, because it inevitably involves tracking personal and financial data of individuals. (As an opinion question, students’ answers will vary.)

Application Case 4.4: Data Mining in Cancer Research 1.

How can data mining be used for ultimately curing illnesses like cancer?

Even though cancer research has traditionally been clinical and biological in nature, in recent years data-driven analytic studies have become a common complement. In medical domains where data- and analytics-driven research have been applied successfully, novel research directions have been identified to further advance the clinical and biological studies. Using data mining techniques, researchers are able to identify novel patterns, paving the road toward a cancerfree society.

2.

What do you think are the promises and major challenges for data miners in contributing to medical and biological research endeavors?

According to the American Cancer Society, half of all men and one-third of all women in the United States will develop cancer during their lifetimes; approximately 1.5 million new cancer cases will be diagnosed in 2013. Cancer is the second most common cause of death in the United States and in the world, 16 .

exceeded only by cardiovascular disease. Data mining shows tremendous promise for helping to understand cancer, leading to better treatment and saved lives. Data mining is not meant to replace medical professionals and researchers, but to complement their invaluable efforts to provide data-driven new research directions and to ultimately save more lives. Without the cooperation and feedback from the medical experts, data mining results are not of much use. The patterns found via data mining methods should be evaluated by medical professionals who have years of experience in the problem domain to decide whether they are logical, actionable, and novel to warrant new research directions.

Application Case 4.5: 2degrees Gets a 1275 Percent Boost in Churn Identification 1.

What does 2degrees do? Why is it important for 2degrees to accurately identify churn?

2degrees is New Zealand’s fastest growing mobile telecommunications company.

Customer churn (customers leaving) is an all-too-common problem in the mobile telecommunications industry. So, it makes sense that they would be interested in identifying customers most at risk of churning.

2.

What were the challenges, the proposed solution, and the obtained results?

Their main challenge was to identify the most likely churn customers. For this, they used 11Ants Customer Churn Analyzer. This tool allowed them to get up and running quickly and very economically. They could test the water and determine what the ROI was likely to be for predictive analytics. A carefully controlled experiment was run over a period of 3 months. Customers identified as churners were 1275 percent more likely to be churners than customers chosen at random.

The benefits to 2degrees and their customers included identification and appropriateness, so that likely churners would get more intense marketing than unlikely churners.

3.

How can data mining help in identifying customer churn? How do some companies do it without using data mining tools and techniques?

Identifying customers most at risk of churning is done by analyzing data such as time on network, days since last top-up, activation channel, whether the customer ported their number or not, customer plan, and outbound calling behaviors over the preceding 90 days. Companies without data mining cannot score customers as often; previously 2degrees would do this once per month, but now they can do it daily if they want.

Application Case 4.6: Data Mining Goes to Hollywood: Predicting Financial Success of Movies

17 .

1.

Why is it important for many Hollywood professionals to predict the financial success of movies?

The movie industry is the “land of hunches and wild guesses” due to the difficulty associated with forecasting product demand, making the movie business in Hollywood a risky endeavor. If Hollywood could better predict financial success, this would mitigate some of the financial risk.

2.

How can data mining be used for predicting financial success of movies before the start of their production process?

The way Sharda and Delen did it was to use data from movies between 1998 and 2005 as training data, and movies of 2006 as test data. They applied individual and ensemble prediction models, and were able to identify significant variables impacting financial success. They also showed that by using sensitivity analysis, decision makers can predict with fairly high accuracy how much value a specific actor (or a specific release date, or the addition of more technical effects, etc.) brings to the financial success of a film, making the underlying system an invaluable decision aid.

3.

How do you think Hollywood did, and perhaps still is performing, this task without the help of data mining tools and techniques?

Most is done by gut feel and trial-and-error. This may keep the movie business as a financially risky endeavor, but also allows for creativity. Sometimes uncertainty is a good thing.

Application Case 4.7: Predicting Customer Buying Patterns—The Target Story 1.

What do you think about data mining and its implication for privacy? What is the threshold between discovery of knowledge and infringement of privacy?

There is a tradeoff between knowledge discovery and privacy rights. Retailers should be sensitive about this when targeting their advertising based on data mining results, especially regarding topics that could be embarrassing to their customers. Otherwise they risk offending these customers, which could hurt their bottom line. (Answers will vary by student.)

2.

Did Target go too far? Did it do anything illegal? What do you think Target should have done? What do you think Target should do next (quit these types of practices)?

Target might have made a tactical mistake, but they certainly didn’t do anything illegal. They did not use any information that violates customer privacy; rather, they used transactional data that most every other retail chain is collecting and 18 .

storing (and perhaps analyzing) about their customers. Indeed, even the father apologized when realizing his daughter was actually pregnant. The fact is, we live in a world of massive data, and we are all as consumers leaving traces of our buying behavior for anyone to see. (Answers will vary by student.)

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION 1.

  

Define data mining. Why are there many names and definitions for data mining?

Data mining is the process through which previously unknown patterns in data were discovered. Another definition would be “a process that uses statistical, mathematical, and artificial learning techniques to extract and identify useful information and subsequent knowledge from large sets of data.” This includes most types of automated data analysis. A third definition: Data mining is the process of finding mathematical patterns from (usually) large sets of data; these can be rules, affinities, correlations, trends, or prediction models.

Data mining has many definitions because it’s been stretched beyond those limits by some software vendors to include most forms of data analysis in order to increase sales using the popularity of data mining.

2.

What are the main reasons for the recent popularity of data mining?

Following are some of the most pronounced reasons: •

More intense competition at the global scale driven by customers’ everchanging needs and wants in an increasingly saturated marketplace

•

General recognition of the untapped value hidden in large data sources

•

Consolidation and integration of database records, which enables a single view of customers, vendors, transactions, etc.

•

Consolidation of databases and other data repositories into a single location in the form of a data warehouse

•

The exponential increase in data processing and storage technologies

•

Significant reduction in the cost of hardware and software for data storage and processing

•

Movement toward the de-massification (conversion of information resources into nonphysical form) of business practices

19 .

3.

Discuss what an organization should consider before making a decision to purchase data mining software.

Before making a decision to purchase data mining software, organizations should consider the standard criteria to use when investing in any major software: cost/benefit analysis, people with the expertise to use the software and perform the analyses, availability of historical data, a business need for the data mining software.

4.

Distinguish data mining from other analytical tools and techniques.

Students can view the answer in Figure 4.1 (p. 152) which shows that data mining is a composite or blend of multiple disciplines or analytical tools and techniques.

5.

Discuss the main data mining methods. What are the fundamental differences among them?

Three broad categories of data mining methods are prediction (classification or regression), clustering, and association.

Prediction is the act of telling about the future. It differs from simple guessing by taking into account the experiences, opinions, and other relevant information in conducting the task of foretelling. A term that is commonly associated with prediction is forecasting. Even though many believe that these two terms are synonymous, there is a subtle but critical difference between the two. Whereas prediction is largely experience and opinion based, forecasting is data and model based.

Classification is analyzing the historical behavior of groups of entities with similar characteristics, to predict the future behavior of a new entity from its similarity to those groups.

Clustering is finding groups of entities with similar characteristics.

Association is establishing relationships among items that occur together.

The fundamental differences are: •

Prediction (classification or regression) predicts future cases or conditions based on historical data.

•

Clustering partitions pattern records into natural segments or clusters.

Each segment’s members share similar characteristics.

•

Association is used to discover two or more items (or events or concepts) that go together.

20 .

6.

What are the main data mining application areas? Discuss the commonalities of these areas that make them a prospect for data mining studies.

Applications are listed near the beginning of section 4.3: CRM, banking, retailing and logistics, manufacturing and production, brokerage and securities trading, insurance, computer hardware and software, government and defense, travel, healthcare, medicine, entertainment, homeland security and law enforcement, and sports.

The commonalities are the need for predictions and forecasting for planning purposes and to support decision making.

7.

Why do we need a standardized data mining process? What are the most commonly used data mining processes?

In order to systematically carry out data mining projects, a general process is usually followed. Similar to other information systems initiatives, a data mining project must follow a systematic project management process to be successful.

Several data mining processes have been proposed: CRISP-DM, SEMMA, and KDD.

8.

Discuss the differences between the two most commonly used data mining process.

The main difference between CRISP-DM and SEMMA is that CRISP-DM takes a more comprehensive approach—including understanding of the business and the relevant data—to data mining projects, whereas SEMMA implicitly assumes that the data mining project’s goals and objectives along with the appropriate data sources have been identified and understood.

9.

Are data mining processes a mere sequential set of activities? Explain.

No. Even though these steps are sequential in nature, there is usually a great deal of backtracking. Because data mining is driven by experience and experimentation, depending on the problem/situation and the knowledge/experience of the analyst, the whole process can be very iterative (i.e., one should expect to go back and forth through the steps quite a few times) and time consuming. Because latter steps are built on the outcome of the former ones, one should pay extra attention to the earlier steps in order not to put the whole study on an incorrect path from the onset.

10.

Why do we need data preprocessing? What are the main tasks and relevant techniques used in data preprocessing?

Data preprocessing is essential to any successful data mining study. Good data leads to good information; good information leads to good decisions. Data preprocessing includes four main steps (listed in Table 4.1 on page 167):

21 .

data consolidation: access, collect, select and filter data data cleaning: handle missing data, reduce noise, fix errors data transformation: normalize the data, aggregate data, construct new attributes data reduction: reduce number of attributes and records; balance skewed data 11.

Discuss the reasoning behind the assessment of classification models.

The model-building step also encompasses the assessment and comparative analysis of the various models built. Because there is not a universally known best method or algorithm for a data mining task, one should use a variety of viable model types along with a well-defined experimentation and assessment strategy to identify the “best” method for a given purpose.

12.

What is the main difference between classification and clustering? Explain using concrete examples.

Classification learns patterns from past data (a set of information—traits, variables, features—on characteristics of the previously labeled items, objects, or events) in order to place new instances (with unknown labels) into their respective groups or classes. The objective of classification is to analyze the historical data stored in a database and automatically generate a model that can predict future behavior. Classifying customer-types as likely to buy or not buy is an example.

Cluster analysis is an exploratory data analysis tool for solving classification problems. The objective is to sort cases (e.g., people, things, events) into groups, or clusters, so that the degree of association is strong among members of the same cluster and weak among members of different clusters. Customers can be grouped according to demographics.

13.

Moving beyond the chapter discussion, where else can association be used?

Students’ answers will vary.

14.

What are the privacy issues with data mining? Do you think they are substantiated?

Data that is collected, stored, and analyzed in data mining often contains information about real people. This includes identification, demographic, financial, personal, and behavioral information. Most of these data can be accessed through some third-party data providers. In order to maintain the privacy and protection of individuals’ rights, data mining professionals have ethical (and often legal) obligations. As time goes on, this will continue to be a public debate.

As technology advances and more information about people becomes easier to get, the privacy debate will adjust accordingly. People’s expectations about 22 .

privacy will become tempered by their desires for the benefits of data mining, from individualized customer service to higher security. As with all issues of social import, the privacy issue will include social discourse, legal and legislative decisions, and corporate decisions. The fact that companies often choose to selfregulate (e.g., by ensuring their data is de-identified) implies that we may as a society be able to find a happy medium between privacy and data mining.

15.

What are the most common myths and mistakes about data mining?

•

Data mining provides instant, crystal-ball predictions.

•

Data mining is not yet viable for business applications.

•

Data mining requires a separate, dedicated database.

•

Only those with advanced degrees can do data mining.

•

Data mining is only for large firms that have lots of customer data.

23 .

CHAPTER

5

Text and Web Analytics

Learning Objectives for Chapter 5 ▪

Describe text analytics and understand the need for text mining

▪

Differentiate among text analytics, text mining, and data mining

▪

Understand the different application areas for text mining

▪

Know the process of carrying out a text mining project

▪

Appreciate the different methods to introduce structure to text-based data

▪

Describe sentiment analysis

▪

Develop familiarity with popular applications of sentiment analysis

▪

Learn the common methods for sentiment analysis

▪

Become familiar with speech analytics as it relates to sentiment analysis

CHAPTER OVERVIEW This chapter provides a comprehensive overview of text analytics/mining and Web analytics/mining along with their popular application areas such as search engines, sentiment analysis, and social network/media analytics. As we have been witnessing in the recent years, the unstructured data generated over the Internet of things (Web, sensor networks, RFID-enabled supply chain systems, surveillance networks, etc.) is increasing at an exponential pace, and there is no indication of its slowing down. This changing nature of data is forcing organizations to make text and Web analytics a critical part of their business intelligence/analytics infrastructure.

CHAPTER OUTLINE 5.1 OPENING VIGNETTE: MACHINE VERSUS MEN ON JEOPARDY!: THE STORY OF WATSON 1 .

A.

 Questions for the Opening Vignette WHAT WE CAN LEARN FROM THIS VIGNETTE

5.2

TEXT ANALYTICS AND TEXT MINING OVERVIEW ⧫ Technology Insights 5.1: Text Mining Lingo ⧫ Application Case 5.1: Text Mining for Patent Analysis  Section 5.2 Review Questions

5.3

NATURAL LANGUAGE PROCESSING ⧫ Application Case 5.2: Text Mining Improves Hong Kong Government’s Ability to Anticipate and Address Public Complaints  Section 5.3 Review Questions

5.4

TEXT MINING APPLICATIONS A. MARKETING APPLICATIONS B. SECURITY APPLICATIONS ⧫ Application Case 5.3: Mining for Lies C. BIOMEDICAL APPLICATIONS D. ACADEMIC APPLICATIONS ⧫ Application Case 5.4: Text Mining and Sentiment Analysis Helps Improve Customer Service Performance  Section 5.4 Review Questions

5.5

TEXT MINING PROCESS A. TASK 1: ESTABLISH THE CORPUS B. TASK 2: CREATE THE TERM–DOCUMENT MATRIX 1.

Representing the Indices 2.

Reducing the Dimensionality of the Matrix C. TASK 3: EXTRACT THE KNOWLEDGE 1.

Classification 2.

Clustering 3.

Association 4.

Trend Analysis ⧫ Application Case 5.5: Research Literature Survey with Text Mining  Section 5.5 Review Questions

5.6

SENTIMENT ANALYSIS ⧫ Application Case 5.6: Whirlpool Achieves Customer Loyalty and Product Success with Text Analytics A. SENTIMENT ANALYSIS APPLICATIONS 1.

Voice of the Customer (VOC) 2.

Voice of the Market (VOM) 3.

Voice of the Employee (VOE) 4.

Brand Management 2 .

B.

C. D. E.

F. G.

5.

Financial Markets 6.

Politics 7.

Government Intelligence 8.

Other Interesting Areas SENTIMENT ANALYSIS PROCESS 1.

Step 1: Sentiment Detection 2.

Step 2: N-P Polarity Classification 3.

Step 3: Target Identification 4.

Step 4: Collection and Aggregation METHODS FOR POLARITY IDENTIFICATION USING A LEXICON USING A COLLECTION OF TRAINING DOCUMENTS ⧫ Technology Insights 5.2: Large Textual Data Sets for Predictive Text Mining and Sentiment Analysis IDENTIFYING SEMANTIC ORIENTATION OF SENTENCES AND PHRASES IDENTIFYING SEMANTIC ORIENTATION OF DOCUMENT  Section 5.6 Review Questions

5.7

WEB MINING OVERVIEW A. WEB CONTENT AND WEB STRUCTURE MINING  Section 5.7 Review Questions

5.8

SEARCH ENGINES A. ANATOMY OF A SEARCH ENGINE B. 1. DEVELOPMENT CYCLE C. WEB CRAWLER D. DOCUMENT INDEXER 1.

Step 1: Preprocessing the Documents 2.

Step 2: Parsing the Documents 3.

Step 3: Creating the Term-by-Term Document Matrix E. 2. RESPONSE CYCLE F. QUERY ANALYZER G. DOCUMENT MATCHER/RANKER H. SEARCH ENGINE OPTIMIZATION ⧫ Technology Insights 5.3: Top 15 Most Popular Search Engines (March 2013) I. METHODS FOR SEARCH ENGINE OPTIMIZATION ⧫ Application Case 5.7: Understanding Why Customers Abandon Shopping Carts Results in a $10 Million Sales Increase  Section 5.8 Review Questions

5.9

WEB USAGE MINING (WEB ANALYTICS) A. WEB ANALYTICS TECHNOLOGIES

3 .

⧫

B. C. D. E. F.

Application Case 5.8: Allegro Boosts Online Click-Thru Rates by 500 Percent with Web Analysis WEB ANALYTICS METRICS WEBSITE USABILITY TRAFFIC SOURCES VISITOR PROFILES CONVERSION STATISTICS  Section 5.9 Review Questions

5.10

SOCIAL ANALYTICS A. SOCIAL NETWORK ANALYSIS 1.

Communication Networks 2.

Community Networks 3.

Criminal Networks 4.

Innovation Networks B. SOCIAL NETWORK ANALYSIS METRICS ⧫ Application Case 5.9: Social Network Analysis Helps Telecommunication Firms C. CONNECTIONS D. DISTRIBUTIONS E. SEGMENTATION F. SOCIAL MEDIA ANALYTICS G. HOW DO PEOPLE USE SOCIAL MEDIA? ⧫ Application Case 5.10: Measuring the Impact of Social Media at Lollapalooza H. MEASURING THE SOCIAL MEDIA IMPACT I. BEST PRACTICES IN SOCIAL MEDIA ANALYTICS 1.

Think of Measurement as a Guidance System, Not a Rating System 2.

Track the Elusive Sentiment 3.

Continuously Improve the Accuracy of Text Analysis 4.

Look at the Ripple Effect 5.

Look Beyond the Brand 6.

Identify Your Most Powerful Influencers 7.

Look Closely at the Accuracy of Your Analytic Tool 8.

Incorporate Social Media Intelligence into Planning ⧫ Application Case 5.11: eHarmony Uses Social Media to Help Take the Mystery out of Online Dating  Section 5.10 Review Questions Chapter Highlights Key Terms QUESTIONS FOR DISCUSSION Exercises Teradata Student Network (TSN) and Other Hands-on Exercises Team Assignments and Role-Playing Projects Internet Exercises 4 .

⧫ 

End of Chapter Application Case: Keeping Students on Track with Web and Predictive Analytics Questions for the Case

References

TEACHING TIPS/ADDITIONAL INFORMATION

       

Text analytics, text mining, and sentiment analysis are topics that students find especially interesting and (relatively) easier than data mining. They are able to relate to text mining applications. Students are familiar with social media outlets such as Facebook and Twitter, and often contribute sentiments of their own. The opening vignette discusses IBM’s Watson, which utilized text analytics (along with many other AI-related technologies) to win at Jeopardy!. Students can come up with examples of value-added text mining and sentiment analysis applications. It will be useful to distinguish between text mining and the broader topic of text analytics. This will be a good chapter to engage students in class discussions, and especially to relate the concepts of the technologies to the various application cases in the chapter. Students should also be able to discuss other possible applications, perhaps even within the university or college that they are attending. For example, how can text analytics be used to improve the student experience?

The chapter also gives a comprehensive overview of how search engines work and the methods used for ranking pages in response to user queries as well as the methods for search engine optimization. Students will enjoy learning about social analytics and social network analysis, and can relate it to other classes they may have taken in social sciences.

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 5.1 Review Questions 1.

What is Watson? What is special about it?

Watson is a question answering (QA) computer system developed by an IBM Research team and named after IBM’s first president as part of a project called DeepQA. What makes it special is that it is able to compete at the human champion level in real time on the TV quiz show, Jeopardy!; in fact, in 2011, it was able to defeat Ken Jennings, who held the record for the longest winning streak in the game. Like Deep Blue has done with chess, Watson is showing that computer systems are getting quite good at demonstrating human-like intelligence.

5 .

2.

What technologies were used in building Watson (both hardware and software)?

Watson is built on the DeepQA framework. The hardware for this system involves a massively parallel processing architecture. In terms of software, Watson uses a variety of AI-related QA technologies, including text mining, natural language processing, question classification and decomposition, automatic source acquisition and evaluation, entity and relation detection, logical form generation, and knowledge representation and reasoning.

3.

What are the innovative characteristics of DeepQA architecture that made Watson superior?

The DeepQA architecture involves massive parallelism, many experts, pervasive confidence estimation, and integration of the-latest-and-greatest in text analytics, involving both shallow and deep semantic knowledge. As implemented in Watson, DeepQA brings more than 100 different techniques for analyzing natural language, identifying sources, finding and generating hypotheses, finding and scoring evidence, and merging and ranking hypotheses. More important than any particular technique is the combination of overlapping approaches that can bring their strengths to bear and contribute to improvements in accuracy, confidence, and speed.

4.

Why did IBM spend all that time and money to build Watson? Where is the ROI? IBM’s goal was to advance computer science by exploring new ways for computer technology to affect science, business, and society. The techniques IBM developed with DeepQA and Watson are relevant in a wide variety of domains central to IBM’s mission. For example, IBM is currently working on a version of Watson to take on surmountable problems in healthcare and medicine. If successful, this could give IBM a distinct competitive advantage in this important technological application area.

Section 5.2 Review Questions 1. What is text analytics? How does it differ from text mining?

Text analytics is a concept that includes information retrieval (e.g., searching and identifying relevant documents for a given set of key terms) as well as information extraction, data mining, and Web mining. By contrast, text mining is primarily focused on discovering new and useful knowledge from textual data sources. The overarching goal for both text analytics and text mining is to turn unstructured textual data into actionable information through the application of natural language processing (NLP) and analytics. However, text analytics is a broader term because of its inclusion of information retrieval. You can think of text analytics as a combination of information retrieval plus text mining.

6 .

2. What is text mining? How does it differ from data mining?

Text mining is the application of data mining to unstructured, or less structured, text files. As the names indicate, text mining analyzes words; and data mining analyzes numeric data.

3.

Why is the popularity of text mining as a BI tool increasing?

Text mining as a BI is increasing because of the rapid growth in text data and availability of sophisticated BI tools. The benefits of text mining are obvious in the areas where very large amounts of textual data are being generated, such as law (court orders), academic research (research articles), finance (quarterly reports), medicine (discharge summaries), biology (molecular interactions), technology (patent files), and marketing (customer comments).

4.

What are some popular application areas of text mining?

•

Information extraction. Identification of key phrases and relationships within text by looking for predefined sequences in text via pattern matching.

•

Topic tracking. Based on a user profile and documents that a user views, text mining can predict other documents of interest to the user.

•

Summarization. Summarizing a document to save time on the part of the reader.

•

Categorization. Identifying the main themes of a document and then placing the document into a predefined set of categories based on those themes.

•

Clustering. Grouping similar documents without having a predefined set of categories.

•

Concept linking. Connects related documents by identifying their shared concepts and, by doing so, helps users find information that they perhaps would not have found using traditional search methods.

•

Question answering. Finding the best answer to a given question through knowledge-driven pattern matching.

7 .

Section 5.3 Review Questions 1.

What is natural language processing?

Natural language processing (NLP) is an important component of text mining and is a subfield of artificial intelligence and computational linguistics. It studies the problem of “understanding” the natural human language, with the view of converting depictions of human language (such as textual documents) into more formal representations (in the form of numeric and symbolic data) that are easier for computer programs to manipulate.

2.

How does NLP relate to text mining?

Text mining uses natural language processing to induce structure into the text collection and then uses data mining algorithms such as classification, clustering, association, and sequence discovery to extract knowledge from it.

3.

What are some of the benefits and challenges of NLP? NLP moves beyond syntax-driven text manipulation (which is often called “word counting”) to a true understanding and processing of natural language that considers grammatical and semantic constraints as well as the context. The challenges include: •

Part-of-speech tagging. It is difficult to mark up terms in a text as corresponding to a particular part of speech because the part of speech depends not only on the definition of the term but also on the context within which it is used.

•

Text segmentation. Some written languages, such as Chinese, Japanese, and Thai, do not have single-word boundaries.

•

Word sense disambiguation. Many words have more than one meaning.

Selecting the meaning that makes the most sense can only be accomplished by taking into account the context within which the word is used.

•

Syntactic ambiguity. The grammar for natural languages is ambiguous; that is, multiple possible sentence structures often need to be considered.

Choosing the most appropriate structure usually requires a fusion of semantic and contextual information.

•

Imperfect or irregular input. Foreign or regional accents and vocal impediments in speech and typographical or grammatical errors in texts make the processing of the language an even more difficult task.

8 .

•

4.

Speech acts. A sentence can often be considered an action by the speaker.

The sentence structure alone may not contain enough information to define this action.

What are the most common tasks addressed by NLP? Following are among the most popular tasks: •

Question answering

•

Automatic summarization

•

Natural language generation

•

Natural language understanding

•

Machine translation

•

Foreign language reading

•

Foreign language writing

•

Speech recognition

•

Text-to-speech

•

Text proofing

•

Optical character recognition

Section 5.4 Review Questions 1.

List and briefly discuss some of the text mining applications in marketing.

Text mining can be used to increase cross-selling and up-selling by analyzing the unstructured data generated by call centers.

Text mining has become invaluable for customer relationship management.

Companies can use text mining to analyze rich sets of unstructured text data, combined with the relevant structured data extracted from organizational databases, to predict customer perceptions and subsequent purchasing behavior.

2.

How can text mining be used in security and counterterrorism?

Students may use the introductory case in this answer.

9 .

In 2007, EUROPOL developed an integrated system capable of accessing, storing, and analyzing vast amounts of structured and unstructured data sources in order to track transnational organized crime.

Another security-related application of text mining is in the area of deception detection.

3.

What are some promising text mining applications in biomedicine?

As in any other experimental approach, it is necessary to analyze the vast amount of data in the context of previously known information about the biological entities under study. The literature is a particularly valuable source of information for experiment validation and interpretation. Therefore, the development of automated text mining tools to assist in such interpretation is one of the main challenges in current bioinformatics research.

Section 5.5 Review Questions 1.

What are the main steps in the text mining process?

See Figure 5.6 (p. 222). Text mining entails three tasks:

2.

•

Establish the Corpus: Collect and organize the domain-specific unstructured data

•

Create the Term–Document Matrix: Introduce structure to the corpus

•

Extract Knowledge: Discover novel patterns from the T-D matrix

What is the reason for normalizing word frequencies? What are the common methods for normalizing word frequencies?

The raw indices need to be normalized in order to have a more consistent TDM for further analysis. Common methods are log frequencies, binary frequencies, and inverse document frequencies.

3.

What is singular value decomposition? How is it used in text mining?

Singular value decomposition (SVD), which is closely related to principal components analysis, reduces the overall dimensionality of the input matrix (number of input documents by number of extracted terms) to a lower dimensional space, where each consecutive dimension represents the largest degree of variability (between words and documents) possible.

10 .

4.

What are the main knowledge extraction methods from corpus?

The main categories of knowledge extraction methods are classification, clustering, association, and trend analysis.

Section 5.6 Review Questions 1. What is sentiment analysis? How does it relate to text mining?

Sentiment analysis tries to answer the question, “What do people feel about a certain topic?” by digging into opinions of many using a variety of automated tools. It is also known as opinion mining, subjectivity analysis, and appraisal extraction.

Sentiment analysis shares many characteristics and techniques with text mining.

However, unlike text mining, which categorizes text by conceptual taxonomies of topics, sentiment classification generally deals with two classes (positive versus negative), a range of polarity (e.g., star ratings for movies), or a range in strength of opinion.

2.

What are the most popular application areas for sentiment analysis? Why? Customer relationship management (CRM) and customer experience management are popular “voice of the customer (VOC)” applications. Other application areas include “voice of the market (VOM)” and “voice of the employee (VOE).”

3.

What would be the expected benefits and beneficiaries of sentiment analysis in politics?

Opinions matter a great deal in politics. Because political discussions are dominated by quotes, sarcasm, and complex references to persons, organizations, and ideas, politics is one of the most difficult, and potentially fruitful, areas for sentiment analysis. By analyzing the sentiment on election forums, one may predict who is more likely to win or lose. Sentiment analysis can help understand what voters are thinking and can clarify a candidate’s position on issues.

Sentiment analysis can help political organizations, campaigns, and news analysts to better understand which issues and positions matter the most to voters. The technology was successfully applied by both parties to the 2008 and 2012 American presidential election campaigns.

4.

What are the main steps in carrying out sentiment analysis projects?

The first step when performing sentiment analysis of a text document is called sentiment detection, during which text data is differentiated between fact and opinion (objective vs. subjective). This is followed by negative-positive (N-P) polarity classification, where a subjective text item is classified on a bipolar 11 .

range. Following this comes target identification (identifying the person, product, event, etc. that the sentiment is about). Finally come collection and aggregation, in which the overall sentiment for the document is calculated based on the calculations of sentiments of individual phrases and words from the first three steps.

5.

What are the two common methods for polarity identification? Explain.

Polarity identification can be done via a lexicon (as a reference library) or by using a collection of training documents and inductive machine learning algorithms. The lexicon approach uses a catalog of words, their synonyms, and their meanings, combined with numerical ratings indicating the position on the NP polarity associated with these words. In this way, affective, emotional, and attitudinal phrases can be classified according to their degree of positivity or negativity. By contrast, the training-document approach uses statistical analysis and machine learning algorithms, such as neural networks, clustering approaches, and decision trees to ascertain the sentiment for a new text document based on patterns from previous “training” documents with assigned sentiment scores.

Section 5.7 Review Questions 1. What are some of the main challenges the Web poses for knowledge discovery?

•

The Web is too big for effective data mining.

•

The Web is too complex.

•

The Web is too dynamic.

•

The Web is not specific to a domain.

•

The Web has everything.

2. What is Web mining? How does it differ from regular data mining or text mining?

Web mining is the discovery and analysis of interesting and useful information from the Web and about the Web, usually through Web-based tools. Text mining is less structured because it’s based on words instead of numeric data.

3. What are the three main areas of Web mining?

The three main areas of Web mining are Web content mining, Web structure mining, and Web usage (or activity) mining.

4. What is Web content mining? How can it be used for competitive advantage?

12 .

Web content mining refers to the extraction of useful information from Web pages. The documents may be extracted in some machine-readable format so that automated techniques can generate some information about the Web pages.

Collecting and mining Web content can be used for competitive intelligence (collecting intelligence about competitors’ products, services, and customers), which can give your organization a competitive advantage.

5. What is Web structure mining? How does it differ from Web content mining?

Web structure mining is the process of extracting useful information from the links embedded in Web documents. By contrast, Web content mining involves analysis of the specific textual content of web pages. So, Web structure mining is more related to navigation through a website, whereas Web content mining is more related to text mining and the document hierarchy of a particular web page.

Section 5.8 Review Questions 1. What is a search engine? Why are they important for today’s businesses?

A search engine is a software program that searches for documents (Internet sites or files) based on the keywords (individual words, multi-word terms, or a complete sentence) that users have provided that have to do with the subject of their inquiry. This is the most prominent type of information retrieval system for finding relevant content on the Web. Search engines have become the centerpiece of most Internet-based transactions and other activities. Because people use them extensively to learn about products and services, it is very important for companies to have prominent visibility on the Web; hence the major effort of companies to enhance their search engine optimization (SEO).

2. What is a web crawler? What is it used for? How does it work?

A Web crawler (also called a spider or a Web spider) is a piece of software that systematically browses (crawls through) the World Wide Web for the purpose of finding and fetching Web pages. It starts with a list of “seed” URLs, goes to the pages of those URLs, and then follows each page’s hyperlinks, adding them to the search engine’s database. Thus, the Web crawler navigates through the Web in order to construct the database of websites.

3. What is “search engine optimization”? Who benefits from it?

Search engine optimization (SEO) is the intentional activity of affecting the visibility of an e-commerce site or a website in a search engine’s natural (unpaid or organic) search results. It involves editing a page’s content, HTML, metadata, and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines. In addition, SEO efforts include promoting a site to increase its number of inbound links. SEO 13 .

primarily benefits companies with e-commerce sites by making their pages appear toward the top of search engine lists when users query.

4. What things can help Web pages rank higher in the search engine results?

Cross-linking between pages of the same website to provide more links to the most important pages may improve its visibility. Writing content that includes frequently searched keyword phrases, so as to be relevant to a wide variety of search queries, will tend to increase traffic. Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a Web page’s metadata, including the title tag and metadescription, will tend to improve the relevancy of a site’s search listings, thus increasing traffic. URL normalization of Web pages so that they are accessible via multiple URLs. Using canonical link elements and redirects can help make sure links to different versions of the URL all count toward the page’s link popularity score.

Section 5.9 Review Questions 1. What are the three types of data generated through Web page visits?

• • •

Automatically generated data stored in server access logs, referrer logs, agent logs, and client-side cookies User profiles Metadata, such as page attributes, content attributes, and usage data.

2. What is clickstream analysis? What is it used for?

Analysis of the information collected by Web servers can help us better understand user behavior. Analysis of this data is often called clickstream analysis. By using the data and text mining techniques, a company might be able to discern interesting patterns from the clickstreams.

3. What are the main applications of Web mining?

• • • • • •

Determine the lifetime value of clients.

Design cross-marketing strategies across products.

Evaluate promotional campaigns.

Target electronic ads and coupons at user groups based on user access patterns.

Predict user behavior based on previously learned rules and users’ profiles.

Present dynamic information to users based on their interests and profiles.

14 .

4. What are commonly used Web analytics metrics? What is the importance of metrics?

There are four main categories of Web analytic metrics: • • • •

Website usability: How were they using my website? These involve page views, time on site, downloads, click map, and click paths.

Traffic sources: Where did they come from? These include referral websites, search engines, direct, offline campaigns, and online campaigns.

Visitor profiles: What do my visitors look like? These include keywords, content groupings, geography, time of day, and landing page profiles.

Conversion statistics: What does all this mean for the business? Metrics include new visitors, returning visitors, leads, sales/conversions, and abandonments.

These metrics are important because they provide access to a lot of valuable marketing data, which can be leveraged for better insights to grow your business and better document your ROI. The insight and intelligence gained from Web analytics can be used to effectively manage the marketing efforts of an organization and its various products or services.

Section 5.10 Review Questions 1.

What is meant by social analytics? Why is it an important business topic?

From a philosophical perspective, social analytics focuses on a theoretical object called a “socius,” a kind of “commonness” that is neither a universal account nor a communality shared by every member of a body. Thus, social analytics in this sense attempts to articulate the differences between philosophy and sociology.

From a BI perspective, social analytics involves “monitoring, analyzing, measuring and interpreting digital interactions and relationships of people, topics, ideas and content.” In this perspective, social analytics involves mining the textual content created in social media (e.g., sentiment analysis, natural language processing) and analyzing socially established networks (e.g., influencer identification, profiling, prediction). This is an important business topic because it helps companies gain insight about existing and potential customers’ current and future behaviors, and about the likes and dislikes toward a firm’s products and services.

2.

What is a social network? What is social network analysis?

A social network is a social structure composed of individuals/people (or groups of individuals or organizations) linked to one another with some type of connections/relationships. Social network analysis (SNA) is the systematic examination of social networks. Dating back to the 1950s, social network analysis is an interdisciplinary field that emerged from social psychology, sociology, statistics, and graph (network) theory.

15 .

3.

What is social media? How does it relate to Web 2.0?

Social media refers to the enabling technologies of social interactions among people in which they create, share, and exchange information, ideas, and opinions in virtual communities and networks. It is a group of Internet-based software applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content.

4.

What is social media analytics? What are the reasons behind its increasing popularity?

Social media analytics refers to the systematic and scientific ways to consume the vast amount of content created by Web-based social media outlets, tools, and techniques for the betterment of an organization’s competitiveness. Data includes anything posted in a social media site. The increasing popularity of social media analytics stems largely from the similarly increasing popularity of social media together with exponential growth in the capacities of text and Web analytics technologies.

5.

How can you measure the impact of social media analytics?

First, determine what your social media goals are. From here, you can use analysis tools such as descriptive analytics, social network analysis, and advanced (predictive, text examining content in online conversations), and ultimately prescriptive analytics tools.

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 5.1: Text Mining for Patent Analysis 1.

Why is it important for companies to keep up with patent filings?

Patents provide exclusive rights to an inventor, granted by a country, for a limited period of time in exchange for a disclosure of an invention. The disclosure of these inventions is critical to future advancements in science and technology. If carefully analyzed, patent documents can help identify emerging technologies, inspire novel solutions, foster symbiotic partnerships, and enhance overall awareness of business’ capabilities and limitations. Patent filings are therefore important for companies that hope to remain competitive in a highly changing technological climate.

2.

How did Kodak use text analytics to better analyze patents?

Using dedicated analysts and state-of-the-art software tools (including specialized text mining tools from ClearForest Corp.), Kodak continuously digs deep into 16 .

various data sources (patent databases, new release archives, and product announcements) in order to develop a holistic view of the competitive landscape.

3.

What were the challenges, the proposed solution, and the obtained results?

Kodak’s challenges are to apply more than a century’s worth of knowledge about imaging science and technology to new uses and to secure those new uses with patents. The problem is that it is nearly impossible to efficiently process such enormous amounts of semistructured data (patent documents usually contain partially structured and partially textual data). But through the use of text mining tools, Kodak is able to obtain a wide range of benefits. These include enabling competitive intelligence, making critical business decisions, identifying and recruiting new talent, identifying unauthorized use of Kodak’s patents, identifying complementary inventions for forming symbiotic partnerships, and preventing competitors from creating similar products.

Application Case 5.2: Text Mining Improves Hong Kong Government’s Ability to Anticipate and Address Public Complaints 1.

How did the Hong Kong government use text mining to better serve its constituents?

The government built a Complaints Intelligence System in order to analyze social messages hidden in the complaints data, which provides important feedback on public service and highlights opportunities for service improvement. Utilizing SAS Text Miner, the government is able to better understand underlying issues and quickly respond even as situations evolve. Text mining also enables the government to quickly discover the correlations among some key words in the complaints.

2.

What were the challenges, the proposed solution, and the obtained results?

The major challenge facing the Hong Kong government was analyzing and responding to public complaints to their call center. This involves addressing answers to about 2.65 million calls and 98,000 e-mails per year. Originally, they attempted to compile reports on complaint statistics for reference by government departments manually. But through ‘eyeball’ observations, it was impossible to effectively reveal new or more complex potential public issues and identify their root causes, as most of the complaints were recorded in unstructured textual format. So, the government decided to utilize text processing and mining approaches to uncover trends, patterns, and relationships in the text data. This allowed the government to better understand the voice of the people, improve service delivery, make informed decisions, and develop smart strategies. The result was a boost in public satisfaction.

Application Case 5.3: Mining for Lies 17 .

1.

Why is it difficult to detect deception?

Humans tend to perform poorly at deception-detection tasks. This phenomenon is exacerbated in text-based communications. Although some people believe that they can readily identify those who are not being truthful, a summary of deception research showed that, on average, people are only 54 percent accurate in making veracity determinations.

2.

How can text/data mining be used to detect deception in text?

Through a process known as message feature mining, statements are transcribed for processing, then cues are extracted and selected. Text processing software identifies cues in statements and generates quantified cues. Classification models are trained and tested on quantified cues, and based on this, statements are labeled as truthful or deceptive (e.g., by law enforcement personnel). The featureselection methods along with 10-fold cross-validation allow researchers to compare the prediction accuracy of different data mining methods (for example, neural networks).

3.

What do you think are the main challenges for such an automated system?

One challenge is that the training the system depends on humans to ascertain the truthfulness of statements in the training data itself. You can’t know for sure whether these statements are true or false, so you may be using incorrect training samples when “teaching” the machine learning system to predict lies in new text data. (This answer will vary by student.)

Application Case 5.4: Text Mining and Sentiment Analysis Help Improve Customer Service Performance 1.

How did the financial services firm use text mining and text analytics to improve its customer service performance?

The company used PolyAnalyst’s text analysis tools for extracting complex word patterns, grammatical and semantic relationships, and expressions of sentiment.

The results were classified into context-specific themes to identify actionable issues. The relationships between structured fields and text analysis results were established to identify patterns and interactions. Results were presented via graphical, interactive, web-based reports. Actionable issues were assigned to relevant individuals responsible for their resolution, and criteria were established to capture and detect compliance with the company’s Quality Standards.

2.

What were the challenges, the proposed solution, and the obtained results?

18 .

Continually monitoring service levels is essential for service quality control.

Customer surveys and associate-customer interactions are the best way to obtain data for such monitoring. But manually evaluating this data is subjective, errorprone, and time/labor-intensive. Therefore, the company needed a system for (1) automatically evaluating associate-customer interactions for compliance with quality standards and (2) analyzing survey responses to extract positive and negative feedback, while allowing for the diversity of natural language expression. The solution was to use PolyAnalyst. (See answer to Question #1 for the remainder of the answer.)

Application Case 5.5: Research Literature Survey with Text Mining 1.

How can text mining be used to ease the insurmountable task of literature review?

Text mining enables a semiautomated analysis of large volumes of published literature. Clustering was used in this study to identify the natural groupings of the articles, and list the most descriptive terms that characterized those clusters. This led to discovery and exploration of interesting patterns using tabular and graphical representation of their findings. Use of text and data mining can thus speed up and simplify the literature review process for academic researchers.

2.

What are the common outcomes of a text mining project on a specific collection of journal articles? Can you think of other potential outcomes not mentioned in this case?

Common outcomes include identifying natural clusters of similar articles, helping to identify the optimal number of cluster classifications. Using text mining, you can answer questions such as “Are there clusters that represent different research themes specific to a single journal?” and “Is there a time-varying characterization of those clusters?” Text mining also has other possible applications in literature reviews. For example, sentiment analysis can help to identify positive and negative judgments. Text mining can be used to build taxonomies of concepts and terms within and between research articles. You can find common themes by author as well as by journal.

Application Case 5.6: Whirlpool Achieves Customer Loyalty and Product Success with Text Analytics 1.

How did Whirlpool use capabilities of text analytics to better understand their customers and improve product offerings?

Whirlpool uses Attensity products for deep text analytics of their multi-channel customer data, which includes e-mails, CRM notes, repair notes, warranty data, and social media. The company uses text analytics solutions every day to get to the root cause of product issues and receive alerts on emerging issues. Users of Attensity’s analytics products at Whirlpool include product/ service managers, 19 .

corporate/product safety staff, consumer advocates, service quality staff, innovation managers, the Category Insights team, and all of Whirlpool’s manufacturing divisions (across five countries).

2.

What were the challenges, the proposed solution, and the obtained results?

Customer satisfaction and feedback are at the center of how Whirlpool drives its overarching business strategy, so gaining insight into customer satisfaction and product feedback is paramount. Whirlpool needed to more effectively understand and react to customer and product feedback data, originating from blogs, e-mails, reviews, forums, repair notes, and other data sources. Managers needed to report on longitudinal data, and be able to compare issues by brand over time. The solution was to use Attensity’s Text Analytics application. This enabled the company to more proactively identify and mitigate quality issues before issues escalated and claims were filed. Since then, Whirlpool has also been able to avoid recalls, which increased customer loyalty and reduced costs (realizing 80% savings on their costs of recalls due to early detection).

Application Case 5.7: Understanding Why Customers Abandon Shopping Carts Results in $10 Million Sales Increase 1.

How did Lotte.com use analytics to improve sales?

Lotte.com introduced its integrated Web traffic analysis system using the SAS for Customer Experience Analytics solution. This enabled Lotte.com to accurately measure and analyze website visitor numbers (UV), page view (PV) status of site visitors and purchasers, the popularity of each product category and product, clicking preferences for each page, the effectiveness of campaigns, and much more. This information enables Lotte.com to better understand customers and their behavior online, and conduct sophisticated, cost-effective targeted marketing.

2.

What were the challenges, the proposed solution, and the obtained results?

With almost 1 million website visitors each day, Lotte.com needed to know how many visitors were making purchases and which channels were bringing the most valuable traffic. SAS for Customer Experience Analytics solution fully transformed the Lotte.com website. As a result, Lotte.com has been able to improve the online experience for its customers as well as generate better returns from its marketing campaigns. This led to a jump in customer loyalty, improved market analysis, a jump in customer satisfaction, and higher sales.

3.

Do you think e-commerce companies are in better position to leverage benefits of analytics? Why? How?

20 .

To the degree that e-commerce companies integrate analytics into their systems, they can take advantage of these technologies to make improvements in the user experience on their e-commerce systems and thereby increase customer satisfaction.

Application Case 5.8: Allegro Boosts Online Click-Thru Rates by 500 Percent with Web Analysis 1.

How did Allegro significantly improve click-through rates with Web analytics?

Allegro created a new, highly personalized solution integrating predictive analytics and campaign management into a real-time recommendation system.

The company applied Social Network Analysis (SNA) as the analytic methodology underlying its product recommendation system. The SNA process grouped similar products into communities based on their commonalities; then, communities were weighted based on visitor click paths, items placed in shopping carts, and purchases to create predictive attributes. From these, rules were established for optimizing product recommendation placement.

2.

What were the challenges, the proposed solution, and the obtained results?

Allegro’s challenge was how to match the right offer to the right customer while still being able to support the extraordinary amount of data it held. The company had over 75 proprietary websites in 11 European countries, over 15 million products, and generated over 500 million page views per day. In particular Allegro aimed to increase its income and gross merchandise volume from its current network, as measured by click-through rates and conversion rates. The solution was to implement the SNA-based product recommendation system described above. The results were dramatic. User traffic increased by 30%, clickthrough rates increased by 500%, and conversion rates are 40 times what they were previously.

Application Case 5.9: Social Network Analysis Helps Telecommunication Firms 1.

How can social network analysis be used in the telecommunications industry?

It can help telecommunications companies listen to and understand the needs and wants of customers. This enables them to offer communication plans, prices, and features that are tailored to the individual customer. SNA can be applied to call records that already exist in a telecommunication company’s database. This data is analyzed using SNA metrics such as influencers, degree, density, betweenness and centrality.

2.

What do you think are the key challenges, potential solution, and probable results in applying SNA in telecommunications firms?

21 .

Because of the widespread use of free Internet tools and techniques (VoIP, video conferencing tools such as Skype, free phone calls within the United States by Google Voice, etc.), the telecommunication industry is going through a tough time. In order to stay viable and competitive, companies need to make the right decisions and utilize their limited resources optimally. Market pressures force telecommunication companies to be more innovative. Using SNA, companies can manage churn, improve cross-sell and technology transfer, manage viral campaigns, better identify customers, and gain competitor insight.

Application Case 5.10: Measuring the Impact of Social Media at Lollapalooza 1.

How did C3 Presents use social media analytics to improve its business?

The company contracted with Cardinal Path to architect and implement a solution based on an existing Google Analytics implementation. A combination of customized event tracking, campaign tagging, custom variables, and a complex implementation and configuration was deployed to include the tracking of each social media outlet on the site.

2.

What were the challenges, the proposed solution, and the obtained results?

C3 Presents was making extensive use of social media to publicize the Lollapalooza event, and needed to know if it was working. The company wanted to answer several questions. Which fans are using social media and sharing content? What social media is being used the most, and how? Are visitors that interact with social media more likely to buy a ticket? Is social media driving more traffic to the site? Is that traffic buying tickets? As a result of implementing the systems with Cardinal Path, the company was able to gain insights that helped C3 quantify the return on their social media investment.

Application Case 5.11: eHarmony Uses Social Media to Help Take the Mystery Out of Online Dating (This application case has no discussion questions.)

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION 1.

  

Explain the relationship among data mining, text mining, and sentiment analysis.

Technically speaking, data mining is a process that uses statistical, mathematical, and artificial intelligence techniques to extract and identify useful information and subsequent knowledge (or patterns) from large sets of data. Data mining is the general concept. Text mining is a specific application of data mining: applying it to unstructured text files. Sentiment analysis is a specialized form of text and data

22 .

mining that identifies and classifies terms in text sources according to sentiment (e.g. judgment, opinion, and emotional content).

2.

In your own words, define text mining and discuss its most popular applications.

Students’ answers will vary.

3.

What does it mean to induce structure into the text-based data? Discuss the alternative ways of inducing structure into text-based data.

Text mining, like other data mining approaches, is an inductive approach for finding patterns and trends in data. One difference between text mining and other data mining approaches is the use of natural language processing.

Text-based data is inherently unstructured and must be converted to a structured format for predictive modeling or other type of analysis. The third step of the 3step text mining process utilizes inductive algorithms to classify and structure the corpus of text sources so that knowledge can be extracted.

Four possible approaches for inducing structure in text in order to extract knowledge are (a) classification (grouping terms into predefined categories), (b) clustering (coming up with “natural” groupings), (c) association rule learning (finding frequent combinations of terms), and (d) trend analysis (recognizing concept distributions based on specific collections of documents).

4.

What is the role of natural language processing in text mining? Discuss the capabilities and limitations of NLP in the context of text mining.

NLP is an important component of text mining. It studies the problem of “understanding” the natural human language, with the view of converting depictions of human language (such as textual documents) into more formal representations (in the form of numeric and symbolic data) that are easier for computer programs to manipulate. The goal of NLP is to move beyond syntaxdriven text manipulation (which is often called “word counting”) to a true understanding and processing of natural language that considers grammatical and semantic constraints as well as the context.

5.

List and discuss three prominent application areas for text mining. What is the common theme among the three application areas you chose?

National security and counterterrorism: The FBI’s system is expected to create a gigantic data warehouse along with a variety of data and text mining modules to meet the knowledge discovery needs of federal, state, and local law enforcement agencies.

Biomedical: A system extracts disease–gene relationships from literature accessed via MEDLINE. 23 .

Marketing: Text mining can be used to increase cross-selling and up-selling by analyzing the unstructured data generated by call centers.

The value of all of these applications is significantly increased by extracting knowledge from huge volumes of text-data and documents through the use of text mining tools.

6.

What is sentiment analysis? How does it relate to text mining?

Sentiment analysis tries to answer the question, “What do people feel about a certain topic?” by digging into opinions of many using a variety of automated tools. It is also known as opinion mining, subjectivity analysis, and appraisal extraction. Sentiment analysis shares many characteristics and techniques with text mining. However, unlike text mining, which categorizes text by conceptual taxonomies of topics, sentiment classification generally deals with two classes (positive versus negative), a range of polarity (e.g., star ratings for movies), or a range in strength of opinion.

7.

What are the common challenges that sentiment analysis has to deal with?

Sentiment that appears in text comes in two flavors: explicit, where the subjective sentence directly expresses an opinion (“It’s a wonderful day”), and implicit, where the text implies an opinion (“The handle breaks too easily”). Implicit sentiment analysis is harder to analyze because it may not include words that are obviously evaluations or judgments. Another challenge involves the timeliness of collection/analysis of textual data coming from a wide variety of data sources. A third challenge is the difficulty of identifying whether a piece of text involves sentiment or not, especially with implicit sentiment analysis. The same sorts of issues involving text mining in natural language settings also apply to sentiment analysis.

8.

What are the most popular application areas for sentiment analysis? Why? Customer relationship management (CRM) and customer experience management are popular “voice of the customer (VOC)” applications. Other application areas include “voice of the market (VOM)” and “voice of the employee (VOE).”

9.

What are the main steps in carrying out sentiment analysis projects?

The first step when performing sentiment analysis of a text document is called sentiment detection, during which text data is differentiated between fact and opinion (objective vs. subjective). This is followed by negative-positive (N-P) polarity classification, where a subjective text item is classified on a bipolar range. Following this comes target identification (identifying the person, product, event, etc. that the sentiment is about). Finally come collection and aggregation, in which the overall sentiment for the document is calculated based on the calculations of sentiments of individual phrases and words from the first three steps.

24 .

10.

What are the two common methods for polarity identification? Explain.

Polarity identification can be done via a lexicon (as a reference library) or by using a collection of training documents and inductive machine learning algorithms. The lexicon approach uses a catalog of words, their synonyms, and their meanings, combined with numerical ratings indicating the position on the NP polarity associated with these words. In this way, affective, emotional, and attitudinal phrases can be classified according to their degree of positivity or negativity. By contrast, the training-document approach uses statistical analysis and machine learning algorithms, such as neural networks, clustering approaches, and decision trees to ascertain the sentiment for a new text document based on patterns from previous “training” documents with assigned sentiment scores.

11.

Discuss the differences and commonalities between text mining and Web mining.

Text mining is a specific application of data mining: applying it to unstructured text files. Web mining is a specific application of data mining: applying it to information on and about the Web (content, structure, and usage).

12.

In your own words, define Web mining and discuss its importance.

Students’ answers will vary.

13.

What are the three main areas of Web mining? Discuss the differences and commonalities among these three areas.

Web mining consists of three areas: Web content mining, Web structure mining, and Web usage mining.

Web content mining refers to the automatic extraction of useful information from Web pages. It may be used to enhance search results produced by search engines.

Web structure mining refers to generating interesting information from the links included in Web pages. Web structure mining can also be used to identify the members of a specific community and perhaps even the roles of the members in the community.

Web usage mining refers to developing useful information through analysis of Web server logs, user profiles, and transaction information.

14.

What is a search engine? Why are they important for businesses?

A search engine is a software program that searches for documents (Internet sites or files) based on the keywords (individual words, multi-word terms, or a complete sentence) that users have provided that have to do with the subject of their inquiry. This is the most prominent type of information retrieval system for finding relevant content on the Web. Search engines have become the centerpiece of most Internet-based transactions and other activities. Because people use them 25 .

extensively to learn about products and services, it is very important for companies to have prominent visibility on the Web; hence the major effort of companies to enhance their search engine optimization (SEO).

15.

What is search engine optimization? Who benefits from it? How? Search engine optimization (SEO) is the intentional activity of affecting the visibility of an e-commerce site or a website in a search engine’s natural (unpaid or organic) search results. It involves editing a page’s content, HTML, metadata, and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines. In addition, SEO efforts include promoting a site to increase its number of inbound links. SEO primarily benefits companies with e-commerce sites by making their pages appear toward the top of search engine lists when users query.

16.

What is Web analytics? What are the metrics used in Web analytics?

Web analytics, also called Web usage mining, can be considered a part of Web mining, and aims to describe what has happened on the website (employing a predefined, metrics-driven descriptive analytics methodology). There are four main categories of Web analytic metrics: • • • •

Website usability: How were they using my website? These involve page views, time on site, downloads, click map, and click paths.

Traffic sources: Where did they come from? These include referral websites, search engines, direct, offline campaigns, and online campaigns.

Visitor profiles: What do my visitors look like? These include keywords, content groupings, geography, time of day, and landing page profiles.

Conversion statistics: What does all this mean for the business? Metrics include new visitors, returning visitors, leads, sales/conversions, and abandonments.

These metrics are important because they provide access to a lot of valuable marketing data, which can be leveraged for better insights to grow your business and better document your ROI. The insight and intelligence gained from Web analytics can be used to effectively manage the marketing efforts of an organization and its various products or services.

17.

Define social analytics, social network, and social network analysis. What are the relationships among them?

Social analytics involves monitoring, analyzing, measuring and interpreting digital interactions and relationships of people, topics, ideas, and content. It involves mining the textual content created in social media (e.g., sentiment analysis, natural language processing) and analyzing socially established networks (e.g., influencer identification, profiling, prediction). A social network is 26 .

a social structure composed of individuals/people (or groups of individuals or organizations) linked to one another with some type of connections/relationships.

Social network analysis (SNA) is the systematic examination of social networks, and is an interdisciplinary field that emerged from social psychology, sociology, statistics, and graph (network) theory. Social analytics therefore combines text analysis for content and sentiment in online communications with social network analysis to identify and analyze relationships between individuals in a community.

18.

What is social media analytics? How is it done? Who does it? What comes out of it?

Social media analytics refers to the systematic and scientific ways to consume the vast amount of content created by Web-based social media outlets, tools, and techniques for the betterment of an organization’s competitiveness. It is done using many analytic methods, including text mining, sentiment analysis, and social network analysis. Companies use it to get a better understanding of their customer base, and can gain financial and competitive advantages from doing so.

Governments use it to track potential terrorist threats, which can lead to enhanced national security. Social scientists use it to get a better understanding of how communities and societies work, which can provide guidance on how to best manage these societies.

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS

 

1. Describe challenges that APUS was facing. Discuss the ramifications of such challenges.

One big challenge is keeping and improving retention in an online university. But educators lacked the hard data to prove conclusively what really drives retention.

Colleges and universities have struggled to understand how to lower dropout rates and keep students on track all the way to graduation. APUS faced this challenge in a new online environment, which further complicates the retention dilemma.

2. What types of data did APUS tap into? What do you think are the main obstacles one would have to overcome when using data that comes from different domains and sources?

APUS tapped into demographic data, registration data, and course level data. They used online activity as well as end-of-course survey data. (The remainder of this answer may vary according to student.) One obstacle with bringing in data from different domains is the complexity of finding meaningful patterns in the data. This could require data extraction, cleaning, and transforming the source data.

27 .

3. What solutions did they pursue? What tools and technologies did they use?

They used predictive analytics to zero in on those factors that most influence a student’s decision to stay in school or drop out. In addition, they built dashboards to put the predictive analytics into the hands of deans and other administrators.

4. What were the results? Do you think these results are also applicable to other educational institutions? Why? Why not?

Some of the findings generated by its predictive models actually came as a surprise to APUS. For example, it had long been assumed that gender and ethnicity were good predictors of attrition, but the models proved otherwise. Educators also assumed that a preparatory course called “College 100: Foundations of Online Learning” was a major driver of retention, but an in-depth analysis using Modeler came to a different conclusion. Also, the results showed that a sense of community was an important retention factor. So, APUS is putting more effort into promoting interaction between students via social media and online collaboration tools.

Predictive analytics are clearly applicable at other universities. But some metrics may not apply. For example, click-stream analysis won’t apply for universities with traditional face-to-face classroom delivery.

5. What additional analysis are they planning on conducting? Can you think of other data analyses that they could apply and benefit from?

APUS also plans to delve deeper into the surveys by mining the open-ended text responses that are part of each questionnaire (IBM SPSS Text Analytics for Surveys will help with that initiative). All of these data-driven initiatives aim to increase student learning, enhance the student’s experience, and build an environment that encourages retention at APUS.

28 .

CHAPTER

6

Big Data and Analytics

Learning Objectives for Chapter 6 ▪

Learn what Big Data is and how it is changing the world of analytics

▪

Understand the motivation for and business drivers of Big Data analytics

▪

Become familiar with the wide range of enabling technologies for Big Data analytics

▪

Learn about Hadoop, MapReduce, and NoSQL as they relate to Big Data analytics

▪

Understand the role of and capabilities/skills for data scientist as a new analytics profession

▪

Compare and contrast the complementary uses of data warehousing and Big Data

▪

Become familiar with the vendors of Big Data tools and services

▪

Understand the need for and appreciate the capabilities of stream analytics

▪

Learn about the applications of stream analytics

CHAPTER OVERVIEW

              

Big Data, which means many things to many people, is not a new technological fad. It is a business priority that has the potential to profoundly change the competitive landscape in today’s globally integrated economy. In addition to providing innovative solutions to enduring business challenges, Big Data and analytics instigate new ways to transform processes, organizations, entire industries, and even society all together. Yet extensive media coverage makes it hard to distinguish hype from reality. This chapter aims to provide a comprehensive coverage of Big Data, its enabling technologies, and related 1 .

analytics concepts to help understand the capabilities and limitations of this emerging technology. The chapter starts with a definition and related concepts of Big Data followed by the technical details of the enabling technologies including Hadoop, MapReduce, and NoSQL. After describing data scientist as a new fashionable organizational role/job, we provide a comparative analysis between data warehousing and Big Data analytics. The last part of the chapter is dedicated to stream analytics, which is one of the most promising value propositions of Big Data analytics.

This chapter contains the following sections:

CHAPTER OUTLINE 6.1

OPENING VIGNETTE: BIG DATA MEETS BIG SCIENCE AT CERN  Questions for the Opening Vignette A. WHAT WE CAN LEARN FROM THIS VIGNETTE

6.2

DEFINITION OF BIG DATA A. THE VS THAT DEFINE BIG DATA 1.

Volume ⧫ Technology Insights 6.1: The Data Size Is Getting Big, Bigger, and Bigger 2.

Variety 3.

Velocity 4.

Veracity 5.

Variability 6.

Value Proposition ⧫ Application Case 6.1: Big Data Analytics Helps Luxottica Improve Its Marking Effectiveness  Section 6.2 Review Questions

6.3

FUNDAMENTALS OF BIG DATA ANALYTICS A. BUSINESS PROBLEMS ADDRESSED BY BIG DATA ANALYTICS ⧫ Application Case 6.2: Top 5 Investment Bank Achieves Single Source of the Truth  Section 6.3 Review Questions

6.4

BIG DATA TECHNOLOGIES A. MAPREDUCE B. WHY USE MAPREDUCE? C. HADOOP D. HOW DOES HADOOP WORK? E. HADOOP TECHNICAL COMPONENTS 1.

Hive 2.

Pig 3.

HBase 4.

Flume 2 .

F.

G.

5.

Oozie 6.

Ambari 7.

Avro 8.

Mahout 9.

Sqoop 10.

HCatalog HADOOP: THE PROS AND CONS ⧫ Technology Insights 6.2: A Few Demystifying Facts about Hadoop NOSQL ⧫ Application Case 6.3: eBay’s Big Data Solution  Section 6.4 Review Questions

6.5

DATA SCIENTIST A. WHERE DO DATA SCIENTISTS COME FROM? ⧫ Technology Insights 6.3: A Typical Job Post for Data Scientists ⧫ Application Case 6.4: Big Data and Analytics in Politics  Section 6.5 Review Questions

6.6

BIG DATA AND DATA WAREHOUSING A. USE CASES FOR HADOOP B. USE CASES FOR DATA WAREHOUSING C. THE GRAY AREAS (ANY ONE OF THE TWO WOULD DO THE JOB) D. COEXISTENCE OF HADOOP AND DATA WAREHOUSE  Section 6.6 Review Questions

6.7

BIG DATA VENDORS ⧫ Application Case 6.5: Dublin City Council Is Leveraging Big Data to Reduce Traffic Congestion ⧫ Technology Insights 6.4: How to Succeed with Big Data ⧫ Application Case 6.6: Creditreform Boosts Credit Rating Quality with Big Data Visual Analytics  Section 6.7 Review Questions

6.8

BIG DATA AND STREAM ANALYTICS A. STREAM ANALYTICS VERSUS PERPETUAL ANALYTICS B. CRITICAL EVENT PROCESSING C. DATA STREAM MINING  Section 6.8 Review Questions

6.9

APPLICATIONS OF STREAM ANALYTICS A. E-COMMERCE B. TELECOMMUNICATIONS

3 .

⧫

C. D. E. F. G.

Application Case 6.7: Turning Machine-Generated Streaming Data into Valuable Business Insights LAW ENFORCEMENT AND CYBER SECURITY POWER INDUSTRY FINANCIAL SERVICES HEALTH SCIENCES GOVERNMENT  Section 6.9 Review Questions

Chapter Highlights Key Terms Questions for Discussion Exercises Teradata University Network (TUN) and Other Hands-on Exercises ⧫ End of Chapter Application Case: Discovery Health Turns Big Data into Better Healthcare References

TEACHING TIPS/ADDITIONAL INFORMATION

       

This chapter is about Big Data and its ramifications for the field of analytics. Here we are talking about new and emerging trends. Big Data technologies are considered “cutting edge,” and this should be exciting for your students to consider. Have them imagine what it would be like to be a “data scientist.” The data scientist characteristics are things they can reflect on; this will give the chapter a personal relevance.

If your course has a technical component, you may want to consider giving some examples and/or exercises utilizing the Hadoop and MapReduce APIs. These are Javabased, and could be incorporated into a curriculum that includes Java programming.

It is important for students to understand the differences and complementary potentials that exist between the traditional RDBMS data warehouse type of data and the largely unstructured (or multi-structured) data that you find with Big Data. It is also important for them to understand that the very term “Big Data” has many interpretations.

Stress the six Vs often in your lecture.

The concept of stream analytics is exciting, and presents a new paradigm to the concept of “data” for students to consider. So far, they’ve considered data analytics in the context of permanently stored data, whether in a database or in a (preserved) e-mail thread or in a (preserved) social media post or blog. With stream analytics, you are considering data that streams in and is not permanently stored, which changes the nature of the analytic processing that can take place. This is a great topic of conversation to have in the classroom.

4 .

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 6.1 Review Questions 1.

What is CERN, and why is it important to the world of science?

CERN is the European Organization for Nuclear Research. It plays a leading role in fundamental studies of physics. It has been instrumental in many key global innovations and breakthrough discoveries in theoretical physics and today operates the world’s largest particle physics laboratory, home to the Large Hadron Collider (LHC).

2.

How does Large Hadron Collider work? What does it produce?

The LHS houses the world’s largest and the most sophisticated scientific instruments to study the basic constituents of matter—the fundamental particles.

These instruments include purpose-built particle accelerators and detectors.

Accelerators boost the beams of particles to very high energies before the beams are forced to collide with each other or with stationary targets. Detectors observe and record the results of these collisions, which are happening at or near the speed of light. This process provides the physicists with clues about how the particles interact, and provides insights into the fundamental laws of nature.

3.

What is the essence of the data challenge at CERN? How significant is it?

Collision events in LHC occur 40 million times per second, resulting in 15 petabytes of data produced annually at the CERN Data Centre. CERN does not have the capacity to process all of the data that it generates, and therefore relies on numerous other research centers all around the world to access and process the data. Processing all the data of their experiments requires an enormously complex distributed and heterogeneous computing and data management system. With such vast quantities of data, both structured and unstructured, information discovery is a big challenge for CERN.

4.

What was the solution? How were the Big Data challenges addressed with this solution?

CMS’s data management and workflow management (DMWM) created the Data Aggregation System (DAS), built on MongoDB (a Big Data management infrastructure) to provide the ability to search and aggregate information across this complex data landscape. MongoDB’s non-schema structure allowed flexible data structures to be stored and indexed.

5.

What were the results? Do you think the current solution is sufficient?

5 .

DAS is used 24 hours a day, seven days a week, by CMS physicists, data operators, and data managers at research facilities around the world. The performance of MongoDB has been outstanding, with an ability to offer a free text query system that is fast and scalable. Without help from DAS, information lookup would have taken orders of magnitude longer. The current solution is outstanding, but more improvements can be made and CERN is looking to apply big data approached beyond CMS.

Section 6.2 Review Questions 1.

Why is Big Data important? What has changed to put it in the center of the analytics world?

As more and more data becomes available in various forms and fashions, timely processing of the data with traditional means becomes impractical. The exponential growth, availability, and use of information, both structured and unstructured, brings Big Data to the center of the analytics world. Pushing the boundaries of data analytics uncovers new insights and opportunities for the use of Big Data.

2.

How do you define Big Data? Why is it difficult to define?

Big Data means different things to people with different backgrounds and interests, which is one reason it is hard to define. Traditionally, the term “Big Data” has been used to describe the massive volumes of data analyzed by huge organizations such as Google or research science projects at NASA. Big Data includes both structured and unstructured data, and it comes from everywhere: data sources include Web logs, RFID, GPS systems, sensor networks, social networks, Internet-based text documents, Internet search indexes, detailed call records, to name just a few. Big data is not just about volume, but also variety, velocity, veracity, and value proposition.

3.

Out of the Vs that are used to define Big Data, in your opinion, which one is the most important? Why? Although all of the Vs are important characteristics, value proposition is probably the most important for decision makers’ “big” data in that it contains (or has a greater potential to contain) more patterns and interesting anomalies than “small” data. Thus, by analyzing large and feature rich data, organizations can gain greater business value that they may not have otherwise. While users can detect the patterns in small data sets using simple statistical and machine-learning methods or ad hoc query and reporting tools, Big Data means “big” analytics. Big analytics means greater insight and better decisions, something that every organization needs nowadays. (Different students may have different answers.)

6 .

4.

What do you think the future of Big Data will be like? Will it lose its popularity to something else? If so, what will it be?

Big Data could evolve at a rapid pace. The buzzword “Big Data” might change to something else, but the trend toward increased computing capabilities, analytics methodologies, and data management of high volume heterogeneous information will continue. (Different students may have different answers.)

Section 6.3 Review Questions 1.

What is Big Data analytics? How does it differ from regular analytics?

Big Data analytics is analytics applied to Big Data architectures. This is a new paradigm; in order to keep up with the computational needs of Big Data, a number of new and innovative analytics computational techniques and platforms have been developed. These techniques are collectively called high-performance computing, and include in-memory analytics, in-database analytics, grid computing, and appliances. They differ from regular analytics which tend to focus on relational database technologies.

2.

What are the critical success factors for Big Data analytics?

Critical factors include a clear business need, strong and committed sponsorship, alignment between the business and IT strategies, a fact-based decision culture, a strong data infrastructure, the right analytics tools, and personnel with advanced analytic skills.

3.

What are the big challenges that one should be mindful of when considering implementation of Big Data analytics?

Traditional ways of capturing, storing, and analyzing data are not sufficient for Big Data. Major challenges are the vast amount of data volume, the need for data integration to combine data of different structures in a cost-effective manner, the need to process data quickly, data governance issues, skill availability, and solution costs.

4.

What are the common business problems addressed by Big Data analytics?

Here is a list of problems that can be addressed using Big Data analytics: • Process efficiency and cost reduction • Brand management • Revenue maximization, cross-selling, and up-selling • Enhanced customer experience • Churn identification, customer recruiting • Improved customer service 7 .

• • • •

Identifying new products and market opportunities Risk management Regulatory compliance Enhanced security capabilities

Section 6.4 Review Questions 1.

What are the common characteristics of emerging Big Data technologies?

They take advantage of commodity hardware to enable scale-out, parallel processing techniques; employ nonrelational data storage capabilities in order to process unstructured and semistructured data; and apply advanced analytics and data visualization technology to Big Data to convey insights to end users.

2.

What is MapReduce? What does it do? How does it do it?

MapReduce is a programming model that allows the processing of large-scale data analysis problems to be distributed and parallelized. The MapReduce technique, popularized by Google, distributes the processing of very large multistructured data files across a large cluster of machines. High performance is achieved by breaking the processing into small units of work that can be run in parallel across the hundreds, potentially thousands, of nodes in the cluster. The map function in MapReduce breaks a problem into sub-problems, which can each be processed by single nodes in parallel. The reduce function merges (sorts, organizes, aggregates) the results from each of these nodes into the final result.

3.

What is Hadoop? How does it work?

Hadoop is an open source framework for processing, storing, and analyzing massive amounts of distributed, unstructured data. It is designed to handle petabytes and exabytes of data distributed over multiple nodes in parallel, typically commodity machines connected via the Internet. It utilizes the MapReduce framework to implement distributed parallelism. The file organization is implemented in the Hadoop Distributed File System (HDFS), which is adept at storing large volumes of unstructured and semistructured data.

This is an alternative to the traditional tables/rows/columns structure of a relational database. Data is replicated across multiple nodes, allowing for fault tolerance in the system.

4.

What are the main Hadoop components? What functions do they perform?

Major components of Hadoop are the HDFS, a Job Tracker operating on the master node, Name Nodes, Secondary Nodes, and Slave Nodes. The HDFS is the default storage layer in any given Hadoop cluster. A Name Node is a node in a Hadoop cluster that provides the client information on where in the cluster particular data is stored and if any nodes fail. Secondary nodes are backup name 8 .

nodes. The Job Tracker is the node of a Hadoop cluster that initiates and coordinates MapReduce jobs or the processing of the data. Slave nodes store data and take direction to process it from the Job Tracker.

Querying for data in the distributed system is accomplished via MapReduce. The client query is handled in a Map job, which is submitted to the Job Tracker. The Job Tracker refers to the Name Node to determine which data it needs to access to complete the job and where in the cluster that data is located, then submits the query to the relevant nodes which operate in parallel. A Name Node acts as facilitator, communicating back to the client information such as which nodes are available, where in the cluster certain data resides, and which nodes have failed.

When each node completes its task, it stores its result. The client submits a Reduce job to the Job Tracker, which then collects and aggregates the results from each of the nodes.

5.

What is NoSQL? How does it fit into the Big Data analytics picture?

NoSQL, also known as “Not Only SQL,” is a new style of database for processing large volumes of multi-structured data. Whereas Hadoop is adept at supporting large-scale, batch-style historical analysis, NoSQL databases are mostly aimed at serving up discrete data stored among large volumes of multi-structured data to end-user and automated Big Data applications. NoSQL databases trade ACID (atomicity, consistency, isolation, durability) compliance for performance and scalability.

Section 6.5 Review Questions 1.

What is a data scientist? What makes them so much in demand?

Data scientists use a combination of their business and technical skills to investigate Big Data, looking for ways to improve current business analytics practices (from descriptive to predictive and prescriptive) and hence to improve decisions for new business opportunities. One of the biggest differences between a data scientist and a business intelligence user—such as a business analyst—is that a data scientist investigates and looks for new possibilities, while a BI user analyzes existing business situations and operations. Data scientist is an emerging profession, and there is no consensus on where data scientists come from or what educational background a data scientist has to have. But there is a common understanding of what skills and qualities they are expected to possess, which involve a combination of soft and hard skills.

2.

What are the common characteristics of data scientists? Which one is the most important?

One of the most sought-out characteristics of a data scientist is expertise in both technical and business application domains. Data scientists are expected to have 9 .

soft skills such as creativity, curiosity, communication/interpersonal skills, domain expertise, problem definition skills, and managerial skills as well as sound technical skills such as data manipulation, programming/hacking/scripting, and knowledge of Internet and social media/networking technologies. Data scientists are supposed to be creative and curious, and should be excellent communicators, with the ability to tell compelling stories about their data.

3.

Where do data scientists come from? What educational backgrounds do they have?

Data scientist is an emerging profession, and there is no consensus on where data scientists come from or what educational background a data scientist has to have.

Master of Science (or Ph.D.) in Computer Science, MIS, Industrial Engineering, of postgraduate analytics are common examples. But many data scientists have advanced degrees in other disciplines, like the physical or social sciences, or more specialized fields like ecology or system biology.

4.

What do you think is the path to becoming a great data scientist?

Becoming a great data scientist requires you to delve deeply into developing quantitative and technical skills, as well as interpersonal and communication skills. In addition, you will need to gain significant domain knowledge (e.g., in business). This effort will most likely require an advanced degree. It also requires a continuous thirst for knowledge and an intense curiosity; you will always be learning in this profession. In addition to meticulous analytical skills, it also requires creativity and imagination. (Students will vary in their answers to this question.)

Section 6.6 Review Questions 1.

What are the challenges facing data warehousing and Big Data? Are we witnessing the end of the data warehousing era? Why or why not?

What has changed the landscape in recent years is the variety and complexity of data, which made data warehouses incapable of keeping up. It is not the volume of the structured data but the variety and the velocity that forced the world of IT to develop a new paradigm, which we now call “Big Data.” But this does not mean the end of data warehousing. Data warehousing and RDBMS still bring many strengths that make them relevant for BI and that Big Data techniques do not currently provide.

2.

What are the use cases for Big Data and Hadoop?

In terms of its use cases, Hadoop is differentiated two ways: first, as the repository and refinery of raw data, and second, as an active archive of historical data. Hadoop, with their distributed file system and flexibility of data formats 10 .

(allowing both structured and unstructured data), is advantageous when working with information commonly found on the Web, including social media, multimedia, and text. Also, because it can handle such huge volumes of data (and because storage costs are minimized due to the distributed nature of the file system), historical (archive) data can be managed easily with this approach.

3.

What are the use cases for data warehousing and RDBMS? Three main use cases for data warehousing are performance, integration, and the availability of a wide variety of BI tools. The relational data warehouse approach is quite mature, and database vendors are constantly adding new index types, partitioning, statistics, and optimizer features. This enables complex queries to be done quickly, a must for any BI application. Data warehousing, and the ETL process, provide a robust mechanism for collecting, cleaning, and integrating data.

And, it is increasingly easy for end users to create reports, graphs, and visualizations of the data.

4.

In what scenarios can Hadoop and RDBMS coexist?

There are several possible scenarios under which using a combination of Hadoop and relational DBMS-based data warehousing technologies makes sense. For example, you can use Hadoop for storing and archiving multi-structured data, with a connector to a relational DBMS that extracts required data from Hadoop for analysis by the relational DBMS. Hadoop can also be used to filter and transform multi-structural data for transporting to a data warehouse, and can also be used to analyze multi-structural data for publishing into the data warehouse environment. Combining SQL and MapReduce query functions enables data scientists to analyze both structured and unstructured data. Also, front end query tools are available for both platforms.

Section 6.7 Review Questions 1.

What is special about the Big Data vendor landscape? Who are the big players?

The Big Data vendor landscape is developing very rapidly. It is in a special period of evolution where entrepreneurial startup firms bring innovative solutions to the marketplace. Cloudera is a market leader in the Hadoop space. MapR and Hortonworks are two other Hadoop startups. DataStax is an example of a NoSQL vendor. Informatica, Pervasive Software, Syncsort, and MicroStrategy are also players. Most of the growth in the industry is with Hadoop and NoSQL distributors and analytics providers. There is still very little in terms of Big Data application vendors. Meanwhile, the next-generation data warehouse market has experienced significant consolidation. Four leading vendors in this space— Netezza, Greenplum, Vertica, and Aster Data—were acquired by IBM, EMC, HP, and Teradata, respectively. Mega-vendors Oracle and IBM also play in the Big 11 .

Data space, connecting and consolidating their products with Hadoop and NoSQL engines.

2.

How do you think the Big Data vendor landscape will change in the near future?

Why? As the field matures, more and more traditional data vendors will incorporate Big Data into their architectures. We already saw something similar with the incorporation of XML data types and XPath processing engines in relational database engines. Also, the Big Data market will be increasingly cloud-based, and hosting services will include Big Data data storage options, along with the traditional MySql and SqlServer options. Vendors providing Big Data applications and services, for example in the finance domain or for scientific purposes, will begin to proliferate. (Different students will have different answers.)

3.

What is the role of visual analytics in the world of Big Data?

Visual analytics help organizations uncover trends, relationships, and anomalies by visually shifting through very large quantities of data. Many vendors are developing visual analytics offerings, which have traditionally applied to structured data warehouse environments (relational and multidimensional), for the Big Data space. To be successful, a visual analytics application must allow for the coexistence and integration of relational and multistructured data.

Section 6.8 Review Questions 1.

What is a stream (in the Big Data world)?

A stream can be thought of as an unbounded flow or sequence of data elements, arriving continuously at high velocity. Streams often cannot be efficiently or effectively stored for subsequent processing; thus Big Data concerns about Velocity (one of the six Vs) are especially prevalent when dealing with streams.

Examples of data streams include sensor data, computer network traffic, phone conversations, ATM transactions, web searches, and financial data.

2.

What are the motivations for stream analytics?

In situations where data streams in rapidly and continuously, traditional analytics approaches that work with previously accumulated data (i.e., data at arrest) often either arrive at the wrong decisions because of using too much out-of-context data, or they arrive at the correct decisions but too late to be of any use to the organization. Therefore it is critical for a number of business situations to analyze the data soon after it is created and/or as soon as it is streamed into the analytics system. It is no longer feasible to “store everything.” Otherwise, analytics will either arrive at the wrong decisions because of using too much out-of-context 12 .

data, or at the correct decisions but too late to be of any use to the organization.

Therefore it is critical for a number of business situations to analyze the data as soon as it is streamed into the analytics system.

3.

What is stream analytics? How does it differ from regular analytics?

Stream analytics is the process of extracting actionable information from continuously flowing/streaming data. It is also sometimes called “data in-motion analytics” or “real-time data analytics.” It differs from regular analytics in that it deals with high velocity (and transient) data streams instead of more permanent data stores like databases, files, or web pages.

4.

What is critical event processing? How does it relate to stream analytics?

Critical event processing is a method of capturing, tracking, and analyzing streams of data to detect events (out of normal happenings) of certain types that are worthy of the effort. It involves combining data from multiple sources to infer events or patterns of interest. An event may also be defined generically as a “change of state,” which may be detected as a measurement exceeding a predefined threshold of time, temperature, or some other value. This applies to stream analytics because the events are happening in real time.

5.

Define data stream mining. What additional challenges are posed by data stream mining?

Data stream mining is the process of extracting novel patterns and knowledge structures from continuous, rapid data records. Processing data streams, as opposed to more permanent data storages, is a challenge. Traditional data mining techniques can process data recursively and repetitively because the data is permanent. By contrast, a data stream is a continuous flow of ordered sequence of instances that can only be read once and must be processed immediately as they come in.

Section 6.9 Review Questions 1.

What are the most fruitful industries for stream analytics?

Many industries can benefit from stream analytics. Some prominent examples include e-commerce, telecommunications, law enforcement, cyber security, the power industry, health sciences, and the government.

2.

How can stream analytics be used in e-commerce?

Companies such as Amazon and eBay use stream analytics to analyze customer behavior in real time. Every page visit, every product looked at, every search conducted, and every click made is recorded and analyzed to maximize the value 13 .

gained from a user’s visit. Behind the scenes, advanced analytics are crunching the real-time data coming from our clicks, and the clicks of thousands of others, to “understand” what it is that we are interested in (in some cases, even we do not know that) and make the most of that information by creative offerings.

3.

In addition to what is listed in this section, can you think of other industries and/or application areas where stream analytics can be used?

Stream analytics could be of great benefit to any industry that faces an influx of relevant real-time data and needs to make quick decisions. One example is the news industry. By rapidly sifting through data streaming in, a news organization can recognize “newsworthy” themes (i.e., critical events). Another benefit would be for weather tracking in order to better predict tornados or other natural disasters. (Different students will have different answers.)

4.

Compared to regular analytics, do you think stream analytics will have more (or less) use cases in the era of Big Data analytics? Why? Stream analytics can be thought of as a subset of analytics in general, just like “regular” analytics. The question is, what does “regular” mean? Regular analytics may refer to traditional data warehousing approaches, which does constrain the types of data sources and hence the use cases. Or, “regular” may mean analytics on any type of permanent stored architecture (as opposed to transient streams). In this case, you have more use cases for “regular” (including Big Data) than in the previous definition. In either case, there will probably be plenty of times when “regular” use cases will continue to play a role, even in the era of Big Data analytics. (Different students will have different answers.)

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 6.1: Big Data Analytics Helps Luxottica Improve Its Marketing Effectiveness 1.

What does “big data” mean to Luxottica?

For Luxottica, Big Data includes everything they can find about their customer interactions (in the form of transactions, click streams, product reviews, and social media postings). They see this as constituting a massive source of business intelligence for potential product, marketing, and sales opportunities.

2.

What were their main challenges?

Because Luxottica outsourced both data storage and promotional campaign development and management, there was a disconnect between data analytics and 14 .

marketing execution. Their competitive posture and strategic growth initiatives were compromised for lack of an individualized view of their customers and an inability to act decisively and consistently on the different types of information generated by each retail channel.

3.

What were the proposed solution and the obtained results?

Luxottica deployed the Customer Intelligence Appliance (CIA) from IBM Business Partner Aginity LLC. This product is built on IBM PureData System for Analytics. This solution helps Luxottica highly segment customer behavior and provide a platform and smart database for marketing execution systems, such as campaign management, e-mail services, and other forms of direct marketing.

Anticipated benefits include a 10% improvement in marketing effectiveness, identifying the highest valued customers, and the ability to target customers based on preference and history.

Application Case 6.2: Top 5 Investment Bank Achieves Single Source of the Truth 1.

How can Big Data benefit large-scale trading banks?

Big Data can potentially handle the high volume, high variability, continuously streaming data that trading banks need to deal with. Traditional relational databases are often unable to keep up with the data demand.

2.

How did MarkLogic infrastructure help ease the leveraging of Big Data?

MarkLogic was able to meet two needs: (1) upgrading existing Oracle and Sybase platforms, and (2) compliance with regulatory requirements. Their solution provided better performance, scalability, and faster development for future requirements than competitors. Most importantly, MarkLogic was able to eliminate the need for replicated database servers by providing a single server providing timely access to the data.

3.

What were the challenges, the proposed solution, and the obtained results?

The Bank’s legacy system was built on relational database technology. As data volumes and variability increased, the legacy system was not fast enough to respond to growing business needs and requirements. It was unable to deliver real-time alerts to manage market and counterparty credit positions in the desired timeframe. Big data offered the scalability to address this problem. The benefits included a new alert feature, less downtime for maintenance, much faster capacity to process complex changes, and reduced operations costs.

Application Case 6.3: eBay’s Big Data Solution 15 .

1.

Why did eBay need a Big Data solution? eBay is the world’s largest online marketplace, and its success requires the ability to turn the enormous volumes of data it generates into useful insights for customers. Big Data is essential for this effort.

2.

What were the challenge, the proposed solution, and the obtained results? eBay was experiencing explosive data growth and needed a solution that did not have the typical bottlenecks, scalability issues, and transactional constraints associated with common relational database approaches. The company also needed to perform rapid analysis on a broad assortment of the structured and unstructured data it captured. eBay’s solution includes NoSQL via Apache Cassandra and DataStax Enterprise. It also uses integrated Apache Hadoop analytics that come with DataStax. The solution incorporates a scale-out architecture that enables eBay to deploy multiple DataStax Enterprise clusters across several different data centers using commodity hardware. Now, eBay can more cost effectively process massive amounts of data at very high speeds. The new architecture serves a wide variety of new use cases, and its reliability and fault tolerance has been greatly enhanced.

Application Case 6.4: Big Data and Analytics in Politics 1.

What is the role of analytics and Big Data in modern day politics?

Big Data and analytics have a lot to offer to modern-day politics. The main characteristics of Big Data, namely volume, variety, and velocity (the three Vs), readily apply to the kind of data that is used for political campaigns. Big Data Analytics can help predict election outcomes as well as targeting potential voters and donors, and have become a critical part of political campaigns.

2.

Do you think Big Data analytics could change the outcome of an election?

It may well have changed the outcome of the 2008 and 2012 elections. For example, in 2012, Barack Obama had a data advantage and the depth and breadth of his campaign’s digital operation, from political and demographic data mining to voter sentiment and behavioral analysis, reached beyond anything politics had ever seen. So there was a significant difference in expertise and comfort level with modern technology between the two parties. The usage and expertise gap between the party lines may disappear over time, and this will even the playing field. But new data regarding voter sympathies may itself change the thinking and ideological directions of the major parties. This in itself could shift election outcomes. (Answers will vary by student.)

16 .

3.

What do you think are the challenges, the potential solution, and the probable results of the use of Big Data analytics in politics?

One of the big challenges is to ensure some sort of reliability in news media coverage of election issues. For example, most people, including the so-called political experts (who often rely on gut feelings and experiences), thought the 2012 presidential election would be very close. They were wrong. On the other hand, data analysts, using data-driven analytical models, predicted that Barack Obama would win easily with close to 99 percent certainty. The results will be an ever-increasing use of Big Data analytics in politics, both by the parties and candidates themselves and by the news media and analysts who cover them.

Application Case 6.5: Dublin City Council Is Leveraging Big Data to Reduce Traffic Congestion 1.

Is there a strong case to make for large cities to use Big Data Analytics and related information technologies? Identify and discuss examples of what can be done with analytics beyond what is portrayed in this application case.

For the Dublin case, Big Data Analytics were used primarily to ease traffic problems and better understand the traffic network. Municipalities could use these technologies for many other governmental tasks. For example, providing social services is a very complex and difficult process that can be benefit from Big Data technologies. Other areas include tax collection, sanitation services, environmental management, crime prevention, and management of police and fire departments.

2.

How can big data analytics help ease the traffic problem in large cities?

They can help get a better sense of the “traffic health” by identifying traffic congestion in its early stages. By integrating geospatial data from buses and data on bus timetables into a central geographic information system, you can create a digital map of the city. Then, using the dashboard screen, operators can drill down to see the number of buses that are on time or delayed on each route. With big data analytics, users can produce detailed reports on areas of the network where buses are frequently delayed, and take prompt action to ease congestion. Data and analytics can also assist with future planning of roads, infrastructure, and public transportation in order to further ease traffic problems.

3.

What were the challenges Dublin City was facing; what were the proposed solution, initial results, and future plans?

The major problem was the difficulty in getting a good picture of traffic in the city from a high-level perspective. The proposed solution was to team up with IBM Research, and especially their Smarter Cities Technologies Centre. Using 17 .

IBM InfoSphere Streams and mapping software, IBM researchers created a digital map of the city, overlaid with the real-time positions of Dublin’s 1,000 buses.

This gave operators the ability to see the system as a whole instead of just individual corridors. The new system gave insight to the operators and managers.

They could now start answering questions such as: “Are the bus lane start times correct?” and “Where do we need to add additional bus lanes and bus-only traffic signals?” For the future, the Dublin City Council and IBM plan to enhance the system by incorporating meteorological data, under-road sensors, and bicycleusage data into their predictive analytics.

Application Case 6.6: Creditreform Boosts Credit Rating Quality with Big Data Visual Analytics 1.

How did Creditreform boost credit rating quality with Big Data and visual analytics?

Using Big Data and visual analytics, executives can make quicker, better decisions with instant access, via PC or tablet, to insights based on the latest data.

Creditreform employees can use the software to make smart decisions without needing extensive training. This makes Creditreform better at providing the highest quality financial information and credit ratings to its client businesses.

2.

What were the challenges, proposed solution, and initial results?

Creditreform makes more than 24 million credit reports from 26 countries in Europe and from China that are available around the clock. Creditreform wants to maintain its leadership and widen its market lead. Big data and visual analytics are seen as principal technologies to help with this effort. Creditreform decided to use SAS Visual Analytics to simplify the analytics process. SAS Visual Analytics is a high-performance, in-memory solution for exploring massive amounts of data very quickly. This made it faster and easier for their analysts to detect correlations in the business data, which improved the quality and forecasting accuracy of their credit ratings.

Application Case 6.7: Turning Machine-Generated Streaming Data into Valuable Business Insights 1.

Why is stream analytics becoming more popular?

Stream analytics is becoming increasingly more popular for two main reasons.

First, time-to-action has become an ever decreasing value, and second, we have the technological means to capture and process the data while it is being created.

18 .

2.

How did the telecommunications company in this case use stream analytics for better business outcomes? What additional benefits can you foresee?

Mainly they used stream analytics to improve their service delivery in the following areas: application troubleshooting, operations, compliance, and security. For application troubleshooting, stream analytics helps to quickly diagnose the causes of 404 errors encountered by users on their Web site. In terms of operations, stream analytics helps to avoid critical downtime by anticipating potential infrastructure failures before they happen. Stream analytics is helping the company’s audit process, which assists with meeting compliance requirements. Finally, stream analytics enhances the company’s ability to detect system vulnerabilities and attacks.

3.

What were the challenges, proposed solution, and initial results?

This telecommunication company encounters over a billion daily events running on a distributed hardware/software infrastructure supporting millions of cable, online, and interactive media customers. It was overwhelming to gather and view this data in one place, and to perform any diagnostics, or hone in on the real-time intelligence that lives in the machine-generated data. Using time-consuming and error-prone traditional search methods, the company’s roster of experts would shuffle through mountains of data to uncover issues threatening data integrity, system stability, and applications performance. They needed a better way to do this. They decided to work with Splunk, one of the leading stream analytics service providers. The insights gained from the analysis and information provided

by Splunk helped improve in several areas: application troubleshooting, operations, compliance, and security.

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION 1.

  

What is Big Data? Why is it important? Where does Big Data come from?

Traditionally, the term “Big Data” has been used to describe the massive volumes of data analyzed by huge organizations like Google or research science projects at NASA. But for most businesses, it’s a relative term: “Big” depends on an organization’s size. In general, Big Data exceeds the reach of commonly used hardware environments and/or capabilities of software tools to capture, manage, and process it within a tolerable time span for its user population. Big Data has become a popular term to describe the exponential growth, availability, and use of information, both structured and unstructured. Big data is important because it is there and because it is rich in information and insight that, if effectively tapped, can lead to better business decisions and improved company performance. Much has been written on the Big Data trend and how it can serve as the basis for innovation, differentiation, and growth. Big Data includes both structured and 19 .

unstructured data, and it comes from everywhere: data sources include Web logs, RFID, GPS systems, sensor networks, social networks, Internet-based text documents, Internet search indexes, and detailed call records, to name just a few.

Big data is not just about volume, but also variety, velocity, veracity, and value proposition.

2.

What do you think the future of Big Data will be? Will it lose its popularity to something else? If so, what will it be?

Big Data could evolve at a rapid pace. The buzzword “Big Data” might change to something else, but the trend toward increased computing capabilities, analytics methodologies, and data management of high volume heterogeneous information will continue. (Different students may have different answers.)

3.

What is Big Data analytics? How does it differ from regular analytics?

Big Data analytics is analytics applied to Big Data architectures. This is a new paradigm; in order to keep up with the computational needs of Big Data, a number of new and innovative analytics computational techniques and platforms have been developed. These techniques are collectively called high-performance computing, and include in-memory analytics, in-database analytics, grid computing, and appliances. They differ from regular analytics which tend to focus on relational database technologies.

4.

What are the critical success factors for Big Data analytics?

Critical factors include a clear business need, strong and committed sponsorship, alignment between the business and IT strategies, a fact-based decision culture, a strong data infrastructure, the right analytics tools, and personnel with advanced analytic skills.

5.

What are the big challenges that one should be mindful of when considering implementation of Big Data analytics?

Traditional ways of capturing, storing, and analyzing data are not sufficient for Big Data. Major challenges are the vast amount of data volume, the need for data integration to combine data of different structures in a cost-effective manner, the need to process data quickly, data governance issues, skill availability, and solution costs.

6.

What are the common business problems addressed by Big Data analytics?

Here is a list of problems that can be addressed using Big Data analytics: • Process efficiency and cost reduction • Brand management • Revenue maximization, cross-selling, and up-selling • Enhanced customer experience 20 .

• • • • • • 7.

Churn identification, customer recruiting Improved customer service Identifying new products and market opportunities Risk management Regulatory compliance Enhanced security capabilities

What is a data scientist? What makes them so much in demand?

Data scientists use a combination of their business and technical skills to investigate Big Data, looking for ways to improve current business analytics practices (from descriptive to predictive and prescriptive) and hence to improve decisions for new business opportunities. One of the biggest differences between a data scientist and a business intelligence user—such as a business analyst—is that a data scientist investigates and looks for new possibilities, while a BI user analyzes existing business situations and operations. Data scientist is an emerging profession, and there is no consensus on where data scientists come from or what educational background a data scientist has to have. But there is a common understanding of what skills and qualities they are expected to possess, which involve a combination of soft and hard skills.

8.

What are the common characteristics of data scientists? Which one is the most important?

One of the most sought-out characteristics of a data scientist is expertise in both technical and business application domains. Data scientists are expected to have soft skills such as creativity, curiosity, communication/interpersonal skills, domain expertise, problem definition skills, and managerial skills as well as sound technical skills such as data manipulation, programming/hacking/scripting, and knowledge of Internet and social media/networking technologies. Data scientists are supposed to be creative and curious, and should be excellent communicators, with the ability to tell compelling stories about their data.

9.

In the era of Big Data, are we about to witness the end of data warehousing?

Why? What has changed the landscape in recent years is the variety and complexity of data, which made data warehouses incapable of keeping up. It is not the volume of the structured data but the variety and the velocity that forced the world of IT to develop a new paradigm, which we now call “Big Data.” But this does not mean the end of data warehousing. Data warehousing and RDBMS still bring many strengths that make them relevant for BI and that Big Data techniques do not currently provide.

21 .

10.

What are the use cases for Big Data/Hadoop and data warehousing/RDBMS? In terms of its use cases, Hadoop is differentiated two ways: first, as the repository and refinery of raw data, and second, as an active archive of historical data. Hadoop, with their distributed file system and flexibility of data formats (allowing both structured and unstructured data), is advantageous when working with information commonly found on the Web, including social media, multimedia, and text. Also, because it can handle such huge volumes of data (and because storage costs are minimized due to the distributed nature of the file system), historical (archive) data can be managed easily with this approach.

Three main use cases for data warehousing are performance, integration, and the availability of a wide variety of BI tools. The relational data warehouse approach is quite mature, and database vendors are constantly adding new index types, partitioning, statistics, and optimizer features. This enables complex queries to be done quickly, a must for any BI application. Data warehousing, and the ETL process, provide a robust mechanism for collecting, cleaning, and integrating data.

And, it is increasingly easy for end users to create reports, graphs, and visualizations of the data.

11.

What is stream analytics? How does it differ from regular analytics?

Stream analytics is the process of extracting actionable information from continuously flowing/streaming data. It is also sometimes called “data in-motion analytics” or “real-time data analytics.” It differs from regular analytics in that it deals with high velocity (and transient) data streams instead of more permanent data stores like databases, files, or web pages).

12.

What are the most fruitful industries for stream analytics? What is common to those industries?

Many industries can benefit from stream analytics. Some prominent examples include e-commerce, telecommunications, law enforcement, cyber security, the power industry, health sciences, and the government.

13.

Compared to regular analytics, do you think stream analytics will have more (or less) use cases in the era of Big Data analytics? Why? Stream analytics can be thought of as a subset of analytics in general, just like “regular” analytics. The question is, what does “regular” mean? Regular analytics may refer to traditional data warehousing approaches, which does constrain the types of data sources and hence the use cases. Or, “regular” may mean analytics on any type of permanent stored architecture (as opposed to transient streams). In this case, you have more use cases for “regular” (including Big Data) than in the previous definition. In either case, there will probably be plenty of times when “regular” use cases will continue to play a role, even in the era of Big Data analytics. (Different students will have different answers.)

22 .

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS 1.

 

How big is “big data” for Discovery Health?

Discovery Health serves 2.7 million members, and its claims system generates a million new rows of data each day. For analytics purposes, they want to use three years of data, which comes to over a billion rows of claims data alone.

2.

What big data sources did Discovery Health use for their analytic solutions?

The data sources include members’ claims data, pathology results, members’ questionnaires, data from pharmacies, hospital admissions, and billing data, to name a few. Looking to the future, Discovery is also starting to analyze unstructured data, such as text-based surveys and comments from online feedback forms.

3.

What were the main data/analytics challenges Discovery Health was facing?

The challenge faced by Discovery Health was similar to that faced by any company operating in a Big Data environment. Given the huge amount of data that Discovery Health was generating, how could they identify the key insights that their business and their members’ health depend on? To find the needles of vital information in the big data haystack, the company not only needed a sophisticated data-mining and predictive modeling solution, but also an analytics infrastructure with the power to deliver results at the speed of business.

4.

What were the main solutions they have produced?

To achieve this transformation in its analytics capabilities, Discovery worked with BITanium, an IBM Business Partner with deep expertise in operational deployments of advanced analytics technologies. The solution included use of IBM’s SSPS and PureData System for Analytics.

5.

What were the initial results/benefits? What do you think will be the future of “big data” analytics at Discovery?

Discovery Health is now able to run three years’ worth of data for their 2.7 million members through complex statistical models to deliver actionable insights in a matter of minutes. This helps them to better predict and prevent health risks, and to identify and eliminate fraud. The results have been transformative. From an analytics perspective, the speed of the solution gives Discovery more scope to experiment and optimize its models. From a broader business perspective, the combination of SPSS and PureData technologies gives Discovery the ability to put actionable data in the hands of its decision makers faster.

23 .

CHAPTER

7

Business Analytics: Emerging Trends and Future Impacts

Learning Objectives for Chapter 7 ▪

Explore some of the emerging technologies that may impact analytics, BI, and decision support

▪

Describe how geospatial and location-based analytics are assisting organizations

▪

Describe how analytics are powering consumer applications and creating a new opportunity for entrepreneurship for analytics

▪

Describe the potential of cloud computing in business intelligence

▪

Understand Web 2.0 and its characteristics as related to analytics

▪

Describe the organizational impacts of analytics applications

▪

List and describe the major ethical and legal issues of analytics implementation

▪

Understand the analytics ecosystem to get a sense of the various types of players in the analytics industry and how one can work in a variety of roles

CHAPTER OVERVIEW This chapter introduces several emerging technologies that are likely to have major impacts on the development and use of business intelligence applications. Many other interesting technologies are also emerging, but we have focused on some trends that have already been realized and others that are about to impact analytics further. Using a crystal ball is always a risky proposition, but this chapter provides a framework for analysis of emerging trends. We introduce and explain some emerging technologies and explore their current applications. We then discuss the organizational, personal, legal, ethical, and societal impacts of support systems that may affect their implementation. We conclude with a description of the analytics ecosystem. This section should help readers appreciate different career possibilities within the realm of analytics. This chapter contains the following sections:

1 .

CHAPTER OUTLINE 7.1

OPENING VIGNETTE: OKLAHOMA GAS AND ELECTRIC EMPLOYS ANALYTICS TO PROMOTE SMART ENERGY USE  Questions for the Opening Vignette A. WHAT WE CAN LEARN FROM THIS VIGNETTE

7.2

LOCATION-BASED ANALYTICS FOR ORGANIZATIONS A. GEOSPATIAL ANALYTICS ⧫ Application Case 7.1: Great Clips Employs Spatial Analytics to Shave Time in Location Decisions B. REAL-TIME LOCATION INTELLIGENCE ⧫ Application Case 7.2: Quiznos Targets Customers for Its Sandwiches  Section 7.2 Review Questions

7.3

ANALYTICS APPLICATIONS FOR CONSUMERS ⧫ Application Case 7.3: A Life Coach in Your Pocket  Section 7.3 Review Questions

7.4

RECOMMENDATION ENGINES  Section 7.4 Review Questions

7.5

THE WEB 2.0 REVOLUTION AND ONLINE SOCIAL NETWORKING A. REPRESENTATIVE CHARACTERISTICS OF WEB 2.0 B. SOCIAL NETWORKING C. A DEFINITION AND BASIC INFORMATION D. IMPLICATIONS OF BUSINESS AND ENTERPRISE SOCIAL NETWORKS 1.

Using Twitter to Get a Pulse of the Market  Section 7.5 Review Questions

7.6

CLOUD COMPUTING AND BI A. SERVICE-ORIENTED DSS B. DATA-AS-A-SERVICE (DAAS) C. INFORMATION-AS-A-SERVICE (INFORMATION ON DEMAND) (IAAS) D. ANALYTICS-AS-A-SERVICE (AAAS)  Section 7.6 Review Questions

7.7

IMPACTS OF ANALYTICS IN ORGANIZATIONS: AN OVERVIEW A. NEW ORGANIZATIONAL UNITS B. RESTRUCTURING BUSINESS PROCESSES AND VIRTUAL TEAMS C. JOB SATISFACTION D. JOB STRESS AND ANXIETY

2 .

E.

ANALYTICS’ IMPACT ON MANAGERS’ ACTIVITIES AND THEIR PERFORMANCE  Section 7.7 Review Questions

7.8

ISSUES OF LEGALITY, PRIVACY, AND ETHICS A. LEGAL ISSUES B. PRIVACY 1.

Collecting Information about Individuals 2.

Mobile User Privacy 3.

Homeland Security and Individual Privacy C. RECENT TECHNOLOGY ISSUES IN PRIVACY AND ANALYTICS D. ETHICS IN DECISION MAKING AND SUPPORT  Section 7.8 Review Questions

7.9

AN OVERVIEW OF THE ANALYTICS ECOSYSTEM A. ANALYTICS INDUSTRY CLUSTERS B. DATA INFRASTRUCTURE PROVIDERS C. DATA WAREHOUSE INDUSTRY D. MIDDLEWARE/BI PLATFORM INDUSTRY E. DATA AGGREGATORS/DISTRIBUTORS F. ANALYTICS-FOCUSED SOFTWARE DEVELOPERS G. REPORTING/ANALYTICS H. PREDICTIVE ANALYTICS I. PRESCRIPTIVE ANALYTICS J. APPLICATION DEVELOPERS OR SYSTEM INTEGRATORS: INDUSTRY SPECIFIC OR GENERAL K. ANALYTICS USER ORGANIZATIONS L. ANALYTICS INDUSTRY ANALYSTS AND INFLUENCERS M. ACADEMIC PROVIDERS AND CERTIFICATION AGENCIES  Section 7.9 Review Questions

Chapter Highlights Key Terms Questions for Discussion Exercises Teradata University Network (TUN) and Other Hands-On Exercises ⧫ End of Chapter Application Case: Southern States Cooperative Optimizes Its Catalog Campaign  Questions for the Case References

TEACHING TIPS/ADDITIONAL INFORMATION

3 .

       

How to wrap up your course is a personal decision. A book can support your approach, but specific content is less important at this point in the course than it was earlier.

The primary purpose of this chapter is to introduce several emerging technologies that will provide new opportunities for application and extension of business intelligence and analytics techniques. In particular, the impact of cloud and mobile computing are prominently discussed, as well as location-based geospatial analytics technologies.

Another main theme is the increasing use of analytics for consumers, including recommendation engines. Students will find this relevant, since much of their on-line lives involve their roles as consumers, so they have real-world experience with these engines.

This chapter also explores the individual, organizational, and societal impacts of these technologies, especially the ethical, legal, and privacy concerns related to analytics implementation. After describing many of the emerging technologies or application domains, we will focus on organizational issues.

Also, many of the students in the typical undergraduate BI course will start their professional careers a few weeks after you cover this chapter. They care deeply about how Internet and mobile-based analytics will affect them and will enjoy an active discussion of this topic. (Once the graduating seniors break the ice, students who don’t fit that category will chime in, too.)

ANSWERS TO END OF SECTION REVIEW QUESTIONS

     

Section 7.1 Review Questions 1. Why perform consumer analytics?

Consumer analytics helps a company’s customers make better purchasing and usage decisions. OG&E is using it to help conserve on energy usage and ultimately to delay their need to build new fossil fuel generation plants. This helps both OG&E (by reducing costs), its customers (by reducing their energy costs), and the environment as a whole (by reducing pollution from fossil fuels).

2. What is meant by dynamic segmentation?

Dynamic segmentation refers to real-time or near-real-time customer segmentation analytics that will enhance their understanding about individuals’ responses to the price signals and identify the best customers to be targeted with specific marketing campaigns.

3. How does geospatial mapping help OG&E? Using geospatial mapping and visual analytics, OG&E views a near–real-time version of data about its energy-efficient prospects spread over geographic areas and comes up with marketing initiatives that are most suitable for these customers. Geospatial mapping gives OG&E an easy way to narrow down to the specific customers in a 4 .

geographic region based on their meter usage. In addition, OG&E can find noncommunicating smart meters, track the outages, and weather overlay of their services.

4. What types of incentives might the consumers respond to in changing their energy use?

OG&E has started working on consumer-oriented efficiency programs to shift the customer’s usage out of peak demand cycles. OG&E is targeting customers with its smart hours plan. This plan encourages customers to choose a variety of rate options sent via phone, text, or e-mail. These rate options offer attractive summer rates for all other hours apart from the peak hours of 2 p.m. to 7 p.m. OG&E is making an investment in customers by supplying a communicating thermostat that will respond to the price signals sent by OG&E and help customers in managing their utility consumption. OG&E also educates its customers on their usage habits by providing 5-minute interval data every 15 minutes to the demand-responsive customers.

Section 7.2 Review Questions 1. How does traditional analytics make use of location-based data?

Traditional analytics produce visual maps that are geographically mapped and based on the traditional location data, usually grouped by the postal codes. The use of postal codes to represent the data is a somewhat static approach for achieving a higher level view of things.

2. How can geocoded locations assist in better decision making?

They help the user in understanding “true location-based” impacts, and allow them to view at higher granularities than that offered by the traditional postal code aggregations. Addition of location components based on latitudinal and longitudinal attributes to the traditional analytical techniques enables organizations to add a new dimension of “where” to their traditional business analyses, which currently answer questions of “who,” “what,” “when,” and “how much.” By integrating information about the location with other critical business data, organizations are now creating location intelligence (LI).

3. What is the value provided by geospatial analytics?

Geospatial analysis gives organizations a broader perspective and aids in decision making. Location intelligence (LI) is enabling organizations to gain critical insights and make better decisions by optimizing important processes and applications. By incorporating demographic details into locations, retailers can determine how sales vary by population level and proximity to other competitors; they can assess the demand and efficiency of supply chain operations. Consumer product companies can 5 .

identify the specific needs of the customers and customer complaint locations, and easily trace them back to the products. Sales reps can better target their prospects by analyzing their geography.

4. Explore the use of geospatial analytics further by investigating its use across various sectors like government census tracking, consumer marketing, and so forth.

Students’ answers will vary.

Section 7.3 Review Questions 1. What are the various options that CabSense provides to users?

CabSense provides an interactive map based on current user location obtained from the mobile phone’s GPS locator to find the best street corners for finding an open cab.

It also provides a radar view that automatically points the right direction toward the best street corner. The application also allows users to plan in advance, set up date and time of travel, and view the best corners for finding a taxi. Furthermore, CabSense distinguishes New York’s Yellow Cab services from the for-hire vehicles and readily prompts the users with relevant details of private service providers that can be used in case no Yellow Cabs are available.

2. Explore more transportation applications that may employ location-based analytics.

Another app that is mentioned in the text is one deployed in Pittsburgh, Pennsylvania, and developed in collaboration with Carnegie Mellon University. This app, called ParkPGH, includes predictive capabilities to estimate parking availability. It calculates the number of spaces available in downtown Pittsburgh parking lots and garages. (Student answers will vary.)

3. Briefly describe how the data are used to create profiles of users.

Profiles are often built based on users’ behavior. The behavior is tracked by the system based on user clicks, mobile locations, purchase histories, email connections, social network actions, etc. In this way, companies can construct a profile of the user’s habits, interests, spending patterns, and favorite locations.

4. What other applications can you imagine if you were able to access cell phone location data? Do a search on location-enabled services.

Students’ answers will vary.

6 .

Section 7.4 Review Questions 1. List the types of approaches used in recommendation engines.

Two basic approaches that are employed in the development of recommendation systems are collaborative filtering and content filtering.

2. How do the two approaches differ?

In collaborative filtering, the recommendation system is built based on the individual user’s past behavior by keeping track of the previous history of all purchased items.

Collaborative filtering involves aggregating the user-item profiles to generate user ratings matrices, and can take a user-based approach in which the users take the main role. Collaborative filtering systems require huge amounts of existing data on useritem preferences to make appropriate recommendations, and so are subject to the “cold start” problem.

By contrast, content filtering approaches first consider specifications and characteristics of items. Then content-based individual user profiles are built to store the information about the characteristics of specific items that the user has rated in the past. Content-based filtering involves using information tags or keywords in fetching detailed information about item characteristics and restricts this process to a single user, unlike collaborative filtering, which looks for similarities between various user profiles.

In short, collaborative filtering makes recommendations based on what other consumers like you chose, whereas content filtering makes recommendations based on similar items you chose in the past.

3. Can you identify specific sites that may use one or the other type of recommendation system?

Amazon, Facebook, and LinkedIn all use collaborative filtering. Pandora uses content-based filtering.

Section 7.5 Review Questions 1. Define Web 2.0.

Web 2.0 is the popular term for describing advanced Web technologies and applications, including blogs, wikis, RSS, mashups, user-generated content, and social networks. A major objective of Web 2.0 is to enhance creativity, information sharing, and collaboration.

7 .

2. List the major characteristics of Web 2.0.

The following are representative characteristics of the Web 2.0 environment: • • • • • • • • •

The ability to tap into the collective intelligence of users. The more users who contribute, the more popular and valuable a Web 2.0 site becomes.

Data is made available in new or never-intended ways. Web 2.0 data can be remixed or “mashed up,” often through Web service interfaces, much the way a dance-club DJ mixes music.

Web 2.0 relies on user-generated and user-controlled content and data.

Lightweight programming techniques and tools let nearly anyone act as a Web site developer.

The virtual elimination of software-upgrade cycles makes everything a perpetual beta or work-in-progress and allows rapid prototyping, using the Web as an application development platform.

Users can access applications entirely through a browser.

An architecture of participation and digital democracy encourages users to add value to the application as they use it.

A major emphasis on social networks and computing.

Strong support of information sharing and collaboration.

3. What new business model has emerged from Web 2.0?

New business models stress collaboration with customers, partners, and suppliers, as well as among internal users. Businesses are also much more data-driven utilizing metrics on behaviors and sentiments of the participants. Agility (perpetual beta) and global reach are other Web 2.0 contributors to the evolving business models.

4. Define social network.

A social network is a place where people create their own space, or homepage, on which they write blogs (Web logs); post pictures, videos, or music; share ideas; and link to other Web locations they find interesting. In addition, members of social networks can tag the content they create and post it with keywords they choose themselves, which makes the content searchable.

5. List some major social network sites.

Facebook, LinkedIn, Orkut, Google+.

Section 7.6 Review Questions 1. Define cloud computing. How does it relate to PaaS, SaaS, and IaaS?

8 .

Cloud computing offers the possibility of using software, hardware, platform, and infrastructure, all on a service-subscription basis. Cloud computing enables a more scalable investment on the part of a user. Like PaaS, etc., cloud computing offers organizations the latest technologies without significant upfront investment.

In some ways, cloud computing is a new name for many previous related trends: utility computing, application service provider grid computing, on-demand computing, software as a service (SaaS), and even older centralized computing with dumb terminals. But the term cloud computing originates from a reference to the Internet as a “cloud” and represents an evolution of all previous shared/centralized computing trends.

2. Give examples of companies offering cloud services.

Companies offering such services include 1010data, LogiXML, and Lucid Era. These companies offer feature extract, transform, and load capabilities as well as advanced data analysis tools. Other companies, such as Elastra and Rightscale, offer dashboard and data management tools that follow the SaaS and DaaS models.

3. How does cloud computing affect business intelligence?

Cloud-computing-based BI services offer organizations the latest technologies without significant upfront investment.

4. What are the three service models that provide the foundation to service-oriented DSS? The three service models are data-as-a-service (DaaS), information-as-a-service (IaaS), and analytics-as-a-service (AaaS).

5. How does DaaS change the way data is handled?

In the DaaS model, the actual platform on which the data resides doesn’t matter. Data can reside in a local computer or in a server at a server farm inside a cloud-computing environment. With DaaS, any business process can access data wherever it resides.

Customers can move quickly thanks to the simplicity of the data access and the fact that they don’t need extensive knowledge of the underlying data.

6. What is MaaS? What does it offer to businesses?

MaaS stands for models-as-services. This subset of IaaS provides a collection of industry-specific business processes, reports, dashboards, and other service models for key industries (e.g., banking, insurance, and financial markets) to accelerate enterprise business initiatives for business process optimization and multi-channel transformation.

9 .

7. Why is AaaS cost effective?

AaaS in the cloud has economies of scale and scope by providing many virtual analytical applications with better scalability and higher cost savings. The capabilities that a service orientation (along with cloud computing, pooled resources, and parallel processing) brings to the analytic world enable cost-effective data/text mining largescale optimization, highly-complex multi-criteria decision problems, and distributed simulation models.

8. Why is MapReduce mentioned in the context of AaaS? As we learned from Chapter 6, MapReduce is a programming model that allows the processing of large-scale data analysis problems to be distributed and parallelized.

High performance is achieved by breaking the processing into small units of work that can be run in parallel across the hundreds, potentially thousands, of nodes in the cluster. This fits very well with the AaaS model, based on a service orientation of analytics functionality in a cloud platform involving highly distributed computing.

Section 7.7 Review Questions 1. List the impacts of analytics on decision making.

Analytics can change the manner in which many decisions are made and can consequently change managers’ jobs. They can help managers gain more knowledge, experience, and expertise, and consequently enhances the quality and speed of their decision making. In particular, information gathering for decision making is completed much more quickly when analytics are in use. This affects both strategic planning and control decisions, changing the decision-making process and even decision-making styles.

2. List the impacts of analytics on other managerial tasks.

Less expertise (experience) is required for making many decisions. Faster decision making is possible because of the availability of information and the automation of some phases in the decision-making process. Less reliance on experts and analysts is required to provide support to top executives. Power is being redistributed among managers. (The more information and analysis capability they possess, the more power they have.) Support for complex decisions allows decisions to be made faster and of better quality. Information needed for high-level decision making is expedited or even self-generated. Automation of routine decisions or phases in the decisionmaking process (e.g., for frontline decision making and using ADS) may eliminate some managers, especially middle level managers. Routine and mundane work can be done using an analytic system, freeing up managers and knowledge workers to do more challenging tasks.

10 .

3. Describe new organizational units that are created because of analytics.

One change in organizational structure is the possibility of creating an analytics department, a BI department, or a knowledge management department in which analytics play a major role. This special unit can be combined with or replace a quantitative analysis unit, or it can be a completely new entity.

4. How can analytics affect restructuring of business processes?

When a company introduces a data warehouse and BI, the information flows and related business processes (e.g., order fulfillment) are likely to change because information flows change. For example, before IBM introduced e-procurement, it restructured all related business processes, including decision making, searching inventories, reordering, and shipping.

5. Describe the impacts of ADS systems.

Automated decision support (ADS) applications will probably have the following impacts: • reduction of middle management • empowerment of customers and business partners • improved customer service • increased productivity of help desks and call centers 6. How can analytics affect job satisfaction?

Analytics has positive or negative impacts. Although many jobs may be substantially enriched by analytics, other jobs may become more routine and less satisfying. But a study by Davenport and Harris found that employees using ADS systems, especially those who were empowered, were more satisfied with their jobs.

Section 7.8 Review Questions 1. List some legal issues of analytics.

• • • • •

What is the value of an expert opinion in court when the expertise is encoded in a computer?

Who is liable for wrong advice (or information) provided by an intelligent application?

What happens if a manager enters an incorrect judgment value into an analytic application and the result is damage or a disaster?

Who owns the knowledge in a knowledge base?

Can management force experts to contribute their expertise?

2. Describe privacy concerns in analytics.

11 .

In general, privacy is the right to be left alone and the right to be free from unreasonable personal intrusions. The Internet, in combination with large-scale databases, has created an entirely new dimension of accessing and using data. The inherent power in systems that can access vast amounts of data can be used for the good of society. For example, by matching records with the aid of a computer, it is possible to eliminate or reduce fraud, crime, government mismanagement, tax evasion, welfare cheating, family-support filching, employment of illegal aliens, and so on. The same is true on the corporate level. Private information about employees may aid in better decision making, but the employees’ privacy may be affected.

Similar issues are related to information about customers.

3. Explain privacy concerns on the Web. The Internet offers a number of opportunities to collect private information about individuals. Here are some of the ways it can be done: • • • • • • •

By reading an individual’s newsgroup postings By looking up an individual’s name and identity in an Internet directory By reading an individual’s e-mail By wiretapping wireline and wireless communication lines and listening to employees By conducting surveillance on employees By asking an individual to complete Web site registration By recording an individual’s actions as he or she navigates the Web with a browser, using cookies or spyware

The implications for online privacy are significant. The ability of law enforcement agencies to authorize installation of pen registers and trap-and-trace devices has increased. The U.S. PATRIOT Act also broadens the government’s ability to access student information and personal financial information without any suspicion of wrongdoing by attesting that the information likely to be found is pertinent to an ongoing criminal investigation.

4. List ethical issues in analytics.

Representative ethical issues that could be of interest in MSS implementations include the following: • • • • • • • •

Electronic surveillance Ethics in DSS design Software piracy Invasion of individuals’ privacy Use of proprietary databases Use of intellectual property such as knowledge and expertise Exposure of employees to unsafe environments related to computers Computer accessibility for workers with disabilities 12 .

• • • • •

Accuracy of data, information, and knowledge Protection of the rights of users Accessibility to information Use of corporate computers for non-work-related purposes How much decision making to delegate to computers

Section 7.9 Review Questions 1. Identify the nine clusters in the analytics ecosystem.

1.

2.

3.

4.

5.

6.

7.

8.

9.

Data Infrastructure Providers Data Warehouse Industry Middleware Industry Data-Aggregated Distributers Analytics-Focused Software Developers Analytics Industry Analysts and Influencers Academic Providers and Certification Agencies Analytics User Organizations Application Developers: Industry Specific or General

2. Which clusters represent technology developers?

Most involve developers to some degree. The data infrastructure providers, data warehouse industry, middleware industry, and data aggregators all develop technologies that enable analytics applications. Then there are applications developers, especially those focused on analytics applications. Although the user organizations, academic providers, and influencers will include many non-developers, even these clusters will also have some developers in them.

3. Which clusters represent technology users?

The most prominent one is analytics user organizations. But others will include analysts and influencers as well as academic providers and certification agencies.

Indeed, all stakeholders are users of some sort.

4. Give examples of an analytics professional moving from one cluster to another.

One prominent example is people who move into the analysts and influencers cluster.

These will have been either users or developers or perhaps from other clusters as well. A similar migration can bring people into the Academic Provider category, where experts in the field from some of these other clusters become prominent members of this cluster and thereby bring knowledge and expertise to a wider audience. (This answer will vary by student.)

13 .

ANSWERS TO APPLICATION CASE QUESTIONS FOR DISCUSSION

 

Application Case 7.1: Great Clips Employs Spatial Analytics to Shave Time in Location Decisions 1.

How is geospatial analytics employed at Great Clips?

Great Clips depends on a growth strategy that is driven by rapidly opening new stores in the right locations and markets. They use geospatial analysis to help analyze the locations based on the requirements for a potential customer base, demographic trends, and sales impact on existing franchises in the target location.

They use their Alteryx-based solution to evaluate each new location based on demographics and consumer behavior data, aligning with existing Great Clips customer profiles and the potential revenue impact of the new site on the existing sites.

2.

What criteria should a company consider in evaluating sites for future locations?

Major criteria include potential customer base, demographic trends, and sales impact on existing franchises in the target location.

3.

Can you think of other applications where such geospatial data might be useful?

Geospatial data can be used to help customers find the right location (for example, the closest Great Clips location). It is certainly relevant for other companies in a variety of industries. Analyzing customer profiles and applying these to geographic information can assist with many retail firms. Another possibility is utilizing geospatial analysis to find locations for manufacturing facilities; in this case you would be looking for supplier and raw materials’ locations more than customer locations. In the consumer market, geospatial analysis can help users in a variety of applications; for example finding the best locations for restaurants or stores catering to the customer’s desires. (Student answers will vary.)

Application Case 7.2: Quiznos Targets Customers for Its Sandwiches 1.

How can location-based analytics help retailers in targeting customers?

Location-based behavioral targeting can help to narrow the characteristics of users who are most likely to utilize a retailer’s services or products. This sort of analytics would typically target the tech-savvy and busy consumers of the company in question.

2.

Research similar applications of location-based analytics in the retail domain.

Student answers will vary.

14 .

Application Case 7.3: A Life Coach in Your Pocket 1.

Search online for other applications of consumer-oriented analytical applications.

Student answers will vary.

2.

How can location-based analytics help individual consumers?

If a user on a smart phone enters data, the location sensors of the phone can help find others in that location who are facing similar circumstances, as well as local companies providing services and products that the consumer desires. The user can thus see what others in their location are choosing, and the opportunities for meeting his or her needs. Conversely, the user’s behaviors and choices can then contribute information to other consumers in the same location.

3.

How can smartphone data be used to predict medical conditions?

Via smartphones, users can enter data such as gender, age, weight, height, and the location where he or she lives. From this, an app can create a behavior profile compared with health data from the CDC. Predictive analytics can calculate life expectancy of the user, who can begin discovering health opportunities. Also, most smartphones are equipped with accelerometers and gyroscopes to measure jerk, orientation, and sense motion. Muscle motions may be used to predict the progression of disorders such as Parkinson’s disease, as well as tracking exercise activities.

4.

How is ParkPGH different from a “parking space-reporting” app?

ParkPGH does more than just report current parking-space availability. It is also capable of predicting future parking availability. Depending on historical demand and current events, the app is able to provide information on which lots will have free space by the time the driver gets to the destination. The app’s underlying algorithm uses data on current events around the area—for example, a basketball game—to predict an increase in demand for parking spaces later that day, thus saving commuters valuable time searching for parking spaces in the busy city.

ANSWERS TO END OF CHAPTER QUESTIONS FOR DISCUSSION

  

1. What are the potential benefits of using geospatial data in analytics? Give examples.

Geospatial analytics gives organizations a broader perspective and aids in decision making. Geospatial data helps companies with managing operations, targeting customers, and deciding on promotions. It also helps consumers directly, making use 15 .

of integrated sensor technologies and global positioning systems installed in their smartphones. Using geospatial data, companies can identify the specific needs of the customers and customer complaint locations, and easily trace them back to the products. Another example is in the telecommunications industry, where geospatial analysis can enable communication companies to capture daily transactions from a network to identify the geographic areas experiencing a large number of failed connection attempts of voice, data, text, or Internet.

2. What type of new applications can emerge from knowing locations of users in real time? What if you also knew what they have in their shopping cart, for example?

One prominent application is in the emerging area of reality mining, which uses location-enabled devices for finding nearby services, locating friends and family, navigating, tracking of assets and pets, dispatching, and engaging in sports, games, and hobbies. Adding shopping cart knowledge will enhance the application’s ability to provide targeted information to a customer; for example, the app could find prices for similar products in nearby stores.

3. How can consumers benefit from using analytics, especially based on location information?

Consumer-oriented analytics based on location information fit into two major categories: (a) GPS navigation and data analysis, and (b) historic and current location demand analysis. Consumers benefit from analytics-based applications in many areas, including fun and health, as well as enhanced personal productivity.

4. “Location-tracking-based profiling (reality mining) is powerful but also poses privacy threats.” Comment.

Students’ answers will differ. Privacy threats relate to user-profiling, intrusive use of personal information, and not being able to control what is being collected.

5. Is cloud computing “just an old wine in a new bottle”? How is it similar to other initiatives? How is it different?

In some ways, cloud computing is a new name for many previous related trends: utility computing, application service provider grid computing, on-demand computing, software as a service (SaaS), and even older centralized computing with dumb terminals. But the term cloud computing originates from a reference to the Internet as a “cloud” and represents an evolution of all previous shared/centralized computing trends.

Cloud computing offers the possibility of using software, hardware, platform, and infrastructure, all on a service-subscription basis. Cloud computing enables a more scalable investment on the part of a user. Like PaaS, etc., cloud-computing offers organizations the latest technologies without significant upfront investment.

16 .

6. Discuss the relationship between mobile devices and social networking.

Mobile social networking enables social networking where members converse and connect with one another using cell phones or other mobile devices.

7. Some say that analytics in general, and ES in particular, dehumanize managerial activities, and others say they do not. Discuss arguments for both points of view.

Students’ answers will differ. A common criticism of traditional data-processing systems is their negative effects on people’s individuality. Such systems are criticized as being impersonal: They may dehumanize and depersonalize activities that have been computerized because they reduce or eliminate the human element that was present in noncomputerized systems. Some people feel a loss of identity; they feel like just another number. On the bright side, one of the major objectives of analytics is to create flexible systems and interfaces that allow individuals to share their opinions and knowledge and work together with computers. Despite all these efforts, some people are still afraid of computers, so they are stressed; others are mostly afraid of their employers watching what they do on the computer.

8. Diagnosing infections and prescribing pharmaceuticals are the weak points of many practicing physicians (according to E. H. Shortliffe, one of the developers of MYCIN). It seems, therefore, that society would be better served if MYCIN (and other ES) were used extensively, but few physicians use ES. Answer the following questions: a. Why do you think such systems are little used by physicians?

Students’ answers will differ. Some reasons are: physicians do not understand and therefore do not trust the ES; malpractice insurance does not cover recommendations made by MYCIN; administrators will not invest in it; physicians fear they will be replaced or earn less. b. Assume that you are a hospital administrator whose physicians are salaried and report to you. What would you do to persuade them to use ES? Students’ answers will differ. Students should identify a combination of positive (more money) and negative (calm their fears) motivators. c. If the potential benefits to society are so great, can society do something that will increase doctors’ use of such analytic systems?

Probably not--or at least not yet.

9. What are some of the major privacy concerns in employing analytics on mobile data?

17 .

Loss of privacy is a key concern in employing analytics on mobile data. If someone can track the movement of a cell phone, the privacy of that customer is a big issue.

Some of the app developers claim that they only need to gather aggregate flow information, not individually identifiable information. But many stories appear in the media that highlight violations of this general principle. Sometimes, retailers provide information on their customers to the federal government, in violation of their stated privacy policies.

Legally, the right of privacy is not absolute. The public’s right to know is superior to the individual’s right to privacy. For example, the USA PATRIOT Act broadens the government’s ability to access student information and personal financial information without any suspicion of wrongdoing. Location information from devices has been used to locate victims and criminals, so provides a social good. But at what point is the information not the property of the individual?

10. How can one move from a technology provider cluster to a user cluster?

Individuals in a technology provider cluster move into a user cluster simply by utilizing the technology and tools for solving their own business problems or meeting their consumer/personal needs. In effect, everyone is potentially in the user cluster.

ANSWERS TO END OF CHAPTER APPLICATION CASE QUESTIONS

 

1. What is the main business problem faced by Southern States Cooperative?

Southern States Cooperative is one of the largest farmer-owned cooperatives in the United States, with over 300,000 farmer members being served at over 1,200 retail locations across 23 states. The firm realized the need to continually optimize marketing activities by gaining insights into its customers. It wanted to maintain and extend its success by better targeting the right customers in its direct-marketing campaigns. Specifically, the main business challenges included determining the right set of customers to be targeted for mailing the catalogs, choosing the right combination of storage keeping units (SKUs) to be included in the catalog, cutting down mailing costs, and increasing customer response.

2. How was predictive analytics applied in the application case?

SSC first built a predictive model to determine which catalogs the customer was most likely to prefer. The data for the analysis included Southern States’ historical customer transaction data analyzed on the basis of recency, frequency, and monetary value of customer transactions (known as RFM analysis). Sample data was analyzed via logistic regression models, a decision tree, and a random forest model. Based on the predictive modeling results, an incremental revenue model based on linear

18 .

regression was used to estimate revenue per customer. In sum, the predictive model aimed to determine the customer’s catalog use in purchasing selected items.

3. What problems were solved by the optimization techniques employed by Southern States Cooperative?

An optimization model was developed to maximize the total income from mailing the catalogs to customers. This model was solved using genetic algorithms, aiming to maximize the combined selection of the catalog SKUs and the customers to whom the catalog should be sent to result in increased response, at the same time increasing the revenues and cutting down the mailing costs. In sum, the prescriptive model helped the marketing department prepare the customized catalogs containing the SKUs that suited the targeted customer needs, resulting in better revenue generation.

19 .