Instructor Manual, Product Analytics
Applied Data Science for Actionable Consumer Insights, by Joanne Rodrigues Companion site for the manual: actiondatascience.com Course Description: The goal of the course is to prepare students for real-life business applications of product data science in industry. There are over 40 Zettabytes of consumer data created. The sheer amount of data is staggering. How do we make sense of all of these consumer actions? Can we influence or change consumer behavior to meet our product directives? Many data science courses and books only focus on the modeling component of data science. However, in business contexts, the modeling component is often at the end of a long funnel that starts with theory building. This course will teach you not only how to work with data, but also how to contextual, theorize, conceptualize, and operationalize theories quantitatively to generate actionable business insights. The aim of the course is to provide an overview of qualitative and quantitative toolkit used in product analytics, with an R programming component. The course takes a novel approach to product analytics/product data science by starting with qualitative theory building techniques and moving to quantitative techniques to validate theory propositions and test hypotheses all within the context of the modern web or mobile product. Course Competencies/Learning Goals: ● Understand common analytics pitfalls related to consumer behavior ● Learn how to develop theories about consumer behavior ● Understand concept building, conceptualization, and operationalization of concepts ● Understand quantitative measurement techniques of conceptual ideas ● Develop core metrics and effective KPIs for user analytics in a web product ● Understand statistical inference and the differences in correlation and causation ● Build effective A/B tests to validate hypotheses ● Build intuitive predictive models to capture user behavior in a product ● Tease out causal effects from observational data using modern, quasi-experimental designs and statistical matching ● Improve response of marketing campaigns through uplift modeling ● Project business costs and product populations with advanced analytics techniques ● Learn how to carry out all of these quantitative techniques in the R programming language The remainder of this document provides chapter-by-chapter supplementary material, including: ● Learning aims ● Lists of new key terms ● Course suggestions ● Comprehension questions, with solutions
● ● ● ●
Exercises, including project ideas Outside resources and readings Data Guide (at the end of the document to aid with data exercises) Slides for Chapters 1-6
Product Analytics Course Plan: ❖ Theory Building: Weeks 1-3 ➢ The goal of the first three weeks of the course is to lay the groundwork for how to approach product analytics questions. The focus should be on understanding theory building techniques and sociological and psychological theory of human behavior change. ❖ Basic Statistical Methods: Weeks 4-6 ➢ The goal of the next three weeks of the course is to understand basic statistical concepts and metric development. These three weeks can be paired with some data cleaning exercises in R to help students better work with consumer analytics data. ❖ Basic Predictive Methods: Weeks 7-9 ➢ The goal of the next three weeks of the course is to learn predictive methods in machine learning and demography. It’s best to pair this learning with some traditional machine learning texts to aid students in understanding the nuts and bolts of predictive modeling. ❖ Causal Inference Methods: Weeks 10-13 ➢ The goal of the final three weeks of the course is to go over causal inference methods. It’s best to focus on the techniques discussed in these chapters as they are rigorous and advanced content. Assessments: The preferred assessments for this class would be in the form of projects as they are applied to real-life business contexts. 1. Weekly Quizzes/Homework: 40% 2. Project 1: 20% Building a model of a web product. Observe a web product and create a theory with hypotheses and detail how to test them. In the context of this theory, operationalize at least two concepts and discuss your strategy and thinking. 3. Project 2: 40% Find a natural experiment or quasi-experiment in real-life and analyze it. Write a paper that includes the set-up, your theory and hypotheses, the mechanisms of your causal link, and the actionable insights from this relationship. Calculate the ATE, discuss all your assumptions, and explain the relevance of the results. Run some robustness checks. Note: The section “Basic, Predictive and Causal Inference Methods in R” includes R programming examples to apply theoretical concepts and the programming exercises will be included with the theory chapters. You can assign reading of these three chapters with the
theory chapters—i.e., Chapter 14 during weeks 4-6 and Chapter 15 during weeks 7-9 and Chapter 16 during weeks 10-13. Data Resources: Here are some open-source sites where learners can find data sets or search for data sets to practice the skills learned in the book: 1. Kaggle Competition Data: https://www.kaggle.com/datasets 2. Census and government data: https://usa.ipums.org/usa/ 3. Amazon Web Services (AWS) registry: https://registry.opendata.aws/ 4. Ckan: https://ckan.org/ 5. Free Code Camp: https://freeCodeCamp.org 6. Yelp open data sets: https://www.yelp.com/dataset 7. Pew research data: https://www.pewresearch.org/internet/datasets/ 8. Find data: https://datasetsearch.research.google.com/
Section I: Qualitative Methods
Chapter 1: Data in Action: A Model of a Dinner Party Learning Aims: The goal of this chapter is to change the way in which the reader approaches product analytics. ● Understand some common pitfalls in product analytics ● Think about web products and social behavior in a new way, as a social process and not as a problem to be solved ● Explore tools to help us understand product analytics as a social process Key Terms: causal inference, predictive inference, outcomes, norms, problem, social process, confirmation bias, bayes rule, counterfactual Concepts and Themes: ● Understand that all behavior in an online product is human behavior, following the same basic presets of other types of human behavior ● Online behavior is real and should not be aggregated away immediately, but thought of as individual and specific human behaviors that you might observe in real life ● A common pitfall in consumer analytics is to focus on disparate aggregate facts on user behavior without having a general theory about behavior in a web product ● There are 6 basic misconceptions about consumer behavior in a web/mobile product: ○ Social behavior is a process, not a problem to be solved ○ Social systems are open systems, meaning there are omitted or unmeasurable variables that can affect outcomes
○ ○
When exploring social behavior, there are often no clear and defined outcomes Social systems are rampant with problems of incomplete or one-sided information ○ Social systems consist of millions of potential behaviors ○ Inferring causation (or inferring why something happens) is almost impossible Course Suggestions: ● Have the students read the beginning sections of the McKinsey reports on data science and the future of analytics prior to the first class. See outside resources and readings. ● One could start with this video on TED Talk, by Sociologist Stefena Broadbent, on social processes online: https://www.ted.com/talks/stefana_broadbent_how_the_internet_enables_intimac y. It’s important to understand that consumer behavior on the internet is real, social, and real-time. ● Discuss the core concepts in sociology of social norms, structures, culture, and social behavior. Refer to the Giddens reading to build some content for class discussion. ● Next, you could discuss the six core elements of social processes in detail, fleshing them out with the dinner party example. ● Then, use the discussion questions to ask what additional elements might separate a social process from a problem. Also, start using this framework to examine other social processes. Outside Resources and Readings: I. Readings related to analytics in the 21st century ● Henke, Nicolaus, Bughin, Jacques, Chui, Michael, Manyika, James, Saleh, Tamin, Wiseman, Bill, and Sethupathy, Guru. The Age of Analytics: competing in a data-driven world. McKinsey Global Institute, 2003. https://www.mckinsey.com/businessfunctions/mckinsey-analytics/our-insights/the-age-of-analytics-competing-in-a-datadriven-world ● Manyika, James, Chui, Michael, Brown, Brad, Bughin, Jacques, Dobbs, Richard, Roxburg, Charles and Byers, Angela. Big data: the next frontier for innovation, competition, and productivity. McKinsey Global Institute, 2011. https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-thenext-frontier-for-innovation II. Readings related to social processes ● Giddens, Anthony, Duneier, Mitchel, Appelbaum, Richard, and Carr, Deborah. Introduction to Sociology. New York: W. W. Norton, 2009. ● https://opentextbc.ca/principlesofeconomics/chapter/16-1-the-problem-of-imperfectinformation-and-asymmetric-information/ III. Readings related to algorithms and algorithmic development ● Applegate, David, Bixby, Robert, Chvátal, Vasek, and Cook, William. The Traveling Salesman Problem. Princeton: Princeton University Press, 2006. ● Copeland, Jack. "The Turing Test", Minds and Machines, 10, no. 4 (2000): 519-539.
● IV. ●
Crevier, Daniel. AI: The Tumultuous Search for Artificial Intelligence. New York, NY: BasicBooks, 1993. Introductory statistics textbook Wheelan, Charles J. Naked Statistics: Stripping the Dread from the Data. New York: W.W. Norton, 2013.
Comprehension Questions: (1) Why is weaving a story together based on disparate facts not a good idea? (a) We generalize from a narrow set of conditions (b) We fail to account for selection bias (c) We like to infer causal relationships when they may not exist (d) All of the above Answer: D (2) What is causal inference? (a) Inference based on randomness (b) Inference based on external conditions (c) Inference related to understanding why something happens (d) Inference related to predicting a phenomenon Answer: C (3) What is a counterfactual? (a) An opposing view (b) An examination of what would happen in a circumstance where another condition was occurring (c) An examination of circumstance when condition A precedes condition B (d) A contradictory statement Answer: B (4) Why is inferring causation almost impossible? (a) Causal inference is a higher level process than inference from randomization. (b) Causal inference is more difficult than predictive inference. (c) Causal inference is difficult, unless there we have designed and implemented an experiment properly, because there could always be a spurious relationship between variables. (d) This statement is incorrect. Causal inference is possible under more conditions. Answer: C Exercises, Projects, and Discussion Questions: (1) Writing Project: Think of another social process such as a job interview or buying a car. Allow the students to pick a social process of their choosing. Then, have them go through the six elements, similar to the dinner party example in this chapter. You can have the students go through the following process: (1) Define the process, (2) Search for omitted variables, (3) Consider potential outcomes, (4) Explore issues of incomplete information,