Page 1

COINs 2013  Course   Project  9  Report          

Exploration of  Methods  to  Measure  Emotional  Responses  to  User   Interfaces:  A  Case  Study  on  Citrix’s  GoToMeeting   Emanuel  Castillo,  Priscilla  Mendoza1*,  Bhavika  Shah2*,  Yulia  Tammisto   University  of  Cologne,  Albertus-­‐Magnus-­‐Platz,  Cologne  50923,  Germany     Savannah  College  of  Art  and  Design,  3116  Montgomery  Street,  Savannah  31401,  USA   Aalto  University  School  of  Business,  Runebrtginkatu  14-­‐16,  Helsinki  00100,  Finland    

Abstract   Online  social  communities  are  an  invaluable  avenue  for  understanding  user  emotions.  In  this  paper,  our  collaborative   innovation  network  (CoIN)  describes  our  attempt  to  find  methods  suitable  for  measuring  user  emotions  through  sentiment   analysis  of  online  data.  Our  aim  was  to  develop  a  methodology  that  could  help  companies  use  customer  reviews  and  posts   from  social  media  platforms  to  spot  insights  for  improving  product  strategy  and  thus  performance.  In  order  to  conduct  the   analysis  of  the  data,  we  tested  a  number  of  social  network  data  gathering  and  sentiment  analysis  tools  to  be  able  to  choose   the  few  that  were  the  most  suitable  for  measuring  emotions.  During  testing,  we  quickly  saw  the  shortcomings  of  the   quantitative  sentiment  analysis  tools  and  therefore  searched  for  sentiment  analysis  tools  that  would  analyze  qualitative  data   in  order  to  capture  the  full  meaning  of  each  review  and  post.  The  findings  developed  into  new  opportunities  specifically  for   Citrix’s  GoToMeeting  product  team,  but  can  potentially  be  expanded  to  other  companies  using  social  network  analysis  and   sentiment  analysis  on  their  customers’  online  reviews  and  posts  to  understand  and  process  their  information.   Keywords:  Social  Network  Analysis,  Sentiment  Analysis,  Qualitative  Sentiment  Analysis,  User  Interface,  Emotions,  Corporate  Identity,  AC2ID  Test  

   

                                                                                                                      * Corresponding  author.  Tel.:  +1-­‐912-­‐306-­‐6919   E-­‐mail  address:  priscila.mendoza.lopez@gmail.com * Corresponding  author.  Tel.:  +1-­‐281-­‐217-­‐0019   E-­‐mail  address:  bhavika1204@gmail.com

1


1. Introduction Our  research  began  as  an  attempt  to  help  companies  better  understand  and  connect  to  their  customers   through  social  media.  In  order  to  do  this,  our  study  focused  on  the  video  conferencing  software   GoToMeeting  by  Citrix.  Citrix  is  a  multinational  software  company  that  develops  desktop  visualization,   networking,  software  as  a  service  and  cloud  computing  solutions.  GoToMeeting  is  video  conferencing   software  that  helps  customers  participate  in  online  meetings  and  includes  functions  as  desktop  sharing,   one-­‐click  recording  and  HD  faces.  GotoMeeting  transmissions  are  protected  with  high-­‐security   encryption  and  optional  passwords.  Thus,  sales  are  mostly  focused  on  b2b  markets  such  as  employees   of  companies  that  have  purchased  GoToMeeting.  Their  motto  is  “online  meetings  made  easy.”     Citrix’s  management  was  particularly  interested  in  the  emotional  reactions  of  their  end  users  to  their   software  interfaces.  This  would  be  possible  through  Social  Network  Analysis  and  possibly  discovering   the  potential  to  develop  improvements  for  their  product.  Therefore  from  the  very  beginning,  our  team’s   goal  was  to  provide  practical  results  that  could  be  applied  to  improve  business  performance  for  the   company.       We  began  by  developing  a  project  plan  that  would  help  us  to  deliver  implementable  yet  suitable  results   for  application  by  Citrix.  The  initial  goal  of  the  project  was  as  follows:  To  research  users’  emotional   reactions  to  an  interface  design  through  social  media.  However,  after  conducting  initial  research,  we   saw  that  there  was  an  opportunity  to  understand  and  analyze  users’  emotional  reactions  to  spot   innovation  opportunities  for  products  and  companies.     Our  plan  began  by  collecting  discussions,  comments,  and  reviews  about  GoToMeeting  by  end  users  and   experts  on  various  social  media  platforms  to  perform  a  sentiment  analysis.  We  also  gathered  blog  posts,   updates,  and  reviews  posted  by  Citrix  and  GoToMeeting,  and  ran  the  same  sentiment  analysis.  By  doing   both,  we  were  able  to  compare  the  company’s  emotional  appeal  to  the  product  with  users’  emotional   response  to  its  interface.  The  comparison  helped  us  understand  and  describe  the  gap  between  the   company’s  self  image  and  the  way  customers  perceive  the  company.  Knowing  and  understanding  this   gap,  in  turn,  would  help  Citrix  address  the  product  development  and  communications  accordingly  and   to  potentially  minimize  the  disappointment  and  negative  effect  caused  by  the  mismatch.  

2. Data Collection  Channels   According  to  our  plan  we  started  gathered  two  types  of  data:  “internal”  from  Citrix  about  GoToMeeting   and  “external”  from  GoToMeeting’s  users  and  tech  experts  reviewing  and  posting  their  experience  with   the  software.  The  sources  were  searched  throughout  the  Internet.  We  used  the  software  Condor  to   help  us  identify  the  websites  that  mention  and  post  about  GoToMeeting.  Combined  with  Google  Search,   this  gave  us  a  wide  range  of  different  sources  including:  Citrix’s  corporate  pages,  their  blog,  and  their   Facebook  profile;  various  tech  expert  blogs;  and  websites  collecting  user  reviews  on  the  product   (including  Google  Play,  Amazon.com,  AppStore  and  other  smaller  resources.)  We  were  also  able  to   obtain  data  from  Citrix’s  management  about  their  vision  of  the  product  that  we  used  for  analysis.  In   order  to  be  able  to  analyze  the  data  in  our  project  timeframe,  we  limited  the  data  gathered  to  between   2011  to  2013.  A  short  description  of  each  source  group  is  presented  in  the  following  sections.  

2  


2.1. Citrix Corporate  Pages   We  collected  the  mission  and  vision  statements  and  product  descriptions  for  GoToMeeting  from  Citrix’s   corporate  website  (citrix.com),  and  data  for  analysis  of  Citrix’s  image  were  gained  through  reviews  from   Citrix  website  (citrixonline.com).  In  the  corporate  blog  (blogs.citrix.com)  we  also  gathered  updates  and   news  about  GoToMeeting.  Also,  we  collected  all  GoToMeeting  related  reviews  and  manually  copied  the   content  into  an  Excel  file.  We  copied  other  words  found  on  the  website  such  as  the  text  in  sidebars  in   order  to  understand  the  image  and  message  that  Citrix  Online  wants  to  communicate.    

2.2. Facebook –  GoToMeeting  Page   Facebook  has  about  1.19  billion  monthly  active  users  (https://newsroom.fb.com/Key-­‐Facts  (01-­‐23-­‐2014))   and  therefore  plays  a  big  role  in  the  area  of  social  media  analysis.  For  the  purpose  of  our  work,  we   collected  user  posts  on  the  GoToMeeting  Facebook  Page  (https://www.facebook.com/GoToMeeting)   from  January  2012  to  November  2013.  This  source  was  selected  in  order  to  obtain  the  unprompted   experiences  and  emotions  of  users  that  consider  GoToMeeting  as  a  “friend.”  Posts  that  weren’t  related   to  the  GoToMeeting  product,  such  as  posts  about  Halloween  or  Christmas,  were  not  collected.  

2.3. Blogs and  other  general  websites   Our  Condor  and  Google  searches  revealed  a  large  number  of  tech-­‐industry  related  pages  such  as  expert   blogs  or  online  journal  websites  that  wrote  posts  and  reviews  about  GoToMeeting.  We  copied  all  the   material,  articles,  and  reviews  to  our  Excel  data  file  for  further  analysis  in  order  to  learn  what  those   proficient  in  computer  software  were  saying  about  their  experience  with  GoToMeeting.  The  more   detailed  list  of  sources  is  presented  in  Table  1.   Table  1.  Blogs  and  other  general  websites  sources  discussing  GoToMeeting   Source  type  

Source location  

Tech Blog  

http://reviews.cnet.com/conferencing/citrix-­‐gotomeeting/4505-­‐10259_7-­‐31137633.html

Magazine

http://www.forbes.com/companies/citrix-­‐systems/

Employee review  

http://www.indeed.com/cmp/Citrix-­‐Systems-­‐Inc./reviews

Employee review  

http://www.glassdoor.com/Reviews/Citrix-­‐Systems-­‐Reviews-­‐E5432.htm

Employee review  

http://www.careerbliss.com/citrix-­‐systems/reviews/

Tech Rating  Blog  

http://web-­‐conferencing-­‐services.toptenreviews.com/gotomeeting-­‐review.html

Tech Blog  

http://www.itpro.co.uk/631837/citrix-­‐gotomeeting-­‐review/page/0/1

Review aggregator  

http://www.g2crowd.com/products/gotomeeting/reviews

Tech Blog  

http://www.pcmag.com/article2/0,2817,2387935,00.asp

Tech Blog  

http://reviews.cnet.com/conferencing/citrix-­‐gotomeeting/4852-­‐10259_7-­‐31137633-­‐4.html

Tech Blog  

http://www.macworld.com/article/1153703/citrixgotomeeting.html

Tech Blog  

http://www.ondemandreport.com/saas/collaboration-­‐a-­‐conferencing/77-­‐citrix-­‐gotomeeting-­‐review

Journal

http://www.businesscomputingworld.co.uk/review-­‐citrix-­‐gotomeeting/

Review aggregator  

http://community.spiceworks.com/product/21755-­‐gotomeeting

Tech Blog  

http://www.inspectorjones.com/reviews/communication/gotomeeting-­‐citrix/

3  


Table 1.  Blogs  and  other  general  websites  sources  discussing  GoToMeeting  (continues)   Source  type  

Source location  

Tech Blog  

http://www.business-­‐software.com/product/citrix-­‐gotomeeting/

Tech Blog  

http://www.techrepublic.com/blog/product-­‐spotlight/review-­‐gotomeeting-­‐hosted-­‐conferencing-­‐system/

Tech Blog  

http://thenextweb.com

Review aggregator  

http://www.engadget.com/

2.4. Twitter Twitter  was  one  of  the  most  recognizable  social  media  platforms  to  find  emotional  user  responses  and   measure  sentiment.  We  collected  and  analyzed  Twitter  data  concerning  GoToMeeting  and  Citrix   through  tools  that  gathered  and  grouped  the  tweets  instantaneously  (described  in  the  following  section   “Methods”).  This  was  the  only  data  we  did  not  have  to  collect  manually.  The  tweets  helped  us  discover   ephemeral  yet  emotional  user  responses  concerning  GoToMeeting  and  Citrix.  

2.5. Markets (Google  Play,  AppStore,  Amazon.com)   The  digital  products  markets  were  a  source  of  multiple  reviews  about  GoToMeeting.  The  largest   database  was  available  through  the  AppStore  (over  1000  entries),  since  the  GoToMeeting  app  has  been   sold  through  that  avenue  for  a  few  years.  However,  Amazon  has  only  sold  the  app  for  about  a  year  and   less  than  a  year  on  Google  Play.  We  collected  the  reviews  left  in  2011,  2012  and  2013  to  be  able  to  analyze   them  in  conjunction  with  development  and  upgrades  of  the  app.  

2.6. Youtube video  comments   During  our  search  for  data  about  GoToMeeting,  we  found  a  number  of  video  comments  posted  on  the   YouTube  commercials.  We  decided  to  include  data  from  this  channel  since  the  people  left  both   comments  about  their  experiences  with  GoToMeeting  software  as  well  as  critiques  about  the  video.   Overall  we  were  able  to  gather  over  100  video  comments.  Those  comments  that  were  not  relevant  to   GoToMeeting  or  contained  random  comments  were  excluded.  

3. Methodology   To  proceed  to  analysis,  we  first  explored  the  wide  range  of  different  tools  and  tested  them  on  the  data   collected.  The  purpose  was  to  understand  how  each  method  worked  and  to  be  able  to  choose  the  most   suitable  tool  that  would  help  us  reach  and  validate  our  project  goal.  Each  tool  we  found  and  applied  to   our  data  is  presented  in  the  following  subsections  including  a  short  description  and  examples  from  the   data.  

4  


3.1. Social Network  Analysis  tools   3.1.1. Condor   Condor  is  a  data  collection  software  developed  by  Peter  Gloor  that  located  the  websites  that   reference  Citrix  and  GoToMeeting.  We  also  ran  searches  on  Wikipedia  and  Twitter  feeds  that  gave  us   the  most  prominent  actors  that  discuss  Citrix    (Mark  Templeton)  and  GoToMeeting  (Keith  Ferrazzi).   This  information  was  used  to  locate  further  resources  in  our  search  for  data,  but  not  used  in  analysis.    

Figure  1.  Print  screen  of  Condor  Wikipedia  search  with  GotoMeeting,  retrieved  11/04/13  

3.1.2. Topsy3 Topsy  is  a  data  collecting  and  analysis  software  for  socially  shared  content  primarily  from  Twitter  and   Google  Plus.  It  provides  analytics  based  on  the  key  word  counts,  key  actors  and  top  influencers,   sentiments  over  time,  and  sentiments  according  to  each  tweet.  We  use  Topsy  to  search  and  analyze   tweets  discussing  Citrix  and  GoToMeeting.  Some  examples  of  each  are  presented  below.

                                                                                                                        3

http://topsylabs.com/election/  

5  


Figure 2.  Print  screen  of  Topsy  search  for  CITRIX  and  GotoMeeting,  retrieved  11/03/13  

Figure  3.  Print  screen  of  Topsy  sentiment  score  for  CITRIX  and  GotoMeeting,  retrieved  11/03/13  

Figure 4.  Print  screen  of  Topsy  top  influencers  talking  about  CITRIX  and  GotoMeeting   6    


3.2. Sentiment Analysis  tools   3.2.1. IBM  Many  Eyes4   IBM  Many  Eyes  provides  a  wide  functionality  for  visualization  of  all  sorts  of  data,  from  text  to   mathematical  data.  However,  we  only  used  the  text  analysis  tools:  word  clouds,  word  trees,  tag  clouds,   and  phrase  net.  The  word  cloud  provides  a  picture  of  the  most  used  words  in  our  dataset  where  size  and   boldness  of  the  font  corresponds  to  frequency  (see  Figure  5).  The  cloud  tag  is  an  expanded  version  of   word  cloud;  it  calculates  the  frequencies  of  each  work  in  a  text  and  visualizes  those  words  as  tags.   Although  on  the  top  of  the  tag  cloud  visualization  the  exact  number  of  each  word  occurrences  and  the   context  in  which  it  was  used  is  shown  (see  Figure  6).    The  phrase  net  diagrams  the  relationships   between  different  words  used  in  a  text.  Thus  it  helps  pinpoint  and  evaluate  the  meaning  behind  each   word  used,  in  what  context  the  word  was  used,  and  what  other  words  linked  to  it  most  frequently  (see   Figure  7).  Similarly,  the  word  tree  shows  a  visualization  that  connects  words  that  were  paired  together.   This  way  we  don’t  only  evaluate  if  the  sentiment  is  positive  or  negative  but  can  track  the  subject  of  the   message  and  see  what  particular  features  of  the  service  were  praised  or  criticized  by  authors  of  the   texts  we  analyzed.  Therefore,  we  were  able  to  check  the  individual  words  of  interest  and  their  relations   within  our  data,  (see  Figure  8,  the  word  “need”).  By  looking  at  the  words  “good”  and  “bad”  (and  other   nouns  with  similar  meaning)  we  were  able  to  understand  the  product  features  that  customers  were   satisfied  with  and  which  ones  left  them  feeling  dissatisfied.  

Figure  5.  Print  screen  of  IMB  Many  Eyes  word  cloud  analysis  of  Blog  posts  about  GotoMeeting,  2011  

                                                                                                                      4

http://www-­‐958.ibm.com/software/analytics/manyeyes/  

7  


Figure 6.  Print  screen  of  IMB  Many  Eyes  tag  cloud  analysis  of  Blog  posts  about  GotoMeeting,  2011    

Figure  7.  Print  screen  of  IMB  Many  Eyes  phrase  net  analysis  of  iTunes  GotoMeeting  reviews,  2013    

8  


Figure 8.  Print  screen  of  IMB  Many  Eyes  word  tree  analysis  of  iTunes  GotoMeeting  reviews,  2013    

3.2.2. Lexalytics5 Lexalytics  is  a  software  package  that  uses  its  own  algorithm  to  calculate  sentiment  of  a  given  text.  It   does  so  through  transforming  unstructured  text  into  structured  data.  The  software  extracts  entities   (people,  places,  companies,  products,  etc.),  sentiment,  quotes,  opinions,  and  themes  (generally  noun   phrases)  from  text.  The  software  uses  natural  language  processing  technology  to  extract  the  above-­‐ mentioned  items  from  social  media,  forums,  or  general  text.  The  example  of  using  Lexalytics  on  our   data  is  shown  in  Figure  9.  

Figure  9.  Print  screen  of  Lexalytics  analysis  of  iTunes  GotoMeeting  reviews,  2013                                                                                                                           5

http://lexalytics.com/  

9  


3.2.3. LIWC6 Linguistic  Inquiry  and  Word  Count  (LIWC)  is  a  text  analysis  software  program.  LIWC  calculates  the   degree  to  which  people  use  different  categories  of  words  across  a  wide  array  of  texts.  As  an  outcome,  it   estimates  the  degree  any  text  uses  positive  or  negative  emotions,  self-­‐references,  causal  words,  and  70   other  language  dimensions.   The  LIWC  program  analyzes  various  standardized  ASCII  text  files  or  Microsoft  Word  documents.  The   LIWC2007  licensed  program  also  allows  you  to  build  your  own  dictionaries  to  analyze  dimensions  of   language  specifically  relevant  to  your  interests.  The  student  version  of  LIWC,  LIWClite7,  we  used  only   analyzes  plain  text  files  using  the  LIWC2007  and  earlier  LIWC2001  dictionaries.  The  demo  version  is  also   limited  to  the  amount  of  data  you  can  input  at  once,  so  it  was  rather  inconvenient  to  use  for  our   purposes.  The  screenshot  of  the  analysis  we  made  with  this  tool  is  shown  in  Figure  10.  

Figure 10.  Print  screen  of  LIWC  analysis  of  iTunes  GotoMeeting  reviews,  2013  

3.2.4. Wordle7 Wordle  is  a  tool  very  similar  to  IBM  Many  Eyes  word  cloud  function.  It  has  a  simple  and  user-­‐friendly   interface  for  quick  generation  of  word  clouds  that  can  be  customized  by  color  and/or  font.  Since  it  did   not  provide  any  additional  functionality  over  IBM  Many  Eyes,  we  used  the  latter.  However,  Wordle  is  a   good  tool  when  word  cloud  analysis  is  sufficient  and  may  be  preferable  because  of  its  simplicity  for  use   and  ease  of  access.  

3.2.5. Text-­‐processing.com API8   The  text-­‐processing.com  API  offers  various  functionalities  for  text  mining  and  natural  language   processing  through  a  JSON  over  HTTP  web  service.  Two  functionalities  were  used  for  our  purpose,   sentiment  analysis  and  word  stemming.  The  website  of  this  tool  offers  client  libraries  for  different   programming  languages  such  as  Java,  PHP  and  Python.  Since  we  didn’t  need  all  the  various                                                                                                                           6

http://www.liwc.net/  

7

http://www.wordle.net/  

8

http://text-­‐processing.com/docs/  

10  


functionalities, we  decided  to  write  a  short  program  code  in  PHP  and  to  import  the  data  and  results  in  a   MySQL  database.   The  sentiment  analysis  of  the  text-­‐processing.com  API  is  composed  by  two  bayes  classifier  and  was   trained  by  movie  reviews.  The  first  classifier  calculates  the  probability  for  a  text  to  be  neutral.  The   second  classifier  calculates  the  probability  of  the  text  to  be  negative  or  positive.  Given  that  the   probability  of  the  neutrality  is  higher  0.5  the  final  classification  would  be  neutral.  If  the  text  is  not   neutral,  this  means  probability  is  lower  than  0.5,  the  text  will  be  classified  negative  or  positive   depending  on  which  value  is  higher.   Table2.  The  code  running  analysis  on  reviews  and  other  GoToMeeting  data  in  text-­‐processing.com  API    

11  


Figure 11.  Print  screen  of  Text-­‐processing.com  API  analysis  of  a  GoToMeeting  review    

3.2.6. Sentiment.Viz9 Sentiment.Viz  is  a  social  media  data  analysis  tool  for  Twitter.  It    searches  tweets  by  an  inputted  key   word,  in  our  case  –  GoToMeeting,  and  uses  a  sentiment  dictionary  (Figure  12)  to  estimate  the  sentiment  of   each  tweet.  It  analyzes  the  words  in  each  tweet,  and  then  combines  the  words'  pleasure  and  arousal  ratings   to  estimate  sentiment  for  the  entire  tweet.  The  full  functionalities  of  this  tool  that  we  used  for  our   analysis  are  described  below:   •

• •

Sentiment. Each  tweet  is  represented  as  a  circle  positioned  by  its  individual  sentiment.   Unpleasant  tweets  are  drawn  as  blue  circles  on  the  left,  and  pleasant  tweets  as  green  circles   on  the  right.  Sedate  tweets  are  drawn  as  darker  circles  on  the  bottom,  and  active  tweets  as   brighter  circles  on  the  top  (see  Figure  13).     Topics.  Tweets  about  a  common  topic  are  grouped  into  topic  clusters.  Keywords  above  a   cluster  indicate  its  topic.  Tweets  that  do  not  belong  to  a  topic  are  visualized  as  singletons  on   the  right  (see  Figure  15).     Heat  Map.  Pleasure  and  arousal  are  used  to  divide  sentiment  into  an  8×8  grid.  The  number  of   tweets  that  lie  within  each  grid  cell  are  counted  and  used  to  color  the  cell:  red  for  more  tweets   than  average,  and  blue  for  fewer  tweets  than  average.  White  cells  contain  no  tweets.     Tag  Cloud.  Common  words  from  the  emotional  regions  upset,  happy,  relaxed,  and  unhappy   are  shown.  Words  that  are  more  frequent  are  larger.     Timeline.  Tweets  are  drawn  in  a  bar  chart  to  show  the  number  of  tweets  posted  at  different   times.  Pleasant  tweets  are  shown  in  green  on  the  top  of  the  chart,  and  unpleasant  tweets  are   shown  in  blue  on  the  bottom.     Map.  Tweets  are  drawn  on  a  map  of  the  world  at  the  location  where  they  were  posted.  Please   note  that  most  Twitter  users  do  not  provide  their  location,  so  only  a  few  tweets  will  be  shown   on  the  map.    

                                                                                                                      9

http://www.csc.ncsu.edu/faculty/healey/tweet_viz/tweet_app/

12  


• •

Affinity. Frequent  tweets,  people,  hashtags,  and  URLs  are  drawn  in  a  graph  to  show  important   actors  in  the  tweet  set,  and  any  relationship  or  affinity  they  have  to  one  another.     Tweets.  Tweets  are  listed  to  show  their  date,  author,  pleasure,  arousal,  and  text.    

Axis!for!Sen)ment!Analysis!/!Range!of!emo)ons!ploCed!

_!

+!

Figure 12.  Sentiment.Viz  scale  for  evaluating  different  emotions  through  sentiment  analysis  

Figure 13.  Sentiment.Viz  analysis  of  Twits  about  GoToMeeting,  retrieved  11/03/13  

13  


Figure 14.  Sentiment.Viz  analysis  of  GoToMeeting  with  twits  view,  retrieved  11/03/13    

Figure  15.  Sentiment.Viz  analysis  of  Twits  about  GoToMeeting  the  clustering  of  results    

3.3. Evaluation of  sentiment  analysis  tools   After  running  our  data  through  a  range  of  various  algorithms  for  sentiment  analysis,  we  needed  to   choose  the  tools  that  proved  to  be  valid  and  helped  deliver  us  to  our  project  goals.  In  order  to  narrow   the  tools  down,  we  created  a  set  of  conditions:   • •

The first  condition:  the  analysis  of  all  our  data  used  the  same  or  at  least  similar  algorithms.     The  second  condition:  determine  the  quality  of  the  sentiment  analysis.  The  technical  ability  of   each  method  to  distinguish  sentiment  correctly  is  limited  compared  to  human  analysis,  so  we   wanted  to  choose  the  tool  that  would  perform  the  best  quality  that  could  as  closely  match   human  analysis.     14  


The third  condition:  expand  beyond  the  classification  of  positive/negative/neutral  sentiment  and   highlight  different  emotions  that  users  are  expressing.  To  deliver  our  project  goal  –  “analysis  of   user  emotional  responses  to  identify  the  opportunities  for  further  development  of  the  target   product/company”  we  needed  to  capture  a  broader  range  of  emotions  than  “good  vs.  bad.”  

3.3.1. Data input  methods  of  the  tools   Many  of  the  tools  we  used  did  not  allow  large  text  or  data  input.  Despite  the  great  functionalities  of   Topsy  and  Sentiment.VIZ,  we  discovered  that  these  tools  were  only  suitable  for  analysis  of  tweets.  We   did  contact  the  developers  of  Sentiment.VIZ  to  find  out  if  it  could  support  data  from  other  sources  or   text  input,  but  this  functionality  was  not  available.  However,  we  still  could  take  into  account  the  results   derived  from  Topsy  and  Sentiment.VIZ  but  only  as  supplementary  information.     The  tools  with  sufficient  input  options  were  Lexalytics,  word-­‐processing.com,  IMB  Many  Eyes,  and   Wordle.  LIWC  would  have  been  an  option,  but  there  were  limitations  in  how  much  data  could  be   inputted  using  the  demo  version  so  it  was  not  feasible  for  our  analysis.        

3.3.2. Validation of  Sentiment  Analysis     We  chose  our  final  evaluation  tools  according  to  the  availability  of  inputting  large  texts  and  how  similar   they  were  to  human  sentiment  analysis.  According  to  our  first  requirement,  the  tools  were  narrowed   down  to  Word-­‐Processing.com,  Lexalytics,  and  Qdap  (R-­‐package).  Then,  by  evaluating  the  quality  of  the   sentiment  analysis  and  their  algorithms,  we  performed  a  manual  sentiment  of  a  random  sample  from   our  dataset  (approximately  10%  or  100  entries)  and  compared  the  results  of  each  tool.  We  then   calculated  the  Euclidian  distance  (see  Figure  14)  to  the  manual  answer  from  each  of  the  tools  values  to   verify  their  sentiment  analysis  and  evaluate  which  tool  we  should  move  forward  with.    

3.3.2.1.

Manual Analysis  100  cases  

For validation  of  the  various  tools  we  used,  we  performed  a  manual  sentiment  evaluation.  We  read   10%  (100  entries)  of  our  AppStore  reviews  and  gave  each  a  value  of  -­‐1  if  the  sentiment  sounded   negative;  the  value  of  +1  if  the  sentiment  sounded  positive;  and  0  for  neutral  reviews.    

3.3.2.2.

QDap –  R  package  

Quantitative Discourse  Analysis  Package  (QDap)  is  a  statistical  software  tool  originally  designed  for   analysis  of  transcripts  but  is  also  suitable  for  other  purposes  such  as  sentiment  analysis.  Its  functional   possibilities  are  frequency  counts  of  sentence  types,  words,  sentences,  turns  of  talk,  syllables  and   other  assorted  analysis  tasks.  QDap  was  only  used  as  a  comparison  tool  for  the  manual  analysis.  QDap   is  one  of  the  tested  analysis  tools  in  Table  3.    

15  


Figure 16.  Euclidian  Distance  Calculation   We  calculated  the  distances  of  each  individual  entry  assessed  by  each  tool,  gained  the  final  score,  and   compared  each  entry  to  see  which  one  performed  closest  to  the  manual  sentiment  analysis.  The   comparison  is  illustrated  in  Table  3.    According  to  the  total  distance  score,  the  closest  algorithm  to   manual  sentiment  analysis  was  Lexalytics.  However,  the  Euclidian  distance  validation  technique  has   flaws  so  we  cannot  take  is  an  absolute.  For  example  if  a  tool  provides  a  lot  of  neutral  judgments  then   the  distance  would  be  smaller  even  if  most  of  the  values  do  not  coincide  (since  the  distance  from  0  to  +1   as  well  as  -­‐1  will  be  the  shortest).  In  order  to  combat  this  flaw,  we  took  the  cumulative  frequencies  of   each  judgment  (-­‐1,  0,  +1)  made  by  each  method.  The  result  is  summarized  in  a  histogram  in  Figure  15.   Table  3.  Comparison  of  the  results  of  the  tools  to  manual  sentiment  analysis  

         

16  


Figure 17.  Comparison  of  Cumulative  Frequencies  of  Sentiment  Analysis  made  by  different  algorithms   According  to  Figure  17,  the  algorithm  of  Text-­‐processing.com  API  tool  performed  the  closest  to  manual   analysis.  The  results  of  two  validation  techniques  we  applied  are  controversial  as  they  point  to  different   tools  thus  there  is  not  an  absolute  winner.  Therefore,  we  decided  to  use  both  Lexalytics  and  Word-­‐ Processing.com  for  further  analysis.  Finally,  using  the  third  evaluation  method  we  pointed  out  in  the   beginning  of  this  subsection,  the  availability  of  wider  scale  of  emotions,  we  needed  to  supplement   Lexalytics  and  Word-­‐Processing.com  with  another  tool  proceed  with  further  analysis.  The  most  suitable   tool  that  describes  emotions  on  a  relative  scale  is  IBM  Many  Eyes  (word  tree,  word  cloud,  tag  cloud).    By   using  all  three  together,  we  were  able  to  build  a  stronger  foundation  for  our  data  analysis.    

3.4. Theoretical Foundation  to  Frame  the  Findings   Our  plan  also  consisted  of  comparing  the  two  sets  of  data  we  have  collected:  “internal”  what  Citrix  is   saying  about  the  target  product  and  “external”  what  customers,  users,  and  various  experts  are  saying   about  this  product.  To  perform  this  comparison,  we  tested  and  evaluated  the  data  using  the  three   evaluated  sentiment  analysis  tools.    However,  in  order  to  make  solid  conclusions  based  on  our  findings,   we  needed  a  theoretical  framework.  This  framework  would  help  us  better  understand  the   interconnections  between  our  constructs  and  how  to  interpret  our  findings  in  order  to  make  them   comprehensible  and  applicable  for  management.       The  Corporate  Identity  Management  theory  was  used  to  develop  opportunities  and  insights  from  our   data  analysis.  Corporate  Identity  Management  theory  argues  that  each  corporation  has  a  “character”   and  best  serves  customers  who  share  or  like  similar  identity  traits  (Bromley  2000).  Thus  it  is  crucial  for  a   company  to  be  consistent  in  transmitting  its  “character”  to  the  market  to  find  the  “right”  customers.  In   our  research,  comparing  Citrix’s  understanding  of  its  own  identity  (through  the  lenses  of  the  target   product)  to  that  of  the  customer’s  perception,  we  could  see  if  there  were  any  identity  gaps.  The   framework  built  from  Corporate  Management  Theory  is  defined  by  the  acronym  ACID  (Balmer  &  Soenen   1999),  where  “A”  refers  to  actual  identity  of  a  company,  “C”  refers  to  communicated  identity,    “I”  refers   to  ideal  identity,  and  “D”  refers  to  desired  identity.  The  idea  of  the  test  is  to  pinpoint  the  different   identities  a  company  has  and  compare  them  to  each  other.  A  modification  to  the  ACID  Test  is  the  AC2ID   test  (Balmer  &  Greyser  2002)  that  has  an  additional  “C”  for  conceived  identity.  After  distinguishing  the   five  identities  of  the  AC2ID  Test  for  GoToMeeting,  we  were  able  to  compare  the  “internal”  and   “external”  images  and  formulate  the  insights  and  opportunities.   17    


Figure 18.  AC2ID  test  model  by  J.M.T.  Balmer  and  S.A.  Greyser  

4. Insights As  mentioned  earlier,  after  screening  our  data,  we  were  able  to  separate  our  data  to  into  the  constructs   of  the  AC2ID  Test  for  analysis.  The  five  subsections  below  describe  each  of  the  identities  and  attribute   them  to  our  analyzed  datasets  in  detail.  

4.1. Actual Identity   Actual  is  an  identity  that  defines  who  the  company  is,  on  a  structural  and  strategic  decisions  level.   Balmer  and  Greyser  (2002:  74)  state  that  “the  actual  identity  constitutes  the  current  attributes  of  the   corporation.  It  is  shaped  by  a  number  of  elements,  including  corporate  ownership,  the  leadership  style   of  management,  organizational  structure,  business  activities  and  markets  covered,  the  range  and  quality   of  products  and  services  offered,  and  overall  business  performance.  Also  encompassed  is  the  set  of   values  held  by  management  and  employees.”   For  this  identity,  we  analyzed  Citrix’s  mission  and  vision  statements  to  reflect  the  actual  identity  of  the   company.    The  resulting  analysis  of  this  data  is:     GoToMeeting  corporate  content  suggests  that  this  application  has  the  objective  to  be  the  facilitator  for   enterprises  to  collaborate  more  internally,  and  for  individuals  to  accommodate  work  and  life  in  a  more   flexible  way.  

4.2. Communicated Identity   Communicated  identity  refers  to  the  image  the  company  is  transmitting  to  outside  world,  what  it  tells   about  itself.  Balmer  and  Greyser  (2002:  74)  define  it  as  follows:    “The  communicated  identity  is  most  clearly  revealed  through  ‘controllable’  corporate  communication.   This  typically  encompasses  advertising,  sponsorship,  and  public  relations.  In  addition,  it  derives  from   ‘non-­‐controllable’  communication,  e.g.,  word-­‐of-­‐mouth,  media  commentary,  and  the  like.”     For  this  identity,  we  analyzed  marketing  materials  found  through  corporate  blog  and  reviews  posted  by   Citrix.  The  following  is  our  analysis  of  communicated  identity:  

18  


GoToMeeting’s communication  emphasizes  the  collaborative  element  of  working  together  where  in   different  locations.  It  also  highlights  the  cost  effective  results  of  using  this  application.  The  company  has   made  a  great  effort  on  collecting  testimonials  from  a  variety  of  clients  within  different  industries.  

4.3. Conceived Identity   Conceived  identity  is  how  users  and  non-­‐users  view  and  evaluate  the  company,  Balmer  and  Greyser   (2002:  74)  explained  it  as  follows:   “The  conceived  identity  refers  to  perceptual  concepts  —corporate  image,  corporate  reputation,  and   corporate  branding.  These  are  the  perceptions  of  the  company—its  multi-­‐attribute  and  overall   corporate  image  and  corporate  reputation—held  by  relevant  stakeholders.”     In  our  study  conceived  identity  refers  to  all  the  social  media  data  we  collected.  This  dataset  is  compiled   of  over  a  thousand  customer  reviews  and  posts,  expert  opinions  from  blogs  and  web,  ratings  and   reviews  from  app  market  places  (Amazon,  Google  Play  and  AppStore),  social  networks  posts   (Facebook),  popular  tech  blogs,  review  aggregation  pages,  and  video  comments  (Youtube).  The   outcome  of  the  analysis  is  summarized  below:     The  user  impressions  about  GoToMeeting  were  mixed,  positive  and  negative  comments  were  almost   balanced  in  our  dataset.  The  strengths  and  weaknesses  of  GoToMeeting  app,  which  customers  and  other   reviewers  pointed  out  the  most,  are  listed  below.  According  to  our  data  the  app  strengths  that  evoked   positive  reactions  and  comments  were     • • • •

Flexibility on  joining  meetings  remotely   Lifestyle  change   High  Definition  of  the  meeting  participants’  faces   Availability  on  mobile  devices  

The weaknesses  of  the  app  that  people  mentioned  the  most  were  related  to     • • •

Lack of  a  feature  to  inform  the  presenter  when  a  participant  has  a  question  (that  would  refer  to   raising  hand  activity  in  a  classroom  situation)   Lack  of  possibility  to  host  files   A  perceived  complexity  of  hosting  meetings  

4.4. Ideal Identity     This  identity  identifies  the  best  a  company  can  make  out  of  its  capabilities.  In  other  words,  it  is  the   system  perspective  of  a  company’s  best  position  or  role  in  a  global  ecosystem.    The  description  from  the   authors  of  the  concept  (Balmer  &  Greyser  2002>  74)  is  quoted  below:   “The  ideal  identity  is  the  optimum  positioning  of  the  organization  in  its  market  (or  markets)  in  a  given   time  frame.  This  is  normally  based  on  current  knowledge  from  the  strategic  planners  and  others  about   the  organization’s  capabilities  and  prospects  in  the  context  of  the  general  business  and  competitive   environment.  The  specifics  of  a  given  entity’s  ideal  identity  are  subject  to  fluctuation  based  on  external   factors—e.g.,  the  nuclear  power  industry  after  Chernobyl;  and  industries  (such  as  travel,  transport   equipment,  and  security  systems)  affected  negatively  and  positively  by  the  September  11  World  Trade   Center  catastrophe.”     19    


In order  to  evaluate  the  ideal  identity,  it  would  require  collecting  data  from  other  resources  such  as   news  and  financial  analysts’  reports  (that  we  did  not  have  access)  to  make  reliable  judgments.  Due  to   the  limitations,  we  decided  to  omit  this  identity  from  our  analysis  and  proceed  with  the  other  four   identities  that  are  described.    In  the  following  section  we  make  comparisons  between  those  four   identities  and  discuss  possibly  implications  for  the  Citrix  and  GoToMeeting.  

4.5. Desired Identity   This  identity  is  formed  by  what  the  management  wants  the  company  to  be  and  to  what  direction  they   wish  to  develop  their  organization.  Balmer  and  Greyser  (2002:  75)  describe  it  as  follows:   “The  desired  identity  lives  in  the  hearts  and  minds  of  corporate  leaders;  it  is  their  vision  for  the   organization.  Although  this  identity  type  is  often  misguidedly  assumed  to  be  virtually  identical  to  the   ideal  identity,  they  typically  come  from  different  sources.  Whereas  the  ideal  identity  normally  emerges   after  a  period  of  research  and  analysis,  the  desired  identity  may  have  more  to  do  with  a  vision  informed   by  a  CEO’s  personality  and  ego  than  with  a  rational  assessment  of  the  organization’s.”   Our  analysis  of  Citrix  and  GoToMeeting  was  developed  through  our  contact  of  management  at  Citrix   and  by  the  mission  and  vision  statements  of  Citrix  and  GoToMeeting.  Through  discussions  and   presentations,  we  formed  the  desired  identity  of  Citrix:     GoToMeeting  wants  to  be  THE  online  meeting  software  that  provides  the  tools  necessary  to  allow  users  to   feel  as  comfortable  and  share  information  is  if  they  were  meeting  face-­‐to-­‐face.    

5. Opportunities By  developing  the  various  identities  of  a  company,  we  can  determine  whether  the  identities  are  aligned.   In  this  section,  we  compare  Citrix’s  identities  described  in  the  previous  section  and  compare  how  well   they  correlate.  We  create  pairwise  comparisons  of  selected  identities  that  can  potentially  help  Citrix  and   GoToMeeting  better  understand  their  customers  and  perform  adjustments  to  its  communication,   customer  service  and  product  development  strategies.  

5.1. Actual vs.  Communicated   By  comparing  our  analysis  of  actual  and  communicated  identities,  we  found  that  although  GoToMeeting   communicates  what  it  stands  for  in  its  vision  and  mission  there  were  still  areas  for  improvement  in   addressing  more  niche  customers.  They  can  add  the  focus  on  B2C  activities  to  its  B2B  strategy  to  help   improve  product  development  and  deliver  offerings  to  individual  users  along  with  those  for   organizations.  For  example,  the  user  sector  of  independent  agents  such  as  nutritionists  or  personal  and   financial  coaches  have  not  been  yet  addressed  by  the  company’s  communication.  There  is  an   opportunity  for  establishing  contact  to  this  entrepreneurial  sector  via  marketing  channels  and  product   development  strategy  add-­‐ons  (in  features  or  capabilities  of  the  app).  

5.2. Desired vs.  Conceived   Although,  the  management  sees  Citrix  as  an  end  user-­‐centered  company,  our  analysis  discovered  that   there  are  still  issues  to  be  resolved  in  order  to  achieve  this  state.  GoToMeeting  thrives  for  ease  of  use   and  practicality,  however  there  are  some  users  that  struggle  with  existing  features  or  require  more   20    


capabilities to  get  their  job  done.  Thus  there  are  possibilities  in  product  development  that  can  help   Citrix  fulfill  the  vision  of  management  and  become  a  company  they  are  sought  to  be.  

5.3. Communicated vs.  Conceived   The  comparison  between  communicated  and  conceived  identities  revealed  that  GoToMeeting’s   marketing  materials  generally  addressed  organizational  customers  (companies)  and  the  front-­‐end  users   that  set  up  the  meetings,  whereas  the  social  network  data  was  usually  reviewed  and  rated  by  end  users:   employees  of  the  organizational  customers  and  individual  customers.  Thus  we  could  detect  the   opportunity  for  Citrix  and  GoToMeeting  to  publish  experiences  from  individuals  that  are  using  the   software  to  relate  more  to  end  users  and  help  develop  the  image  of  a  user-­‐centric  company  that  Citrix   represents.  

6. Conclusion and  Next  Steps   GoToMeeting  by  Citrix  is  a  popular  yet  young  software  with  thousands  of  downloads  and  millions  of   users.  Although  a  multitude  of  information  could  be  collected  through  the  web,  our  method  applied  to   the  selected  amount  of  data  helped  determine  new  opportunities  for  further  development  for  Citrix  to   better  understand  and  connect  with  their  customers.     Our  research,  analysis  and  insights  aimed  at  understanding  user  emotional  responses  to  the  product   through  social  network  analysis.  By  collecting  data  from  a  multitude  of  sources  and  analyzing  them   through  various  tools,  we  were  able  to  build  a  reference  for  comparing  sentiment  analysis.  By  checking   the  results  to  the  manual  sentiment  analysis,  we  validated  our  findings  and  expanded  beyond  the   positive/negative  score  to  encompass  a  broader  range  of  emotions.  Finally,  the  application  of  the  AC2ID   Test  theoretical  framework  to  our  sentiment  analysis  outcomes  allowed  us  discover  insights  from  the   data  and  create  recommendations  in  which  Citrix  might  embrace  to  improve  user  experiences.  We  hope   that  our  final  recommendations  validate  the  use  of  Social  Network  Analysis  as  a  strategy  for  Citrix  to   build  upon  for  GoToMeeting.     As  we  progressed  on  our  project,  we  discovered  a  few  opportunities  that  were  not  pursued  due  to  lack   of  time,  but  that  are  interesting  and  could  be  developed  further.    For  example,  there  is  potential  to   develop  some  of  the  tools  we  tested,  Sentiment.VIZ  and  Topsy,  to  expand  their  functionality  to  perform   qualitative  sentiment  analysis  on  all  types  of  texts.  We  found  these  tools  very  useful  and  rich  in  their   descriptions  of  emotions  compared  to  the  basic  three-­‐option  sentiment  analysis:  positive,  negative,  and   neutral.  In  our  opinion,  there  is  a  great  opportunity  to  develop  such  software  for  companies  to  use  to   help  in  a  deeper  understanding  of  users  through  social  network  analysis.     Finally,  we  believe  that  social  network  analysis  has  the  potential  to  be  used  as  a  form  of  primary   research  in  the  design  of  new  or  redesigned  projects.  Users  are  already  giving  unprompted  information   about  how  they  feel  about  a  product  or  service,  so  proper  analysis  of  this  information  can  give  a   company  an  edge  on  addressing  the  needs  of  their  customers.    

References Balmer,  G.M.T  &  Greyser,  S.A.  (2002).  Managing  the  Multiple  Identities  of  the  Corporation,  California   Management  Review,  4  (3)  :  72-­‐86   21    


Bromley, D.  B.  (2000).  Psychological  Aspects  of  Corporate  Identity,  Image  and  Reputation,  Corporate   Reputation  Review  3  (3):  240-­‐252   Balmer,  G.M.T  &  Soenen,  G.B.  (1999).  The  Acid  Test  of  Corporate  Identity  Management™,  Journal  of   Marketing  Management,  15(1-­‐3):  69-­‐92   Freeman,  L.  (2004).  The  development  of  social  network  analysis:  A  study  in  the  sociology  of  science.  (1st   ed.).  North  Charleston,  South  Carolina:  Empirical  Press.     Gloor,  P.,  &  Cooper,  S.  (2007).  Coolhunting:  Chasing  down  the  next  big  thing.  (1st  ed.).  New  York,  NY:   AMACOM.     Gloor,  P.  (2010).  Coolfarming:  Turn  your  great  idea  into  the  next  big  thing.  (1st  ed.).  New  York,  NY:   AMACOM.     Rainie,  L.,  &  Wellman,  B.  (2012).  Networked:  The  new  social  operating  system.  (1st  ed.).  Cambridge,  MA:   The  MIT  Press.    

22  

Measuring Emotional Responses to User Interfaces  
Measuring Emotional Responses to User Interfaces  

Online social communities are an invaluable avenue for understanding user emotions. In this paper, our collaborative innovation network (CoI...

Advertisement