An Evaluation of Software Tools for Interactive Storytelling Richard Rabil, Jr. 1/5/2012
two participants voiced concerns about configuration settings and integration with existing technology. “Programmers should be able to define constraints to a specific project. E.g., that the lines of a dialogue must not exceed a distinct number of characters, or, which emotions can be triggered by a dialogue...” said one participant. Another expressed a desire for, “Acces[s]ible API or SDK for user-created modifications and extensions/integration within other game technologies” As a result of this input, I developed an area of evaluation called “Other Factors Suggested by Survey Respondents.” The scores for these factors do not have a weighting associated with them, since I did not collect any survey data on them, but they do factor into the total score for each tool.
4.1.6 Significance of Results As illustrated above in Figure 2, the number of survey participants was not a representative sampling of game writers and narrative designers in the gaming industry. In my conclusion (see Section 6), I suggest an enhanced survey strategy through which data could be collected to result in data with greater statistical significance. Furthermore, as briefly mentioned in Section 3.4.1, “Procedure for Calculating Weightings,” my procedure for calculating the weightings was skewed in two ways. First, not all 33 participants answered every question in the survey. For instance, in the area of Collaboration, 26 participants responded, while under Usability and User Interface, only 19 participants responded. The Usability and User Interface category might have had higher weighted averages if the same number of participants responded to every question. I partially accounted for this by including the number of respondents in my weighted averages, but my calculations did not factor in an appropriate margin of error. Second, the weighted averages for each category are skewed because not every category has the same number of factors or capabilities available for scoring. For example, the Collaboration category has five factors/capabilities that can be scored, whereas the Character Design category only has three. For this reason, the Collaboration category is more likely to have a higher average weighting than the Character Design category. Despite these sources of error, the survey still brought in the perspectives of other individuals, and thus helped reduce the level of subjectivity in my evaluation. Important Note: In the original survey, participants were informed that a “1” referred to a “Critical” factor and a “7” referred to an “Irrelevant” factor. However, this scale was problematic because it assumed that a low score is a good score, whereas my subjective scoring scale (in which a “0” is low and a “4” is high) assumed that a high score is a good score. This led to confusion and inconsistencies in the data when I multiplied the average weightings from the survey against my subjective scores. To solve this problem, I reversed the definition of the numbers in the ranking scale so that “7” referred to “Critical” while 1 referred to “Irrelevant.” This reversal did not change the data I received from participants, since the definitions were arbitrary. Using the data from the survey, I calculated the weightings for each category. In the table below, the categories are sorted the highest to lowest weightings. The weightings suggest that overall, participants place the highest priority on factors related to branching dialogue, and place the lowest priority on factors related to character design. Furthermore, as explained in Section 3.4.2, the weightings are only half of the scoring process. The weightings were used in combination with my subjective scores to calculate a total score for each tool. The total scores can be seen in Section 4.3, “Results from Weightings and Subjective Scoring Process.” Furthermore, tables with the results of the survey and the weighted averages for each table can be found in the Appendix. 13