Question Compare allows you to compare your question results with another question (and a third question with stratification). Statistically significant differences are then highlighted.
Question compare is therefore a very powerful research tool, as it allow you to quickly see how respondents to two (or three, if you use stratification) questions respond. You can see if responses are correlated in some way or if they are independent for instance.
Since the system contains many questions, finding out which pairs of questions have a strong association is a labor-intensive task. To alleviate that, InsightStore™ is constantly crawling the question library, and measuring the strength of association between any pairs of questions. This strength of association is shown when browsing for questions to compare with, so users can have a very strong hint of which pairs of questions have interesting correlations.
Comparing two questions
To use question compare, the first step is to select the first question and go to its results page. Then, navigate to Results>Compare, which will lead you to the results compare page.
The first thing to do is picking a question to compare against. The system will provide a search page, similar to the Questions search page, in which you will be able to use search, sort, filters and tags to hone into the desired question.
Understanding the Question Compare Results
Question compare results can be seen in six different Display Modes. The Display Modes are: Compare Table, Stacked Chart: Overall %, Stacked Chart: % by Answer, Bar Chart: Overall %, Bar Chart, % By Answer, and Sankey Diagram. These six modes provide different visualizations that allow for a robust analysis of the relationship between the questions.
All of the question compare results only include respondents that have answered both questions. That is the reason why the total number of respondents in the compare diagrams is (in most cases) different from the total number of respondents in the top results frame.
The system provides an easy way to flip the order of the question comparison. Clicking on the flip icon on the top right of the question compare subtab will flip the two questions (the one that was in the rows will be in the columns, and vice-versa). Here’s how the flip icon looks like:
Understanding the Compare Table Display Mode
The Compare Table Display Mode show a table that contains the first question answer options listed in rows, and the second question answer options in columns.
Each row and column will have the total number of respondents, as well as the percentage share of those respondents.
Each cell in the table will include the share of respondents in that row and also in that column. In case the number of respondents is substantially different than what would be expected in that cell in case the questions were not correlated, the system will include green or red indicators of those differences.
Below the table there are checkboxes that allow users to control which information to show in the table. There are checkboxes for the Row %, Column %, and also for % from Expected.
Understanding the Stacked Chart Display Modes
The Stacked Chart Display Mode shows a stacked chart that contains a stacked chart for each of the answers in the first question; and each stacked chart will contain the answers to the second question.
When using the % by Answer version, each of the stacked charts will show the percent of respondents that picked that answer (or answer grouping):
When using the Overall % r version, each of the stacked charts will show the percent of total respondents to both questions:
Understanding the Bar Chart Display Modes
The Bar Chart Display Mode shows groups of bar charts, one group per each of the answers in the first question. In each group, there will be a bar chart for each of the answers to the second question.
When using the % by Answer version, each of the bar charts will show the percent of respondents that picked that answer (or answer grouping):
When using the Overall % r version, each of the bar charts will show the percent of total respondents to both questions:
Understanding the Sankey Diagram Display Mode
Sankey diagrams are a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity (source: Wikipedia). In the InsightStore™, he Sankey Diagram Display Mode shows responses to the first question in the top, and results to the second question in the bottom. Flows show how respondents of each question responded the other question. Hovering the mouse over each flow or over each response bar shows statistics about the responses:
Adding a Compare Dashlet to a Dashboard
Question Compare results can be added to dashboards. To do so, simply click on the “Add to Dashboard” link on the bottom right of each display mode. The system will them prompt you to select which dashboard to add the dashlet to.
The compare dashlet supports all of the Display Mode types, except for Compare Table. If compare results are in Compare Table mode, then the dashlet will assume the Stacked Bar Chart, % by Answer format.
Understanding Strength of Association Calculation
InsightStore™ contains many questions. To aid with finding out which pairs of questions have a strong association InsightStore™ is constantly crawling the question library, and measuring the strength of association between any pairs of questions. The strength of association is measured in terms of the Tschuprow’s T coefficient (see statistical definition below).
Once the T is calculated, it will be shown in the search list of questions to provide a visual indication of questions that are likely to be interesting to researchers.
Since having a high strength of association provides a very significant cue to zooming in into interesting question to compare against, the system provides a sort by unweighted strength of association option. That is indeed the default search option.
The discovery of strength of association takes in consideration the weighting scheme. Depending on your account configurations, the system will crawl for strength of associations for a different set of weighting schemes.
Notice that sometimes the system does not show a T coefficient. That happens when the system has not yet calculated the T between that pair of questions, or when there are not enough answers in one of the questions in the pair to allow for a calculation. Because of that, you should be aware that sometimes there are questions that may end up having a higher T coefficient than the ones that have been calculated.
CivicScience advises the following characterizations of various levels of the T statistic:
- T < 0.05: "Low strength of association"
- 0.05 <= T < 0.10: "Medium strength of association"
- 0.10 <= T < 0.20: "High strength of association"
Question compare works well with Answer Groups. For the first question, you can pick an answer group on the top results pane. For the second question, if it contains answer groups, the system will show a drop down to allow you to select which Answer Group to show.
Once you make Answer Group selections, the system will update the whole comparison table immediately, including the chi-squared test and significance.
Using Answer Groups in combination with question compare is very powerful. For instance, if a question has many possible answer options, the correlations may be hard to see. Say a question that has “I like it” and “I love it” answer options. The comparison table may not show that they are statistically significant when looked at independently. But if you group them together in an Answer Group, then they perhaps will be statistically significant.
Question compare works well with Scores, similarly to Answer Groups. You can only select a score for the first question, though. You do so by selecting a score on the display options of the top results pane. You can select Answer Groups for the second question, though, assuming it has any Answer Groups configured.
Once you select a score, the system will update the whole comparison table immediately. There will be only one line in the table, showing the score for each possible answer option (or Answer Group) of the second question.
Statistical Terms Used in Question Compare
- Chi-square: The chi-square value is the result of the Pearson's chi-squared test of independence, which evaluates whether a relationship exists between how the respondents answered each question.
- P-value: The p-value corresponds to the chi-squared value, and represents the probability that there is no relationship between how respondents have answer each question. The lower the p-value, the more confident we can be that such a relationship exists. Results that are shown to be statistically significant have been adjusted according to the Benjamini-Hochberg false discovery rate procedure.
- Tschuprow's T: Tschuprow's (sometimes spelled “Chuprov's”) T is a measure of the strength of the relationship between how the respondents answered each question. Its value ranges from 0 to 1 -- the higher the value for Tschuprow's T, the stronger the relationship between how the respondents answered each question.