In this article
1: Overview
The Significance testing panel lets you test results against one another to identify significant differences. In StoryCreator, you can choose between a Z test (for percent share or mean) or a T test (for categorical or numeric values).
Significance testing can be performed across legend items or axis items. You can define how comparisons are made, for example, test against the first, previous, or last data point, or test all data points against each other.
2: Significance Testing Panel
The panel is divided into three sections: Calculation settings, Test, and Visualization.
2.1: Calculation Settings
In the Calculation section you can define the following:
- Select the significance level (1%, 5%, or 10%)
- Set a base size limit. If using weighted data, choose whether to use unweighted, weighted, or effective counts
- Define a respondent count limit if needed
- If results are weighted, choose which base size option applies
2.2: Test
Select whether the test compares Axis groups or Legend items using the Test between list. This appears only if testing is possible in both directions.
If there is only one series present in the chart, the Test between list does not appear.
- Choose a comparison type: first, last, previous, next, or all vs all
- Select whether to show or hide the comparison point (not available if All vs All is selected)
- Optionally select a specific comparison point (for example, All compared to “Sweet talk”)
This example shows the significance test panel configured to compare all brands to "Sweet talk".
2.2.1: Limiting Types of Test
Administrators can control which test types (Z-test, T-test, or both) are available in the StoryCreator setup panel.
This setting lets you choose whether both the Z-test and T-test are available, only one of them, or neither.
2.3: Visualization
The Visualization section lets you define the following:
- If All vs All is selected, choose to show positive only, negative only, or both types of results
- Select which symbols indicate significant differences
2.3.1: Setting Colors
You can enable colored significance symbols using the “Use colored symbols” toggle in the Statistical analysis panel. The colors are defined in Project settings and apply across charts and tables.
Colored symbols are supported in editable PowerPoint exports and when exporting a whole slide as a single image. When exporting one image per object, the default data value color is used.
3: Significance Testing Examples
3.1: Compare to Previous Data Point
Testing across intervals (axis items), each point is compared to the previous data point. This setup shows significant differences from the previous week.
3.2: Compare to First Data Point
Testing across intervals, each point is compared to the first data point. This setup highlights significant changes from the first week.
3.3: Compare Brands Across Time
Testing across brands (legend items), each brand is compared to the other for each week.
3.4: All Brands Compared
Testing across brands, each brand is compared to all others. This highlights any significant difference between brands. In the below image, each brand is represented by a letter from a to e and these are then used to denote the significance.
3.5: Positive Differences Only
Same setup as Example 4, but only positive significance differences are shown.