Analysis
The result of the calculations performed in mUQSA is a collection of data, presented in the form of graphs and tables, which are relevant for the target analysis of uncertainty and sensitivity of a given model. The analysis is made in respect to a single or many Quantities of Interests (QoIs), which may be a model output variable or any other metric used to assess the model’s performance or behavior.
Presentation of analysis results
The visualisation of particular data and the way how it is presented depends on a selected method and type of the analysed model. For all cases, the detailed data is presented in the form of tables and additionally can be exported to common data file types like CSV or XSLT. Additionally, for the majority of presented data, mUQSA generates interactive charts that facilitate interpretation of results. The list of currently available options is summarised below:
| Type of element | Role | Methods | Chart for scalar QoI | Chart for vector QoI |
|---|---|---|---|---|
| Descriptive Statistics (Basic method) | Presentation of basic moments in regards to internal variability of a model | Basic | - | Line chart with error bars showing main statistics for a given QoI in respect of index value |
| Descriptive Statistics (Parameter Sweep) | Presentation of basic moments in regards to parameter sweep execution | Sweep | - | - |
| Descriptive Statistics | Presentation of basic moments | MC, PCE, SC | - | Line chart showing the moments for a given QoI in respect of index value |
| First-order Sobol Indices | Presentation of contribution of input parameters variance to output variance | MC, PCE, SC | Pie Chart showing contributions of particular parameters on output variance | Line chart showing contributions of particular parameters on output variance in respect of index value |
| Second-order Sobol Indices | Presentation of contribution of joint interactions between two parameters to output variance | PCE | - | - |
| Total Sobol Indices | Presentation of contribution of input parameters, and all its interactions, to output variance | MC, PCE | Pie Chart showing contributions of particular parameters (and all their interactions) on output variance | Line chart showing contributions of particular parameters (and all their interactions) on output variance in respect of index value |
Interpretation
Descriptive statistics
Descriptive statistics allow getting general insight into the behavior of the model. The generated statistics outline the main facts about the distribution of the Quantity of Interest (QoI), and depending on the type of the produced results, show main statistical moments for scalar or vector data. The descriptive statistics are particularly useful to quantify the uncertainty of the model.
Sobol indices
Sobol indices allow discovering the importance of model parameters variance on the output uncertainty. The result of Sobol analysis is a list of percentage values that displays a contribution of individual or combined parameters variance on the variance of the results.
There are three types of Sobol indices supported by the mUQSA:
- First-order Sobol Indices
- Describe the influence of the variance of an individual parameters to the total output variance of the model, without considering possible interactions between parameters. The first-order indices tend to sum to 1.
- Second-order Sobol Indices
- Describe the importance of combined variability of pairs of parameters to the total output variance of the model. Since the second-order indices describe only the influences of interactions, their value is typically much lower than the value of first-order indices.
- Total Sobol Indices
- Provide the complete information about influences of variability of parameters, including both the effect of variability of an individual parameter and all its interactions with other parameters, to the output variance of the model. The Total Sobol Indices will be larger than 1, when there are interactions between parameters.
Sobol indices are a robust tool for discovering which parameters should be taken into account when trying to reduce the uncertainty in the model predictions, improve its accuracy or the computational efficiency. Consequently, based on the analysis of Sobol indices, the model can be variously improved. Let us describe a few strategies:
- If the calculated relative importance of an input parameter on the model’s output is high, the parameter’s measurement can be improved (e.g., more precise thermometer purchased).
- If the calculated relative importance of a parameter on the model’s output is low, the parameter can be fixed or removed from the model to release the model’s computational complexity.
- If the second-order Sobol index value for a pair of input parameter and model configuration parameter is high, the calibration of the configuration parameter can be considered, if possible.
Plots Explanation
Descriptive Statistic Line Plot (MC, PCE, SC)
Descriptive Statistic Plot is a line plot that illustrates the evolution of key descriptive statistics for quality of interest (“QoI”) across multiple iterations during an Uncertainty Quantification (UQ) simulation of a vectorized model.
Read more
X-Axis (Index/Iterations): The horizontal axis represents the UQ simulation index (e.g. iterations), Each point on the X-axis corresponds to a specific stage in the simulation, where various statistics related to QoI were measured and analyzed.
Y-Axis (Values): The vertical axis represents the values of the descriptive statistics such as mean, median, etc. The following statistics can be plotted for the QoI:
Mean: The mean value of the QoI at each index. This statistic provides an indication of the central tendency of the parameter and how it changes over simulation. E.g., if the mean increases or decreases steadily, it suggests a systematic change in the parameter’s behavior.
Standard Deviation (stdmax and stdmin): Two lines are shown, representing the high (stdmax) and low (stdmin) standard deviation values observed during the simulation. These lines indicate the range of variability or dispersion in the QoI.
Median: The median value of the QoI at each index. The median represents the middle value when all observations are ranked in order. It can provide insights into the parameter’s central tendency, especially in the presence of outliers. It will differ from the mean if the output distribution is asymmetric.
Percentiles: The percentiles provide a way to understand the distribution of observations by dividing it into percentile-specific parts, each representing some percentage of the total observed data. E.g. the 10% percentile allows you to easily see below which values are 10% of the observations (or above which are 90% of the observations), while a 90% percentile allows you to easily see above which values are 10% of the observations (or below which are 90% of the observations). Consequently, the percentiles provide a quite clear comparative view on the output distribution and help identify outliers in the analysed QoI.
First-order and Total Sobol Index Line Plots (MC, PCE, SC)
First-order and Total Sobol Index line plots illustrate the First-order/Total Sobol Indices analysis for a vectorized model. They track the impact of different input parameters on a quality of interest (QoI, e.g., a model’s output variable) over simulation (e.g. multiple iterations). Each line on the plot, distinguished by its unique color, represents the contribution of a specific input parameter to the overall variance in the QoI. This analysis provides insights into the relative importance of input parameters in explaining the uncertainty in model output.
Read more
X-Axis (Simulation Index/Iteration): The horizontal axis represents the UQ simulation index (e.g., iteration). Each point on the X-axis corresponds to a specific stage in the result data where the Sobol Indices were calculated.
Y-Axis (Sobol Index): The vertical axis represents the Sobol Index value.
Lines (Series): Each line on the plot, distinguished by different color, is associated with a particular input parameter of the model. Each line represents the contribution of that input parameter to the variance in the QoI: the higher value indicates a larger contribution.
Interpretation:
Relative Importance of Parameters: By observing the individual lines, you can assess the relative importance of each input parameter on the output uncertainty. Parameters associated with lines that show higher values of the First-order/Total Sobol Index have a more significant impact on the QoI.
Contribution Trends: Examining the lines over iterations can help you identify whether certain parameters have consistent impacts or if their influence varies throughout the analysis.
Parameter Relationships: Comparing the lines can also reveal interactions or dependencies between input parameters. For example, if two lines exhibit similar trends, it may indicate that these parameters are correlated in their influence on the quality of interest. You can explore it further by analysing Second-order Sobol Indices.
Model Optimization: This analysis can guide model optimization efforts by identifying which parameters have the most substantial influence on the output. Focusing on the optimization of influential parameters can be the right way to improve model performance.
First-order/Total-order Sobol Index Pie Chart (MC, PCE, SC)
The pie chart provides a visual representation of the First-order/Totalr Sobol Indices analysis for a non-vectorized model. Each segment within the pie chart is uniquely colored and corresponds to a specific input parameter of the model. The size of each segment is proportional to the First-order/Total Sobol Index associated with that parameter for a particular QoI.
Read more
Segments:
Each segment in the pie chart is represented by a distinct color and corresponds to one of the model’s input parameters. The size of each segment is directly proportional to the First-order/Total Sobol Index for that input parameter. Larger segments indicate input parameters with greater influence on the QoI.
Interpretation:
Relative Importance: The size of each segment reflects the relative importance of the corresponding input parameter in explaining variability in the QoI. Larger segments indicate more influential parameters, while smaller segments signify less influential ones.
Model Optimization: The generated Pie chart allows prioritizing input parameters for further analysis and optimization of the model. The model is more sensitive to parameters with substantial First-order/Total Sobol Indices, thus these parameters should receive more attention in model refinement efforts.