Symmetric Mean Absolute Percentage Error (sMAPE)

Pros:

- Simple and Intuitive: It is easy to understand and compute. It provides a percentage error which is often more intuitive for people to understand.

- Symmetry: Unlike the Mean Absolute Percentage Error (MAPE), the symmetric version treats over-forecasting and under-forecasting the same way.

- Scale-Independent: It is a relative measure of error, so it is independent of the scale of the data, which is useful when comparing the performance of different models or datasets with different scales.

Cons:

- Sensitivity to Zero: sMAPE can produce undefined or infinite values if the actual measurements contain zero values. This is a common limitation of all percentage-based error metrics.

- Equal Weightage: All errors, irrespective of their position in the time series, are treated with equal importance. This might not be suitable if certain parts of the series are more important than others.

- Not Suitable for All Data Types: It may not be the most appropriate metric for all kinds of data. For example, it may not work well for data with a lot of zero or near-zero values.

 

Feature Selective Validation (FSV)

Pros:

- Comprehensive: FSV considers both the amplitude and phase of the signals, making it a comprehensive validation technique. It is especially useful for comparing complex signals.

- Visual Interpretation: It provides a visual interpretation of the differences between the measured and simulated data, which can be very useful for understanding the nature and location of the differences.

- Robustness: It is a robust method and can handle zero or near-zero values.

- Widely Accepted: It is widely accepted in the electromagnetic compatibility (EMC) community for comparing time and frequency domain signals.

Cons:

- Complexity: It is more complex to compute than sMAPE. It requires a detailed algorithm and is not as straightforward to implement.

- Subjectivity: The interpretation of the FSV results can sometimes be subjective. It requires expertise to interpret the results accurately.

- Computational Intensity: It can be computationally intensive, especially for large datasets.

 

Conclusion:

The choice between sMAPE and FSV depends on the nature of your data and what aspects of the comparison are most important for your application.

- Use sMAPE if you are looking for a simple, intuitive, and quick way to compare the overall accuracy of your simulation model with the measurement results, and your data does not contain zero or near-zero values.

- Use FSV if you are working with complex signals, need a detailed comparison of the amplitude and phase of the signals, or need a visual interpretation of the differences. It is especially useful for applications in the electromagnetic compatibility (EMC) community.

 

Remember that no single metric can capture all aspects of the model's performance, so it is often helpful to use multiple metrics or visualization techniques to get a complete picture of the model's performance.

 

Back to Home