TeePoly epsilon value
Posted: Tue Aug 07, 2018 4:48 pm
TeeChart Pro v2018.24.180321, Delphi Tokyo 10.2.3.
TeePoly defines Float as Extended. The call in PolyFitting to GaussianFitting uses a fixed epsilon of 1.0e-15.
When TeePoly is compiled for x64, Extended is mapped to Double by the compiler. This causes the epsilon to have different results for Extended vs. Double. As seen in the System.Math unit, IsZero uses an epsilon based on the floating point type (as does TeEngine.IsZero).
Our code has a fitting process that fails in 32-bit mode (using extended) whereas it is fine in x64 (using double).
The change I made here to TeePoly is as follows:
- Deleted "uses Classes" from the interface (it is not used).
- Added "uses Math" to the implementation.
- Deleted the "Error" parameter from GaussianFitting.
- Where Error was used, changed to use Math.IsZero(). (This is in two places, and there is no need for the Abs() call when using IsZero.)
The PolyFitting procedure now works identically in 32 or 64-bit compilations, with float defined as Extended or Double.
TeePoly defines Float as Extended. The call in PolyFitting to GaussianFitting uses a fixed epsilon of 1.0e-15.
When TeePoly is compiled for x64, Extended is mapped to Double by the compiler. This causes the epsilon to have different results for Extended vs. Double. As seen in the System.Math unit, IsZero uses an epsilon based on the floating point type (as does TeEngine.IsZero).
Our code has a fitting process that fails in 32-bit mode (using extended) whereas it is fine in x64 (using double).
The change I made here to TeePoly is as follows:
- Deleted "uses Classes" from the interface (it is not used).
- Added "uses Math" to the implementation.
- Deleted the "Error" parameter from GaussianFitting.
- Where Error was used, changed to use Math.IsZero(). (This is in two places, and there is no need for the Abs() call when using IsZero.)
The PolyFitting procedure now works identically in 32 or 64-bit compilations, with float defined as Extended or Double.