How to properly set precision of whole numbers and decimals? #285
-
I use EvalEx in one of my projects and some people report, that the returned number seems to not be accurate, as it rounds certain digits to 0s. As an example, just providing 1234567891 results in 1234568000 being returned which shouldn't be the case. Right now am I using the following: Expression exp = new Expression(expression);
return exp.eval().setScale(3, RoundingMode.HALF_UP); I assume to fix this would I use Expression exp = new Expression(expression);
return exp.setPrecision(10).eval().setScale(3, RoundingMode.HALF_UP); And if the above is indeed the case, would there be any way to make EvalEx only round the whole number, if A) the scale is set to zero (no decimals) and/or B) the decimals result in rounding the next whole number up? Example: |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 9 replies
-
No, precision/scale of a BigDecimal works by setting the significant number of digits, not decimal places. So if you have the value "123456798" and you set the precision to "3" you get "12300000". Same goes if you have "1.23456789" which becomes "1.23". What you seem to want to do is to set a "high enough" precision to do your calculation, and then round the result. |
Beta Was this translation helpful? Give feedback.
No, precision/scale of a BigDecimal works by setting the significant number of digits, not decimal places. So if you have the value "123456798" and you set the precision to "3" you get "12300000". Same goes if you have "1.23456789" which becomes "1.23".
What you seem to want to do is to set a "high enough" precision to do your calculation, and then round the result.