|
| [WD19] Calculation with rounding on 6 decimals |
| Iniciado por guest, 31,oct. 2016 17:10 - 5 respuestas |
| |
| | | |
|
| |
| Publicado el 31,octubre 2016 - 17:10 |
Hi all,
I was under the impression that using currencies for calculations was the way to use (max.) six decimal places, but that does not seem to be necessarily so. I get a result with only four decimal places instead of the expected 6 decimals.
Now I have used numerics as intermediate fields and found it to be correct calculation. What causes the difference in the calculation of currencies?
lcyKgPrijs = TblSlachtrapport.DEF_PrijsNoteringWeekprijs // 0.595
lcyBtwPerc = TblBtw.BTWPercentage // 6
lcyKgPrijsExcl = round(lcyKgPrijs / (100.0 + lcyBtwPerc) * 100.0, 6)
// lcyKgPrijsExcl is : 0.5613 --> 4 decimals, but should be 6 decimals
lnKgPrijs is numeric(5,7) = TblSlachtrapport.DEF_PrijsNoteringWeekprijs // 0.595
lnBtwPerc is numeric(5,7) = TblBtw.BTWPercentage // 6
lcyKgPrijsExcl = Round(lnKgPrijs / (100.0 + lnBtwPerc) * 100.0, 6)
// lcyKgPrijsExcl is : 0.561321 is OK
Thanks for any hints. |
| |
| |
| | | |
|
| | |
| |
| Publicado el 31,octubre 2016 - 22:36 |
Hi,
Is ths a currency : "DEF_PrijsNoteringWeekprijs"
If so hwo is that field defined in the analysis?
Maybe a currency var takes on the state of this field. And A nemeric one doesnot.
Regards
Allard |
| |
| |
| | | |
|
| | |
| |
| Publicado el 01,noviembre 2016 - 00:00 |
Hi Stefan,
no miracle!
currency has 6 Digits for the decimal part.
your calculation first does: 0.595/106 -> 0.005613208
truncated to 6 decimals -> 0.005613
then multiply by 100 -> 0.5613
Change your calculation like (first multiply by 100):
lcyKgPrijsExcl = round(100 * lcyKgPrijs / (100.0 + lcyBtwPerc), 6)
then you get in the first step : 100*0.595 = 59.5
second step: 59.5/105 = 0.561321
In theory you could leave away the round function, because of the max 6 Digits of a currency, but would surely be not a good idea.
greetings
Erik |
| |
| |
| | | |
|
| | |
| |
| Publicado el 01,noviembre 2016 - 08:28 |
Sorry Allard for being not clear,
DEF_PrijsNoteringWeekprijs is indeed defined as currency in the analysis. Also the other fields in the first example are defined as currency. |
| |
| |
| | | |
|
| | |
| |
| Publicado el 01,noviembre 2016 - 09:43 |
oops:
second step: 59.5/106 = 0.561321 |
| |
| |
| | | |
|
| | |
| |
| Publicado el 01,noviembre 2016 - 10:42 |
Hi Erik,
thanks you for your explanation!
It makes clear what I have to do to make it work. |
| |
| |
| | | |
|
| | | | |
| | |
|