Apache OpenOffice (AOO) Bugzilla – Full Text Issue Listing |
Summary: | Wrong rank computation of sum values | ||||||
---|---|---|---|---|---|---|---|
Product: | Calc | Reporter: | twonkeys <holger> | ||||
Component: | code | Assignee: | AOO issues mailing list <issues> | ||||
Status: | CONFIRMED --- | QA Contact: | |||||
Severity: | Normal | ||||||
Priority: | P3 | CC: | discoleo, issues | ||||
Version: | OOO310m19 | ||||||
Target Milestone: | --- | ||||||
Hardware: | PC | ||||||
OS: | Windows XP | ||||||
Issue Type: | DEFECT | Latest Confirmation in: | 3.4.1 | ||||
Developer Difficulty: | --- | ||||||
Attachments: |
|
Description
twonkeys
2009-12-07 21:10:15 UTC
Created attachment 66544 [details]
Wrong computation of rank
I just double checked this file on a x86_64 Linux system (Ubuntu 9.10) with the exact same OO build number (OOO310m19). The behaviour is equally wrong here. Weird issue, probably because of rounding in one case, and no rounding in the other case. The 2 sums are probably different using floating point arithmetic, though Clac is likely to round them to the nearest "shortest number" when displaying the result. This result may however behave oddly in subsequent operations: the rounded value might be used by the IF() function, while the extended result is used by the RANK() function. The handling of floating point results in Calc needs to be confirmed by a someone familiar with the code, though (either Eike or Niklas). At first glance this does seem like a numeric precision limitation, but when I display the numbers to fill display precision, the first set is the same: 48.70000000000000000000 According to the ODF specification, "If a number in Data occurs more than once it is given the same rank, but increments the rank for subsequent different numbers". So I'm going to mark this as confirmed, and recommend that a developer investigate to see what is going on here. It might be just a precision issue in the end, but it might be something else. |