Apache OpenOffice (AOO) Bugzilla – Issue 15279
ODBC Performance problem
Last modified: 2006-05-31 14:29:06 UTC
I start an sql-odbc query (data->refresh range) in oo. approx. 1000 records are returned from our database in the spreadsheet. The odbc query takes about half an hour. (Which is not very workable). The same query in excel takes about 4! seconds. I analysed the ODBC-logs (oo-14mb, excel 200kb) and found oo is using a very 'clumsy' way of retrieving data (get descriptions from _all_ tables in the database and all fields from used tables. Also retrieving a record, is solved very complicated. MS-excel does the only right thing: init - query - get definition from selected fields - get data. The difference in the amount of data in the logfiles is typical for the performance problem. (let me add, the query-time was measured without logging...) I attached a zip file with another query (returned 10 records from 1 table) and 2 log files, which illustrate this problem also: The odbc-log for excel is 14kb, oo takes up to 3.4Mb. I deleted 300 table definitions from this file (not to worry: still 100 left) thus reducing the size to 1.7Mb. The zip file is (fortunately) only 33kb. I'm using oo1.1b2 on NT4sp6a with a pervasive database. Please contact me, if more info is needed. Thanks, Karel
Created attachment 6648 [details] sql statement and odbc logfiles
trix (for some reason I assume the real name is Karel :), thanks for reporting this. correcting sub component (http://www.openoffice.org/issues/describecomponents.cgi?component=database%20access), and resulting default owner. Marc, this sounds serious enough to be considered for 1.1. Unfortunately, I am here for the next three weeks :(. Can you please follow up on this, and if you can reproduce it, ask Ocke to have a look? Thanks.
Hi Karel, does this problem still ocur when you use ADO with Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c:\temp\ff.xls;Extended Properties=Excel 8.0; ? Best regards, Ocke
trix, which kind of data range did you create in Calc - via "Data|DataPilot|Start", or in some other way? At the very moment you created it, where there already that much records in the result like later on when you did the "refresh range"?
(may be a dumb question, but I am not very familar with all of Calc's data access features, so I a step-by-step description of what you're doing would be nice :)
trix, I played around with the whole thing, but was not able to get a sql log which is as strange as yours (in my logs, I see some redundant calls, but not too many). In your logs, I see a lot of unnecessary calls - if I interpret this right, all calls regarding the "Data" table are unnecessary. However, I have not been able to reproduce something similar. So please, can your give us as much information as possible about your scenario: - what kind of database (what version) - version of the driver you're using - structure of the table in question (if you could attach an empty version of it, this would be great) - what other tables exist in the DB? I have the impression (guessing only) that the bug may be related to the number (or even names?) of other tables which are present - do you have a chance to test the same with a stripped-down version of the database, where only the table you're interested in exists? If so, is the result the same? In addition, please describe in more detail what you're doing - you could even attach a sample Calc document containing the range (you might want to clear any confidential data, if any, before doing so). In particular: - how exactly do you create the range? - Is it based on a OOo-side query, or on a view? A step-by-step description might be usefull here. Please kindly help in reproducing this in our environment!
Hi folks I submitted a new bug recently that turns out to be very similar to this one. You can view the full write up at bug 19984. The behaviour is similar, but everything else is different. Information on a procedure to attempt to reproduce the issue is included in the report, and I've just confirmed that the problem is still present in RC5. I was using networked database access using MySQL Connector/J (latest version - 3.0.8) to talk to a MySQL v3.23.54 server. I'm not only seeing every record between record 1 and the target record being requested, but also each record seems to be fetched three times. A full libpcap packet trace is attached to bug 19984. Note that I do _not_ see this problem when "seeking" around in a form, nor in the database explorer. It does not happen when dragging & dropping rows from the data explorer into writer, either. It only seems to happen when copying (well, dragging and dropping) records from the data explorer into calc.
Repeating my comment on #19984. I'm inclined to agree this is an OO.org problem, and specifically something in Calc. I'm suffering it with the DataPilot. It's so slow the DataPilot is usless. Comparing on a K6-2 400, 64M SunPCI card, in Win98. Using MS ODBC and MyODBC 3.51. This is a small table, 32900 rows, compared to the final desired size, with only 10 columns. Creation a simple pivot table in Excel97 takes ~1 minute, probably due to the low memory/speed of the client. mysqld never takes much CPU. Trying the same with OO.org 1.1 and the DataPilot is horrible. Just to get to the Layout stage take 20 minutes. After laying out the 5 cloumns I need, it take 50 minutes for the table to appear in Calc. Between selecting the table and the layout, or after hitting OK for the layout, OO.or doesn't refresh, and gives no indication of progress. I have also tried this with unixODBC and MyODBC on an Ultra10 with Solaris8, and on the RHL-9 system running MySQL. In both case OO.org takes well over an hour to create this simple table. SQL querries from isql (ODBC) or OO.org are nearly instant. Added to the speed differenc with Excel97 (Same machine, same drivers, same task) it appears the the DataPilot/Calc database interface needs serious work. Not in other comment: I do get huge logs from myodbc. Current waiting for DataPilot to get to the layout stage while logging. It's been running 12+ hours, and the log is 160MB (still growing). This database only has 2 small tables in it, with few columns. I'm about to runout of space for the log file too. The ODBC logs aren't as large. I'll try to run it, just logging ODBC (not the MyODBC driver itself). OO.,org is so unresponsive, I cannot stop the current operation!
Also change the OS to all. I've seen it on linux, Solaris, and Win98. These are all running OO.org 1.1 final.
Trix: Frank requested additional information to confirm/reproduce several months ago -- perhaps you missed the message. Also, the similar referenced Issue 19984 has been confirmed and set to "new" -- there has been quite a bit of discussion as to whether or not this issue is a dup. You might want to take a look at that issue and see if there is anything you can add to that, or if the information there helps you at all. Adding keyword.
Hi all, no response from submitter for a long long time. I will close this as invalid. However fell free to reopen this issue if it still occurr. If you reopen this issue please provide a step-by-step description how I can reproduce this issue. Bye Marc
closed
change subcomponent to 'none'