Issue 15279 - ODBC Performance problem
Summary: ODBC Performance problem
Status: CLOSED NOT_AN_OOO_ISSUE
Alias: None
Product: Base
Classification: Application
Component: code (show other issues)
Version: OOo 1.1 Beta2
Hardware: PC All
: P1 (highest) Trivial with 2 votes (vote)
Target Milestone: ---
Assignee: marc.neumann
QA Contact: issues@dba
URL:
Keywords: oooqa
Depends on:
Blocks:
 
Reported: 2003-06-04 11:38 UTC by Unknown
Modified: 2006-05-31 14:29 UTC (History)
2 users (show)

See Also:
Issue Type: PATCH
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
sql statement and odbc logfiles (32.86 KB, application/octet-stream)
2003-06-04 11:40 UTC, Unknown
no flags Details

Note You need to log in before you can comment on or make changes to this issue.
Description Unknown 2003-06-04 11:38:44 UTC
I start an sql-odbc query (data->refresh range) in oo.
approx. 1000 records are returned from our database in the
spreadsheet. The odbc query takes about half an hour. (Which is not very
workable). The same query in excel takes about 4! seconds.

I analysed the ODBC-logs (oo-14mb, excel 200kb) and found oo is using a
very 'clumsy' way of retrieving data (get descriptions from _all_ tables
in the database and all fields from used tables. Also retrieving a record, is
solved very complicated. 
MS-excel does the only right thing: init - query - get definition from selected
fields - get data. The difference in the amount of data in the logfiles is
typical for the performance problem. (let me add, the query-time was measured
without logging...)

I attached a zip file with another query (returned 10 records from 1 table) and
2 log files, which illustrate this problem also: The odbc-log for excel is 14kb,
oo takes up to 3.4Mb. I deleted 300 table definitions from this file (not to
worry: still 100 left) thus reducing the size to 1.7Mb. The zip file is
(fortunately) only 33kb.

I'm using oo1.1b2 on NT4sp6a with a pervasive database.
Please contact me, if more info is needed.

Thanks, Karel
Comment 1 Unknown 2003-06-04 11:40:57 UTC
Created attachment 6648 [details]
sql statement and odbc logfiles
Comment 2 Frank Schönheit 2003-06-05 07:38:14 UTC
trix (for some reason I assume the real name is Karel :), thanks for
reporting this.

correcting sub component
(http://www.openoffice.org/issues/describecomponents.cgi?component=database%20access),
and resulting default owner.

Marc, this sounds serious enough to be considered for 1.1.
Unfortunately, I am here for the next three weeks :(. Can you please
follow up on this, and if you can reproduce it, ask Ocke to have a
look? Thanks.
Comment 3 ocke.janssen 2003-06-18 14:55:22 UTC
Hi Karel,

does this problem still ocur when you use ADO with
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c:\temp\ff.xls;Extended
Properties=Excel 8.0;
?


Best regards,

Ocke
Comment 4 Frank Schönheit 2003-06-30 19:40:02 UTC
trix, which kind of data range did you create in Calc - via
"Data|DataPilot|Start", or in some other way? At the very moment you
created it, where there already that much records in the result like
later on when you did the "refresh range"?
Comment 5 Frank Schönheit 2003-06-30 19:41:59 UTC
(may be a dumb question, but I am not very familar with all of Calc's
data access features, so I a step-by-step description of what you're
doing would be nice :)
Comment 6 Frank Schönheit 2003-06-30 20:37:02 UTC
trix, I played around with the whole thing, but was not able to get a
sql log which is as strange as yours (in my logs, I see some redundant
calls, but not too many).
In your logs, I see a lot of unnecessary calls - if I interpret this
right, all calls regarding the "Data" table are unnecessary. However,
I have not been able to reproduce something similar.

So please, can your give us as much information as possible about your
scenario:
- what kind of database (what version)
- version of the driver you're using
- structure of the table in question (if you could attach an empty
version of it, this would be great)
- what other tables exist in the DB? I have the impression (guessing
only) that the bug may be related to the number (or even names?) of
other tables which are present
- do you have a chance to test the same with a stripped-down version
of the database, where only the table you're interested in exists? If
so, is the result the same?

In addition, please describe in more detail what you're doing - you
could even attach a sample Calc document containing the range (you
might want to clear any confidential data, if any, before doing so).
In particular:
- how exactly do you create the range?
- Is it based on a OOo-side query, or on a view?

A step-by-step description might be usefull here.

Please kindly help in reproducing this in our environment!
Comment 7 ringerc 2003-09-25 08:39:44 UTC
Hi folks

I submitted a new bug recently that turns out to be very similar to
this one. You can view the full write up at bug 19984. The behaviour
is similar, but everything else is different. Information on a
procedure to attempt to reproduce the issue is included in the report,
and I've just confirmed that the problem is still present in RC5.

I was using networked database access using MySQL Connector/J (latest
version - 3.0.8) to talk to a MySQL v3.23.54 server. I'm not only
seeing every record between record 1 and the target record being
requested, but also each record seems to be fetched three times. 

A full libpcap packet trace is attached to bug 19984.

Note that I do _not_ see this problem when "seeking" around in a form,
nor in the database explorer. It does not happen when dragging &
dropping rows from the data explorer into writer, either. It only
seems to happen when copying (well, dragging and dropping) records
from the data explorer into calc.
Comment 8 stdog 2003-10-16 18:33:35 UTC
Repeating my comment on #19984.

I'm inclined to agree this is an OO.org problem, and specifically
something in Calc. I'm suffering it with the DataPilot. It's so slow
the DataPilot is usless.

Comparing on a K6-2 400, 64M SunPCI card, in Win98. Using MS ODBC and
MyODBC 3.51. This is a small table, 32900 rows, compared to the final
desired size, with only 10 columns.

Creation a simple pivot table in Excel97 takes ~1 minute, probably due
to the low memory/speed of the client. mysqld never takes much CPU.

Trying the same with OO.org 1.1 and the DataPilot is horrible. Just to
get to the Layout stage take 20 minutes. After laying out the 5
cloumns I need, it take 50 minutes for the table to appear in Calc.
Between selecting the table and the layout, or after hitting OK for
the layout, OO.or doesn't refresh, and gives no indication of progress.

I have also tried this with unixODBC and MyODBC on an Ultra10 with
Solaris8, and on the RHL-9 system running MySQL. In both case OO.org
takes well over an hour to create this simple table.

SQL querries from isql (ODBC) or OO.org are nearly instant. Added to
the speed differenc with Excel97 (Same machine, same drivers, same
task) it appears the the DataPilot/Calc database interface needs
serious work.

Not in other comment:
I do get huge logs from myodbc. Current waiting for DataPilot to get
to the layout stage while logging. It's been running 12+ hours, and
the log is 160MB (still growing). This database only has 2 small
tables in it, with few columns. I'm about to runout of space for the
log file too.

The ODBC logs aren't as large. I'll try to run it, just logging ODBC
(not the MyODBC driver itself). OO.,org is so unresponsive, I cannot
stop the current operation!
Comment 9 stdog 2003-10-16 18:35:39 UTC
Also change the OS to all. I've seen it on linux, Solaris, and Win98.
These are all running OO.org 1.1 final.
Comment 10 tamblyne 2003-11-01 22:33:21 UTC
Trix:  Frank requested additional information to confirm/reproduce
several months ago -- perhaps you missed the message.  

Also, the similar referenced Issue 19984 has been confirmed and set to
"new" -- there has been quite a bit of discussion as to whether or not
this issue is a dup.  You might want to take a look at that issue and
see if there is anything you can add to that, or if the information
there helps you at all.  

Adding keyword.  


Comment 11 marc.neumann 2004-01-12 15:14:36 UTC
Hi all,

no response from submitter for a long long time. 

I will close this as invalid. However fell free to reopen this issue if it still
occurr. If you reopen this issue please provide a step-by-step description how I
can reproduce this issue.

Bye Marc
Comment 12 marc.neumann 2004-01-12 15:14:59 UTC
closed
Comment 13 hans_werner67 2004-02-02 12:24:14 UTC
change subcomponent to 'none'