Issue 21305 - Datapilot very slow when using large datasource ( f.e. mysql odbc )
Summary: Datapilot very slow when using large datasource ( f.e. mysql odbc )
Status: CLOSED DUPLICATE of issue 19984
Alias: None
Product: Base
Classification: Application
Component: code (show other issues)
Version: OOo 1.1
Hardware: All All
: P3 Trivial with 2 votes (vote)
Target Milestone: OOo 2.0
Assignee: marc.neumann
QA Contact: issues@dba
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2003-10-16 10:52 UTC by marc.neumann
Modified: 2006-05-31 14:29 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---

Attachments
ODBC log from Calc (gzipped) (232.17 KB, application/octet-stream)
2003-10-22 15:55 UTC, stdog 		no flags Details
ODBC log from Excel (1.32 MB, text/plain)
2003-10-22 15:57 UTC, stdog 		no flags Details
ODBC log from Writer (3.33 MB, text/plain)
2003-10-22 15:59 UTC, stdog 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this issue.
Description marc.neumann 2003-10-16 10:52:13 UTC 
1. create a new mysql odbc datasource 
2. create a table
Mitarbeiter varchar
Region	    varchar
Monat       varchar
Verkauf     varchar
3. insert usefull records  (~100000 records)
4. start a new spreadsheet
5. start the datapilot and use the datasource and table
6. move the column Mitarbeiter to the left
7. move the column Region to the top
8. move the column Verkauf to the middel
9 click OK
==>> wait a long time til the table is insert. 
The same thing in Excel works without waiting.
Comment 1 marc.neumann 2003-10-16 10:52:35 UTC 
set target
Comment 2 stdog 2003-10-20 16:04:20 UTC 
It doesn't require nearly as many records.
Excel v/s OOo, on a 32k record table (slow Win98 machine)
is 1 minuter vs 1hr.

you missed a step too. between 5 and 6, insert:

5.b. Wait a long time for the layout dialog.

For my table, 5.b. take 20 minutes, and then another 50 minutes for
the table to insert.

Frank thinks the ODBC logs may help here. I need to create again, from
a smaller table. The 32k table created a 588M log, and I ran out of
disk space (Excel's log was only 7M).
Comment 3 stdog 2003-10-22 15:55:25 UTC 
Created attachment 10552 [details]
ODBC log from Calc (gzipped)
Comment 4 stdog 2003-10-22 15:57:52 UTC 
Created attachment 10553 [details]
ODBC log from Excel
Comment 5 stdog 2003-10-22 15:59:07 UTC 
Created attachment 10554 [details]
ODBC log from Writer
Comment 6 stdog 2003-10-22 16:11:02 UTC 
Attached 3 logs. The log from Calc is 14M uncompressed. Writer is 3.3M
and Excel is 1.3M.

These uses a 100 row table, with 4 columns.
ID = bigint, auto-increment, primary key
param = varchar, 100
value = double
pnum = varchar, 100


Excel is noticably faster, but writer isn't too bad. Still the form in
writer has a 3x larger log, so more ODBC calls are made.

Calc is still really slow, even with such a small table. And why would
it generate such a large logfile? At 14M it's 4x the writer log, and
13x the excel log. Something in the calc code is horribly wrong.

Niklas Nebel suggested the ScDBDocFunc::DoImport code here
sc/source/ui/docshell/dbdocimp.cxx, sc/source/ui/docshell/dbdocimp.cxx
and the code in sc/source/core/data/dpsdbtab.cxx
Comment 7 Frank Schönheit 2003-10-28 07:08:21 UTC 
removing dependency to issue 21783 (which is the ms interop tracking
bug) - I don't think this is an interop issue. If somebody
re-introduces the dependency, please next time justify it.
Comment 8 hans_werner67 2004-02-02 12:36:02 UTC 
change subcomponent to 'none'
Comment 9 ocke.janssen 2004-02-10 13:40:25 UTC 
.

*** This issue has been marked as a duplicate of 19984 ***
Comment 10 ocke.janssen 2004-02-10 13:43:47 UTC 
.
Comment 11 marc.neumann 2004-04-05 16:19:41 UTC 
.

*** This issue has been marked as a duplicate of 19984 ***
Comment 12 marc.neumann 2004-04-05 16:20:33 UTC 
close