Issue 12889 - Optimize psprint's font collection / use of pspfontcache
Summary: Optimize psprint's font collection / use of pspfontcache
Status: CLOSED FIXED
Alias: None
Product: gsl
Classification: Code
Component: code (show other issues)
Version: 644
Hardware: All All
: P3 Trivial (vote)
Target Milestone: OOo 1.1 Beta2
Assignee: matthias.huetsch
QA Contact: issues@gsl
URL:
Keywords:
: 12619 (view as issue list)
Depends on:
Blocks:
 
Reported: 2003-03-31 20:49 UTC by matthias.huetsch
Modified: 2003-04-29 12:49 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description matthias.huetsch 2003-03-31 20:49:29 UTC
Hi Phillip,

During startup 'SalGraphics::GetDevFontList()' on unix systems does call
'psp::PrintFontManager::getFontList()' to collect the list of fonts installed
on the system.

Currently 'getFontList()' scans all known font directories and does a 'stat()'
system call on every font file, e.g.:

open64("/usr/j2se/jre/lib/fonts", O_RDONLY|O_NDELAY) = 14
27450:  getdents64(14, 0x000EE4C0, 1048)                = 456
27450:  stat("/usr/j2se/jre/lib/fonts/.", 0xFFBEDAE8)   = 0
27450:  stat("/usr/j2se/jre/lib/fonts/..", 0xFFBEDAE8)  = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaBrightDemiBold.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaBrightDemiItalic.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaBrightItalic.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaBrightRegular.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaSansDemiBold.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaSansRegular.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaTypewriterBold.ttf", 0xFFBEDAE8) = 0
27450:  stat("/usr/j2se/jre/lib/fonts/LucidaTypewriterRegular.ttf", 0xFFBEDAE8) = 0

This strategy is applied to possibly hundreds of font files reachable through
the file system. Even worse, at times it looks like this:

open64("/usr/openwin/lib/X11/fonts/F3", O_RDONLY|O_NDELAY) = 14
27450:  getdents64(14, 0x000EE4C0, 1048)                = 1032
27450:  stat("/usr/openwin/lib/X11/fonts/F3/.", 0xFFBEDAE8) = 0
27450:  stat("/usr/openwin/lib/X11/fonts/F3/..", 0xFFBEDAE8) = 0
27450:  stat("/usr/openwin/lib/X11/fonts/F3/afm", 0xFFBEDAE8) = 0
27450:  stat("/usr/openwin/lib/X11/fonts/F3/map", 0xFFBEDAE8) = 0
27450:  stat("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Book.f3b", 0xFFBEDAE8) = 0
27450:  lstat("/usr/openwin/lib/X11/fonts/F3/pspfontcache", 0xFFBED394) Err#2 ENOENT
27450:  open("/usr/openwin/lib/X11/fonts/F3/pspfontcache", O_RDONLY) Err#2 ENOENT
27450:  stat("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Demi.f3b", 0xFFBEDAE8) = 0
27450:  lstat("/usr/openwin/lib/X11/fonts/F3/pspfontcache", 0xFFBED394) Err#2 ENOENT
27450:  open("/usr/openwin/lib/X11/fonts/F3/pspfontcache", O_RDONLY) Err#2 ENOENT
27450:  stat("/usr/openwin/lib/X11/fonts/F3/Courier-Bold.f3b", 0xFFBEDAE8) = 0
27450:  lstat("/usr/openwin/lib/X11/fonts/F3/pspfontcache", 0xFFBED394) Err#2 ENOENT
27450:  open("/usr/openwin/lib/X11/fonts/F3/pspfontcache", O_RDONLY) Err#2 ENOENT
27450:  stat("/usr/openwin/lib/X11/fonts/F3/Courier-BoldOblique.f3b",
0xFFBEDAE8) = 0
27450:  lstat("/usr/openwin/lib/X11/fonts/F3/pspfontcache", 0xFFBED394) Err#2 ENOENT
27450:  open("/usr/openwin/lib/X11/fonts/F3/pspfontcache", O_RDONLY) Err#2 ENOENT
27450:  stat("/usr/openwin/lib/X11/fonts/F3/Courier-Oblique.f3b", 0xFFBEDAE8) = 0

Here, for every font file, a 'pspfontcache' file is accessed twice via 'lstat()'
and 'open()'. There are 3 things wrong with this approach:

1) The ENOENT result of 'lstat()' is ignored, leading to the unnecessary
   'open()' call, failing with ENOENT as well.

2) This sequence is re-tried before accessing each and every font, instead of
   once per directory.

3) The probability that a 'pspfontcache' exists under a system fontpath like
   '/usr/openwin/lib/X11/fonts/...' seems to be equal to zero, anyhow.

These possibly hundreds (if not more) of (file-) system calls during each and
every startup do significantly contribute to the total (cold) startup time.

As the font installation on any particular system is most probably only slowly
varying, I'd like to propose a different strategy to be followed until we can
make widespread use of 'fontconfig' or better.

The current strategy of collecting the fonts should be used once, only,
preferably during setup, but the first startup after setup will do as well.
The collected font information is then saved in a 'pspfontcache' file, either
in the 'share' or a 'user' directory of an office installation.

During normal office startup, the information in 'pspfontcache' should be
regarded as 'trusted' and valid, and at a maximum re-verified against the
'fonts.dir' contents of the individual font directories. Only when actually
needing a particular font, that font should be re-verified by accessing the
respective font file.

I'm leaving it up to you to come up with a better strategy than mine. The only
thing that counts is to avoid the currently enormous number a file system
accesses during startup.

Thanks,
Matthias
Comment 1 matthias.huetsch 2003-03-31 20:50:08 UTC
Added target milestone 'OOo 1.1 Beta2'.
Comment 2 philipp.lohmann 2003-04-01 08:05:16 UTC
The stat and subsequent open is not due to psprint but to
SvFileStream; anyhow the cache's contents will now be accepted until
the mtime of a font directory changes.
Comment 3 philipp.lohmann 2003-04-01 08:05:32 UTC
fixed in vcl07
Comment 4 philipp.lohmann 2003-04-01 15:34:02 UTC
*** Issue 12619 has been marked as a duplicate of this issue. ***
Comment 5 jameslee 2003-04-02 09:57:23 UTC
> ------- Additional Comments From pl@openoffice.org  2003-04-01 07:34
PST -------
> The problem is fixed 2 ways:
> 1) the behaviour was intended since the font cache (that is the C++
> object behind the file pspfontcache) is supposed to check its
> consistency; therefore it checks for files inside the cache and
> removes them from the cache if either they are not available anymore
> or the have changed modification times

This won't clear the fonts from the list if the files are still there
because they were only linked with spadmin the first place. They are
all still there, I just asked to have there reference removed with
spadmin.

> 2) the above method of updating was not deemed fast enough and
> therefore the cache code was rewritten (see issue 12889) so that the
> font cache invalidates itself when a cached directory changes.

No, that won't work either because the directory with my fonts won't
change any more than the files inside of it.

Say, do you have a 2 Mega Byte font cache file that needs clearing? Do
you have >1000 fonts on your system? Try linking several hundred
fonts, then removing before approving any system for resetting the
pspfontcache file.

One would have hoped that my use of spadmin where I *explicitly*
instruct the system the fonts as no longer used would give the
necessary small hint that the fonts are no longer to be used.

My issue is not about how often pspfontcache is read nor what it does
when reading it, it's about why spadmin does not clear the list. I
think 12889 is different from 12619. I can see that if spadmin
*removes* a font then subsequently it will get tidied but this isn't
the case.
Comment 6 philipp.lohmann 2003-04-02 10:28:13 UTC
If yor font directory doesn't change then you haven't removed
anything, so the cache contents are still valid. If you removed any
link inside any directory then the directory WILL have changed,
regardless whether you do this with spadmin or by hand, so the cache
file will be invalidated. What more do you want ?
Comment 7 jameslee 2003-04-02 11:10:48 UTC
<watch_my_lips>
When I use spadmin to notify OOo that a font is no longer used I don't
want OOo to used it.
</watch_my_lips>

I used spadmin to "remove" my fonts but they are still in the font
cache file and still read on start up.

The fonts themselves are *NOT* removed when the linked option of
spadmin was used to install. The fonts stay where they where before,
during and after the use of spadmin and Ooo.
Comment 8 philipp.lohmann 2003-04-02 11:35:57 UTC
Don't you say. Since i wrote the thing i actually knew that. But since
the links are the only thing that matters to OOo (unless you have the
fonts in your regular fontpath anyway in which case it was not
necessary to "install" them in the first place) this will remove the
fonts out of OOo.

The one thing you do not seem to understand is that in OOo 1.0.* the
removed fonts are stat'ed, the cache notes that they are invalid and
removes them. Since this costs time on every startup it is not done
anymore, instead the cache for a directory will be invalidated when
said directory changes (e.g. if you add or remove a font or a link to
a font). Why would you care whether this is done at the end of spadmin
or the start of either spadmin or soffice ? It will cost you the same
time anyway.
Comment 9 jameslee 2003-04-02 11:57:42 UTC
> The one thing you do not seem to understand is that in OOo 1.0.* the
> removed fonts are stat'ed, the cache notes that they are invalid and
> removes them.

I now understand this is your intention. Please understand it's not
happening.


> Why would you care whether this is done at the end of spadmin
> or the start of either spadmin or soffice ?

But evidently it's never being done, not by spadmin nor by soffice.

My duty is to report this, not to understand why.
Comment 10 philipp.lohmann 2003-04-02 12:17:58 UTC
That explains it. I apologize, i did not understand from your
description that it does not work; i thought you simply wanted the
font cache file to update in spadmin. But i think the new
implementation should fix a not working cache, too. I'll check, if it
does.
Comment 11 jameslee 2003-04-02 13:45:04 UTC
Yes, I'm sorry but sometimes it is very difficult when the dumb user
(me) does not understand how the program works in order to explain a
problem. I have done another test and in fact the problem is slightly
different than I thought - and slightly worse, maybe the new
implementation won't fix the problem but I think we are getting closer
to understanding it.

$ cp ~/OpenOffice.org1.0.2/user/psprint/pspfontcache
~/pspfontcache.before

$ spadmin

and add a new font

(1 new font were added)
 ^          ^^^^ ;-)

Now:
$ diff ~/OpenOffice.org1.0.2/user/psprint/pspfontcache
~/pspfontcache.before

and wow! 105 new fonts have been added to this file. I seems that
*all* the fonts that were in directories use by spadmin to add were
added.

Sure enough:
$ truss OpenOffice.org1.0.2/soffice 2>&1 | grep myfontdir

shows the extra 105 xstat()s even though none of these fonts were
actually added; this time I only added one, not a full directory.

Removal with spadmin does not alter this file and the names remain
valid because they are vaild paths to my font storage so the font
cache never twigs that they should be removed from the list.


Obviously your new way of checking the directory for changes will
speed things but it won't get rid of the underlying problem: spadmin
added these fonts to pspfontcache and they don't get removed.

....and while on this subject, you do realise there are many repeated
entries in the pspfontcache file?

For example the line:

OpenSymbol;0;5;5;2;65535;917;313;230;0;1262;1230;1262;1230;8

is repeated 574 times. Bloatware alert.
Comment 12 philipp.lohmann 2003-04-02 15:20:14 UTC
You found another bug with my implementation, so you cannot be a dumb
user :-) The pspfontcache file did not actually get truncated when it
was refreshed, so extra lines at the end tended to occur. But i got
this one nailed while fixing this issue, too. But i'll check this
issue with the new implementation, too.
Comment 13 philipp.lohmann 2003-04-02 20:33:17 UTC
I checked the new implementation and font import as well as removal
works as discussed now. Please have a look at beta 2 when it comes out.
Comment 14 jameslee 2003-04-03 08:17:46 UTC
> I checked the new implementation and font import as well as removal
> works as discussed now.

Great! Thank you for bearing with me on this.

> Please have a look at beta 2 when it comes out.

I wish!

Please ask you chums to compile one for me.
What is the problem with Solaris-x86?
The latest version is 1.0.2 and no SDK either.
Why is it any more complex than typing "make"?
(It is more complex to me because the makefile is broken with gcc.)
Comment 15 philipp.lohmann 2003-04-03 13:51:13 UTC
I don't know about Solaris X86, i think that's a question for
dev@porting.openoffice.org
Comment 16 philipp.lohmann 2003-04-04 10:45:24 UTC
changes are in vcl07
Comment 17 matthias.huetsch 2003-04-04 13:39:17 UTC
Hi Phillip,

so far, so good, i.e. looks much better. But one issue still remains:

There are some font directories (probably those where you were confused
with looking for a 'pspfontcache' file) still show not understandable
behaviour, e.g.:

open64("/usr/openwin/lib/X11/fonts/F3", O_RDONLY|O_NDELAY) = 18
4935:   fcntl(18, F_SETFD, 0x00000001)                  = 0
4935:   fstat64(18, 0xFFBECEF0)                         = 0
4935:   getdents64(18, 0x000EFF80, 1048)                = 1032
4935:   stat("/usr/openwin/lib/X11/fonts/F3/.", 0xFFBED238) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/..", 0xFFBED238) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/afm", 0xFFBED238) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/map", 0xFFBED238) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Book.f3b",
0xFFBED238) = 0
4935:   access("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Book.f3b", 4) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Demi.f3b",
0xFFBED238) = 0
4935:   access("/usr/openwin/lib/X11/fonts/F3/AvantGarde-Demi.f3b", 4) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/Courier-Bold.f3b",
0xFFBED238) = 0
4935:   access("/usr/openwin/lib/X11/fonts/F3/Courier-Bold.f3b", 4) = 0
4935:   stat("/usr/openwin/lib/X11/fonts/F3/Courier-BoldOblique.f3b",
0xFFBED238) = 0
4935:  
access("/usr/openwin/lib/X11/fonts/F3/Courier-BoldOblique.f3b", 4) = 0

Please have another look at it.

Thanks,
Matthias
Comment 18 philipp.lohmann 2003-04-04 14:42:52 UTC
FontCache did not contain directories in which no fonts were found, so
they were searched through every time. fixed now in vcl07.
Comment 19 philipp.lohmann 2003-04-04 15:22:20 UTC
new install sets on vcl07 available
Comment 20 matthias.huetsch 2003-04-04 15:49:12 UTC
Fix verified on 'cws_srx644_vcl07'. Looks good now!
Comment 21 matthias.huetsch 2003-04-04 15:51:07 UTC
Verified...
Comment 22 matthias.huetsch 2003-04-29 12:49:52 UTC
Verified integration into SRX644/m12s1 -> closing.