Log in

DSO memory waste II - Ulrich Drepper [entries|archive|friends|userinfo]
Ulrich Drepper

[ website | My Website ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

DSO memory waste II [Jun. 15th, 2006|01:13 am]
Ulrich Drepper
I want to follow up on the previous posting by giving answers to three questions.

The first question was about dlopen() and and when it should be used instead of linking with DSOs which are only sometimes used. Examples of these DSOs are crypto/authentication libraries like Kerberos. It's not really possible to give a clear answer. It depends. The main factor in using dlopen() is that the program explicitly has to find symbols using dlsym() or dlvsym(). If only a few symbols from a DSO have to be used explicitly this is no big issue. In that case the program code hardly has to be changed and the threshold for the ratio of users who need the feature vs those who don't can be really high. Maybe even a 40%:60% split would justify using dlopen().

If the program has to be substantially changed the situation may be a little bit more different. It is a one-time change so this is a weak argument. But it might be a way to determine low-hanging fruits wrt to this optimization. Anyway, unless the DSO is a real infrastructure library (like libc) the number of interfaces used should be low. This means almost all cases where a DSO is not needed by >50% of the people is a candidate for using dlopen().

There are two things to keep in mind. First, using direct linking will automatically take advantage of the symbol versioning which fortunately is picked up by more and more DSOs. If dlopen() is used one has to explicitly do this. I.e., it is the programmers responsibility to use dlvsym() instead of dlsym() and pass the right version name. Failure to do so would rob the DSO author of the possibility to make incompatible changes. If dlsym() is used it will always pick the latest version which might not be what is wanted. Second, using dlopen() means that prelinking won't help and that the DSO and its dependencies has to be relocated.

The second question was about the supposedly ugly code sequence which I gave (the string table). None of the people who complained about this obviously read appendix B of the DSO HowTo. I'm giving there a nice and clean (IMO) way to accomplish building the string and offset tables. No calculating the offsets by hand. I'm using that techniques in many places myself, it's easy to maintain code. Then there were people who complained that gcc should be able to perform the transformation itself. Yes, in theory. You're welcome to write the necessary optimization phase for gcc. It's very complex and not high enough on anybody's to-do list. I have been talking to the gcc people about some related extensions to gcc but those haven't been implemented yet either. In short, waiting for the compiler to solve the problems isn't going to help us in the foreseeable future. We need manual edits.

The third question/comment was why --as-needed isn't just always used. This is just as stupid as always use hidden symbols visibility via the new gcc options like certain distributions do it now for KDE. Both these methods change the ABI of a program sooner or later. Obviously, default hidden visibility prevents symbols from being exported unless specified otherwise. This means it'll break some code and the appropriate visibility attributes have to be added and then new binaries have to be provided. That's nothing a sane person should do. Similarly for --as-needed. The linked-in DSO form a predictable search scope for symbol lookups. Removing a DSO will change the search scope and some lookups will suddenly find a different definition or will fail. This will only affect symbol lookups performed dynamically via dlsym() but it has happened.

Fortunately this is a rare occurrence but nevertheless reason enough to not use --as-needed explicitly. Add the option to the Makefiles in the the right places, it's not hard. I seem to remember that somebody said this would create problems with libtool. Well, too bad. I've said all along the libtool of today is one of the biggest hindrances of development and deployment and shouldn't be used. When we wrote the first libtool version it was only meant to iron over the minor differences between various ELF tools. Since then it has been hijacked by those idiots who wanted to support the moronic dynamic linking methods used on winblowz and hpsux. Ever since that happened libtool is nothing but a problem for sane ELF platforms and should be avoided [1]. So, just avoid using libtool and link normally with the gcc driver. Then using --as-needed is no problem at all.

[1] Including the wrappers around dlopen() and dlsym() which are completely unnecessary on Linux and just make programs slow and bloated.