From: Simon Wistow
Date: 12:58 on 25 Jan 2006
Subject: python and its dlopen implementation
It's a lovely sunny Wednesday. I recline, trendily, in my Herman Miller
Aeron chair and take a sip from my Latte revelling in my meeja world
comfort.
I get a phone call.
The python wrapper round a C library I recently released a new version
of is not working claiming that it can't find some symbols.
"Oh!", I chuckle, "They probably just didn't set the LD_LIBRARY_PATH
right".
I investigate. Everything seems in order yet still these undefined
symbol error messages are appearing.
"How rum!" I think, switching inexplicably from meeja-luvvy to circa
1915 toff-about-town.
Even rummer is the fact that it's not *my* C library that's failing.
It's not even the C library that my lib links to. It's the plugins for
the second C library. In short, the situation looks like
foo.py OK
- libfoo.so OK
- libbar.so OK
- libbar_plugin1.so FAIL
- libbar_plugin2.so FAIL
- libbar_plugin3.so FAIL
And the plugins are failing because they can't find symbols in
libbar.so. Which are there because nm says they are and it's in the
LD_LIBRARY_PATH because libfoo.so is loading it.
I try gdb, I try Totalview. Nothing seems to explain the fact that this
just shouldn't be happening. It's almost like the LD_LIBRARY_PATH is
being clobbered between libbar.so and the plugins.
A cup of tea. And then a G&T for good measure.
Scratch my head.
No! It couldn't be.
Oh, yes it could.
Python, by default, dlopens its extensions which is sensible. And
depending on a compile time flag it uses RTLD_NOW or RTLD_LAZY.
Which is fine.
Except that means that symbols aren't propagated down through any
further dlopens like, for example, the ones that libbar.so is doing to
fetch its plugins. Both of those should be ORed against RTLD_GLOBAL
I'm sure there are reasons why not doing this is the default and, to be
fair, you can get that back by doing
import dl
import sys
flags = sys.getdlopenflags()
sys.setdlopenflags(flags | dl.RTLD_GLOBAL)
but this is "*HATES* software", not "Accept ugly workarounds as being
acceptable software"
KEEEL THEM. KEEEL THEM ALLL.
Oh look, now I'm some sort of cliched Mexican Bandit Lord.
Bastards have turned me mad.
From: Phil Pennock Date: 11:29 on 31 Jan 2006 Subject: Re: python and its dlopen implementation On 2006-01-25 at 12:58 +0000, Simon Wistow wrote: [ RTLD_GLOBAL not in python ] > I'm sure there are reasons why not doing this is the default and, Software hate of mine: RPC libraries which include some cryptographic checksums which are almost-MD5 but called MD5. MD5 with a different initialisation vector. But the internal library uses the same function names as the standard ones. So to get working MD5, either mess with the semi-automatic ordering of the libraries in the make(1) rules (oh yay, more fragility) or it's necessary to pull the RFC code into my_* routines and #define the regular names in a header, to make the project build again on a platform using that particular stack. I think that this was part of the Transarc DCE libraries, but it might have been another one, it's been a decade and the hate for DCE still lingers, festering, in my soul. DCE has many hates. It took SOAP for me to finally grudgingly admit DCE's good points. And that's just with regular build-time linkage. So yes, I can see why something working to the limited model of dlopen() would leave RTLD_GLOBAL off by default. ;^) RTLD_CHILD_VISIBLE would be a reasonable default, if it existed, but chasing down pollution of "standard" names in the global namespace from every little twerp writing some weird little c00l corner-case plugin for an esoteric library and dicking with things they don't understand would take more time and cause more hatred.
From: Simon Wistow Date: 11:33 on 31 Jan 2006 Subject: Re: python and its dlopen implementation On Tue, Jan 31, 2006 at 12:29:19PM +0100, Phil Pennock said: > Software hate of mine: RPC libraries which include some cryptographic > checksums which are almost-MD5 but called MD5. MD5 with a different > initialisation vector. But the internal library uses the same function > names as the standard ones. Ah, yes. I've had that problem with mysql and various crypt and hashing functions. I know that C is, for many things, by far the best langauge. It has many plus points and it even has a couple of charming quirks. But goddamn is it fucking retarded. There are so many little language features that would have made life these days so much easier.
From: Peter da Silva Date: 12:43 on 31 Jan 2006 Subject: Re: python and its dlopen implementation > I know that C is, for many things, by far the best langauge. It has > many > plus points and it even has a couple of charming quirks. > > But goddamn is it fucking retarded. There are so many little language > features that would have made life these days so much easier. What really torques me off is people who refuse to use the appropriate features in languages that have them. Namespaces in Tcl, for example. Don Libes CGI library doesn't use namespaces. That's OK, it was designed before namespaces. But it doesn't do what we did before namespaces and define names with a common prefix (like, say, cgi_) to keep things tidy. For example, to generate an HTML link (<a href...>...</a>) you use "link". But to be "helpful", it defines all its functions with names like "cgi_link". Then, at the last minute, it checks for a name collision and renames them to their final name (like, link) if the name isn't already used. So they don't override previously defined globals. But all that means is that instead of screwing over people who use other libraries, you screw over people who use YOUR library. Or people working on their code. I spent hours chasing that one down when I was working on wikit last weekend. Because of course I brought in another library that defined a link (TclX) and then I couldn't find the definition of the clobbered version of link anywhere. Because it wasn't ever defined as link. It was defined as cgi_link and renamed. HATE.
From: A. Pagaltzis Date: 15:11 on 31 Jan 2006 Subject: Re: python and its dlopen implementation * Peter da Silva <peter@xxxxxxx.xxx> [2006-01-31 13:45]: >Namespaces in Tcl, for example. Isn't Tcl just one single big hate anyway? Regards,
From: Peter da Silva Date: 16:38 on 31 Jan 2006 Subject: Re: python and its dlopen implementation On Jan 31, 2006, at 9:11 AM, A. Pagaltzis wrote: > * Peter da Silva <peter@xxxxxxx.xxx> [2006-01-31 13:45]: >> Namespaces in Tcl, for example. > Isn't Tcl just one single big hate anyway? No, some of us like using reflection for just about everything.
From: Artur Bergman Date: 17:00 on 31 Jan 2006 Subject: Re: python and its dlopen implementation On 31 Jan 2006, at 07:11, A. Pagaltzis wrote: > * Peter da Silva <peter@xxxxxxx.xxx> [2006-01-31 13:45]: >> Namespaces in Tcl, for example. > > Isn't Tcl just one single big hate anyway? > +1 bad memories of eggdrops in the back of my skull
From: peter (Peter da Silva) Date: 17:19 on 31 Jan 2006 Subject: Re: python and its dlopen implementation > bad memories of eggdrops in the back of my skull Eggdrop is to Tcl as Majordomo is to Perl.
Generated at 10:27 on 16 Apr 2008 by mariachi