Nearly three years ago, I wrote a post about wrangling dynamic/shared libraries for Mac applications. I recently faced the same challenge on Linux, so I figured I’d write a follow up.
If you’re new to dealing with shared libraries, I’d recommend reviewing the previous post first - it covers some fundamentals that I won’t rehash here. Linux and Mac handle dynamic libraries in similar ways - but as you might guess, the tools used and some details vary.
How Shared Libraries Work
“Dynamic” and “shared” libraries are different names for the same thing. On Linux, the preferred term seems to be “shared”, so that’s what I’ll use in the rest of this post.
A shared library file represents compiled code that is distributed separately from your executable. On Linux, shared libraries are easily recognizable by their .so file extension.
Because a shared library is a separate file, the executable needs to load it at runtime before it can be used. While there are mechanisms for a programmer to do this manually, it’s also common for the “loader” to automatically load required shared libraries when starting an executable.
The problem with shared libraries on any platform is that they might not be present when you run the program. If that happens, you get an error message, and the program cannot run.
How Shared Libraries are Located
The executable contains an embedded list of shared libraries required to run. Only the name of the file is stored. The executable loader then searches an ordered list of paths to find a shared library with that name:
- The executable itself contains an embedded list of search paths for shared libraries. This is called the “rpath” or “run path”. Each path in this list is searched in turn.
- As a fallback, the system provides a list of search paths at
/etc/ld.so.conf. These are searched only if the library wasn’t found in the previous step. Each path in the list is searched in turn.
And that’s it! If a shared library with the searched for name is found, it is used. If the shared library isn’t found on any of those paths, you get an error, such as this one on Ubuntu:
EXE_NAME: error while loading shared libraries: LIB_NAME: cannot open shared object file: No such file or directory
Your goal is to have an executable whose list of embedded shared library filenames are either on a search path you specify via the “rpath”, OR are on a search path provided by the system. The former is usually meant for libraries specific to your application while the latter is meant for libraries that are known to be preinstalled on most/all Linux machines.
Viewing an Executable’s Dependencies
The list of shared libraries required by an executable can be viewed with the readelf tool:
readelf -d EXE_PATH | grep 'NEEDED'
The -d flag prints out the executable’s shared library info. Using the grep command is optional and simply filters the output to only show the shared libraries to be loaded (each has the “NEEDED” keyword next to it).
Here’s example output from my game engine project:
0x0000000000000001 (NEEDED) Shared library: [libavcodec.so.58]
0x0000000000000001 (NEEDED) Shared library: [libavformat.so.58]
0x0000000000000001 (NEEDED) Shared library: [libavutil.so.56]
0x0000000000000001 (NEEDED) Shared library: [libswresample.so.3]
0x0000000000000001 (NEEDED) Shared library: [libswscale.so.5]
0x0000000000000001 (NEEDED) Shared library: [libfmod.so.13]
0x0000000000000001 (NEEDED) Shared library: [libGLEW.so.2.1]
0x0000000000000001 (NEEDED) Shared library: [libz.so.1]
0x0000000000000001 (NEEDED) Shared library: [libSDL2-2.0.so.0]
0x0000000000000001 (NEEDED) Shared library: [libGL.so.1]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
Of the libraries listed, some I intend to bundle with the executable, while others I would assume to already be installed on the operating system:
- ffmpeg, fmod, GLEW, zlib, and SDL are all libraries I chose to use, so I will bundle them
- OpenGL, stdc++, gcc, and clib are fundamental OS features that I’d expect to be pre-installed by the OS
Of course, this means I need to make sure the bundled libraries are in a spot the executable can find them. To reiterate, this is done by setting an “rpath” value that is the path to the bundled libraries.
How Shared Library Names are Chosen
The library names that readelf outputs are baked into the executable at build time. How are those names determined?
The library name is pulled directly from the shared library itself. The value of the library’s SONAME field is used. You can use objdump to see a shared library’s SONAME:
objdump -p LIB_PATH | grep SONAME
For example, here’s what I see if I run objdump -p libavcodec.so:
SONAME libavcodec.so.58
You might wonder: why is the SONAME different from the file name? This relates to versioning strategies that are used by library maintainers. With this particular library, I actually have three files:
- libavcodec.so (symlink to actual file)
- libavcodec.58.so (symlink to actual file)
- libavcodec.58.91.100.so (actual file) (SONAME is libavcodec.58)
Specifying an SONAME with the “58” major version number indicates that the program is expected to work with any “58” variant of the shared library. If an end-user wanted, they could use an updated minor or patch release version of the “58” library (perhaps with some bug fixes present) and it should still work. On the other hand, if major version “59” was installed, it would not be considered.
Depending on your use-case, it may be desirable or undesirable to deal with a library’s version numbering when you bundle a shared library with your application. So, you may have a need to change the SONAME used by your executable.
Changing Shared Library Names
Let’s say you have a shared library whose SONAME is incorrect or just not what you want it to be. Fortunately, you can change it with the patchelf tool:
patchelf --set-soname NEW_SONAME LIB_PATH
You can set any SONAME you’d like, but remember: this value will be embedded in the executable and used to locate the shared library at runtime. So this should be the name of a file that is bundled with your application!
Changing Loader Names
On the flipside, you may have an executable with an incorrect shared library name embedded within it. Perhaps the library’s location changed or perhaps the value was errantly set. Fortunately, this can also be changed using patchelf:
patchelf --replace-needed OLD_NAME NEW_NAME EXE_PATH
In this case, you must specify the old and new names, so it acts like a find/replace. This is because there may be numerous library names listed in the executable, so you must specify exactly which one to change.
Verifying Loaded Libraries
After you’ve built your executable, it can be helpful to verify that it’s loading all its shared libraries from the locations you’re expecting.
One simple way to do this is the ldd command, which outputs the paths to all shared libraries that the application will use. These paths are calculated on-the-fly, so the output may change each time you run it and as you install or uninstall libraries from your system.
ldd EXE_PATH
Troubleshooting Loaded Libraries
At runtime, you may run into problems with shared libraries not being loaded successfully, or perhaps you’d like more output about the library load process. Running your application with a special flag lets you see all the details about which shared libraries are being loaded and which paths are being checked:
LD_DEBUG=libs EXE_PATH
Conclusion
If you compare this post to my previous one about Mac dynamic libraries, you’ll notice a lot of parallels. In some cases, all that’s changed is the name of the tool used to perform a certain action. That being said, I think loading shared libraries on Linux is actually a bit simpler than it is on Mac!
I hope this post provides a simple and straightforward explanation of how you can wrangle shared libraries in your Linux applications!