G-Engine uses various third-party libraries: ffmpeg for video playback, fmod for audio playback, zlib for decompression, etc. In all these cases, the library is included as a “dynamic library” (as opposed to a “static library”).

On Windows, when an executable needs a dynamic library, it searches for it in a few predefined locations, such as “the same directory as the executable”. On Mac and Linux, however, the situation is different and requires some consideration.

I was recently learning how Mac and Linux machines load dynamic libraries, so I thought I’d write a quick post about it.

How Dynamic Libraries Work

If you download someone else’s code and want to use it in your application, there are a few ways you could go about it:

Copy the source files directly into your own source code directory and compile them as part of your application.
Create a “static library,” which can then be included in your executable when you compile it.
Create a “dynamic library,” which can then be referenced by your executable when it runs.

Options 1 and 2 result in the external code residing directly inside your executable, so it actually causes the executable’s size to increase. This is fine in some cases, but there can be legal and functional reasons that you do not want to include someone else’s code directly in your executable.

Dynamic libraries allow you to store compiled code in separate files that are loaded by your executable at runtime. When you start the executable, it reviews its list of dynamic libraries and tries to load them.

You can spot dynamic libraries by their extensions. On Windows, these are .dll files. On Mac, these are .dylib files (or .framework files, which you can think of as “fancy” dynamic libraries). On Linux, these are .so files.

There’s a possibility that the executable will fail to find one or more libraries it needs to run. In this case, the application will likely abort and provide some sort of error message:

dyld: Library not loaded: libavcodec.dylib
  Referenced from: /Path/To/Executable
  Reason: image not found

This message indicates that the dynamic library loader failed to load a library referenced from the executable.

It’s also possible for dynamic libraries to reference other dynamic libraries. In this situation, the other dynamic libraries must also be present, or you’ll get a similar error.

How Dynamic Libraries are Located

On Mac and Linux, the system doesn’t make assumptions about the locations of dynamic libraries when you try to load them. Instead, the location of the dynamic library is actually embedded directly inside the executable. The executable loads each library in turn using the file path embedded inside the executable. And if that location is wrong, you get an error message like the one earlier.

In my (limited) experience, it’s common for the dynamic library location to be incorrect, especially if you are trying to bundle a library with your executable, rather than using one provided by the OS. It can be a frustrating and unexpected hurdle, but it isn’t too tricky to resolve on your own.

Viewing Dependencies

To view an executable’s or library’s dependencies, you can run this on the command line:

otool -l EXE_OR_LIB_PATH

Here’s a portion of the output for the G-Engine executable:

Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 902.1.0
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libfmod.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1.0.0
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name out/bin/libavutil.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 56.60.100
compatibility version 56.0.0

We’re primarily interested in blocks with the LC_LOAD_DYLIB keyword. This is the command to load a dynamic library at a particular path. If any of these fail to load because the path is invalid, the application will abort.

There are a few variations of dynamic library paths in that example:

The first one (libc++) uses an absolute path, which is common for libraries installed on the user’s machine. Some libraries are guaranteed to be installed by the OS, but that’s not always the case.
The second one (libfmod) uses a curious path with the @rpath keyword. We’ll discuss these keywords in a bit.
The third one (libavutil) uses a relative path, which is almost certainly wrong. These are relative to the directory you run the executable from, rather than the directory in which the executable is located. Since users can run executables from any arbitrary directory, this is usually not safe.

How Dynamic Library Paths are Chosen

The paths that otool lists are baked into the executable or library at build time. How are these paths determined?

The executable or library being built pulls the path directly from the dependent library. If you run otool -l <LIB> on the dependent library, there’ll be a LC_ID_DYLIB command.

For example, here’s what I see if I run otool -l libavutil.dylib:

Load command 4
          cmd LC_ID_DYLIB
      cmdsize 48
         name out/bin/libavutil.dylib (offset 24)
   time stamp 1 Wed Dec 31 16:00:01 1969
      current version 58.111.101
compatibility version 58.0.0

As you can see, this library’s LC_ID_DYLIB field is copied directly into the executable’s LC_LOAD_DYLIB field.

In other words, a dynamic library tells anyone using it where to load it at runtime.

During development, a library’s LC_ID_DYLIB field is often not equal to its actual location. For example, this library is located at Libraries/ffmpeg/lib/mac in the G-Engine repo, but after the game is built, the library is loaded from the runtime location specified by its LC_ID_DYLIB field.

How Dynamic Libraries Choose IDs

So, how does the dependent library choose it’s LC_ID_DYLIB field?

The LC_ID_DYLIB field is determined when the dependent library is built (whether through Xcode, a makefile, or some other means).

Often, this field is set assuming that the library will be “installed” on the local machine in a location like /user/local. When using a makefile, the default location is in fact /user/local unless you change it. In Xcode, this is specified using the “Dynamic Library Install Name” build setting.

To view a library’s LC_ID_DYLIB field, you can find the value in the otool output. Alternatively, you can run otool -D LIB_PATH, which will output only the ID field:

libavcodec.dylib:
ID_FIELD

Special Keywords for Library IDs

Let’s say we want to specify a dynamic library’s LC_ID_DYLIB field in such a way that the executable will always look in its own directory, or a directory relative to the executable, for the library.

Neither absolute nor relative paths can achieve this:

Absolute paths make assumptions about the user’s filesystem and what software they have installed.
Relative paths make assumptions about where the user will run the application from. If the user executes the program from a different directory, relative paths give the wrong result.

Fortunately, the operating system provides a few keywords we can include in LC_ID_DYLIB to overcome these problems.

@executable_path

This keyword resolves to the executable’s path at runtime.

Therefore, if we set a library’s LC_ID_DYLIB field to @executable_path/LIB_NAME.dylib, it will search for the library in the same directory as the executable. The seems to be exactly what we want, but there are reasons you might not choose this option.

@loader_path

This keyword resolves to the loading object’s path at runtime. Note I said “loading object” and not “executable.” This is primarily meant for situations where libraries depend on other libraries.

Let’s say we have this directory structure where the executable loads libcool.dylib, which in turn loads libinternal.dylib.

executable
/CoolLib
    libcool.dylib
    /InternalLib
        libinternal.dylib

Using @loader_path, the LC_ID_DYLIB field for libinternal.dylib can be set to @loader_path/InternalLib/libinternal.dylib. In other words, the path is relative to libcool.dylib instead of the executable.

You can use @loader_path when loading from the executable too - in this case, it has the exact same value as @executable_path.

@rpath

This keyword instructs the loader to search a list of paths to find the dynamic library. The list of paths is also embedded in the executable, in the LC_RPATH field. This is actually quite powerful because it lets the executable specify where the library will be located.

Let’s say you have a dynamic library that is used in two separate applications: one is a command line tool, and one is a macOS app. For the command line tool, we can put the library in the same directory as the executable. For the macOS app, we may want to put the library in a “Libraries” folder separate from the executable.

If we have a library with the LC_ID_DYLIB field set to @rpath/libcool.dylib, it can be used in both scenarios without modification. Each application just needs to specify an appropriate LC_RPATH field.

For the command line tool, we can use @executable_path as the LC_RPATH field. For the macOS app, we can use @executable_path/../Libraries.

Keep in mind that LC_RPATH can be a list of paths, rather than just one path. This is helpful if you want the executable to search for a library in multiple prioritized locations. For example, use a system library UNLESS the library is found in the executable directory.

Changing Dynamic Library IDs

Let’s say you have a dynamic library whose LC_ID_DYLIB field is incorrect. Perhaps the library was built with a different use case in mind, or perhaps the field was errantly set to something nonsensical when the library was built.

Fortunately, it’s possible to change this field without rebuilding the library using install_name_tool on the command line!

To change a library’s LC_ID_DYLIB field, simply use:

install_name_tool -id NEW_ID LIB_PATH

Changing Loader Paths

On the flipside, you may have an executable or library with an incorrect LC_LOAD_DYLIB field. Perhaps the library’s location changed or perhaps the value is errantly set.

This can also be achieved with install_name_tool. To change a library load path, use:

install_name_tool -change OLD_PATH NEW_PATH LIB_PATH

In this case, you must specify the old and new values, so it acts like a find/replace. This is because there may be numerous LC_LOAD_DYLIB fields, so you must specify exactly which one to change.

A Real-World Example: ffmpeg

I thought it’d be valuable to put the above info into a practical context. Since I recently wrangled ffmpeg into G-Engine, we’ll use that as an example.

ffmpeg has some characteristics that make it an interesting and complex example:

The build instructions for ffmpeg assume that you want to install the libraries on your local machine, rather than bundle them with an executable. So, we’ll need to do some custom processing of the libraries in order to bundle ffmpeg.
The built libraries contain versioned files with symlinks used for redirects. We’ll need to decide how to handle that.
ffmpeg consists of multiple dynamic libraries with interdependencies between one another. So, we’ll have to make sure not only that our executable can load an ffmpeg library, but also that the ffmpeg library can load its dependencies.

Building ffmpeg

You can download ffmpeg’s source code here. Once downloaded, INSTALL.md outlines how to build and install ffmpeg. At a high level, the steps are pretty simple:

Run configure to configure build options.
Run make to build the libraries.
Run make install to “install” the libraries.

The first step (configure) is potentially the most complex because of the sheer number of options available, plus it’s unlikely that the default options will be exactly what you want. Here’s what I used for G-Engine:

./configure --prefix=out --disable-static --enable-shared \
			--disable-doc --disable-avdevice --disable-avfilter --disable-postproc --disable-network --disable-debug \
			--disable-encoders --disable-muxers --disable-hwaccels --disable-parsers --disable-bsfs --disable-indevs --disable-outdevs --disable-filters \
			--disable-decoders --enable-decoder=bink --enable-decoder=binkaudio_rdft --enable-decoder=msrle --enable-decoder=pcm_s16le \
			--disable-demuxers --enable-demuxer=bink --enable-demuxer=avi \
			--disable-protocols --enable-protocol=file

This builds a shared library to the “out” directory, disables modules/features I don’t need and specifically enables the decoders, demuxers, and protocols that I do need. This is actually pretty important because it reduces the library size from 15MB to 464KB!

If you run make, the libraries are created, but they are all in separate locations in the directory structure. Running make install consolidates them all into the desired install directory (/user/local by default, but I switched it to a relative “out” directory using --prefix=out on the configure step).

So now, if I go to FFMPEG_DIR/out/lib, I see all the libraries in one place!

Handling Versioned Files and Symlinks

After building ffmpeg, the output files look something like this:

libavcodec.dylib (symlink to actual file)
libavcodec.56.dylib (symlink to actual file)
libavcodec.56.60.100.dylib (actual file)

Two of the files are “symlinks”, meaning they are aliases or redirects to another file. In this case, both symlinks redirect to the actual library file libavcodec.56.60.100.dylib.

Why is it structured this way? It allows installing and using any number of different versions of the library on your machine. Here are some examples:

Let’s say we install version 56.60.100 on our machine, and then we later install new version 56.60.200. The symlink libavcodec.56.dylib will update to point to the new version. Any application using that symlink is now automatically pointed to the new version.
Let’s say new version 57.0.0 is installed. The symlink libavcodec.dylib will always point to the latest version, so it updates to point at this new version. The libavcodec.56.dylib still points to the latest 56.x.x version. A new symlink called libavcodec.57.dylib exists for the 57.x.x libraries.

This kind of versioning logic is helpful to handle installations on a user’s machine, but again, that is not my use case: I want to bundle this library with an executable. I could use symlinks and library versioning, but it’s kind of overkill for my needs.

Therefore, I am planning to do the following: ignore the symlinks. Rename the actual file to just libavutil.dylib when copying it to the executable location.

Bundling the Library

When make creates the library file, it sets the library’s LC_ID_DYLIB field to the install directory. In my case, I changed the install directory with configure, so the field is set to out/lib/libavcodec.dylib.

Here’s what I get when I run otool -D libavcodec.dylib:

libavcodec.dylib:
out/bin/libavcodec.dylib

This ID is obviously incorrect if my intention is to bundle the library with my executable. To fix this, I need to run install_name_tool:

install_name_tool -id @rpath/libavcodec.dylib out/lib/libavcodec.dylib

Running otool -D libavcodec.dylib now gives me:

libavcodec.dylib:
@rpath/libavcodec.dylib

Because I used @rpath, the executable must specify a list of search paths in LC_RPATH. I’ll cover that shortly.

Changing Library Dependencies

You won’t run into this for every library, but ffmpeg is complex enough where it consists of multiple dynamic libraries with interdependencies. For example, the library libavutil.dylib is a dependency for most of the other libraries.

If we run otool -l libavcodec.dylib, we see this LC_LOAD_DYLIB entry:

Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name out/lib/libavutil.56.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 56.60.100
compatibility version 56.0.0

Again, this path makes sense based on how we used make install, but it won’t work when bundled with an executable. In my case, I plan to put all my library files into the same folder. So, I need to change this path using install_name_tool:

install_name_tool -change out/lib/libavutil.56.dylib @rpath/libavutil.dylib out/lib/libavcodec.dylib

You may notice that I replaced the versioned path with the unversioned path. As mentioned earlier, I am planning to eschew version data for the bundled versions of the library.

Now, running otool -l libavcodec.dylib reflects the change:

Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libavutil.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 56.60.100
compatibility version 56.0.0

In this case, we could use either @rpath or @loader_path - since all libraries will be in the same directory, they will both work.

Including Libraries in Executable

After generating the dynamic libraries and fixing up their LC_ID_DYLIB and LC_LOAD_DYLIB entries, I copied the libraries into a subdirectory of G-Engine (Libraries/ffmpeg/lib/mac).

From there, I can include the library by updating the library search paths and linker flags in build settings.

For the library search path, I added $(SRCROOT)/../Libraries/ffmpeg/lib/mac.
For linker flags, I added -lavutil -lavformat -lavcodec -lswresample -lswscale.

Finally, I have a script that copies libraries from dev locations to the executable directory after a build. I needed to add these lines:

cp "$SRCROOT"/../Libraries/ffmpeg/lib/mac/libavcodec.dylib $LIB_DIR
cp "$SRCROOT"/../Libraries/ffmpeg/lib/mac/libavformat.dylib $LIB_DIR
cp "$SRCROOT"/../Libraries/ffmpeg/lib/mac/libavutil.dylib $LIB_DIR
cp "$SRCROOT"/../Libraries/ffmpeg/lib/mac/libswresample.dylib $LIB_DIR
cp "$SRCROOT"/../Libraries/ffmpeg/lib/mac/libswscale.dylib $LIB_DIR

Note that these cp commands are copying the symlinks, but that’s OK - it properly follows the redirects, copying over the correct file to the executable directory with the correct name.

For me, LIB_DIR is actually different depending on whether I’m building a console app or a macOS app. For the console app, this is equal to the executable directory. For a macOS app, you can’t include libraries in the executable folder, so I put them in ../Libraries in the app bundle.

This also means that the executable’s LC_RPATH needs to be set directly, since I used @rpath for many of these libraries. For the console app, I used @executable_path for this. For the macOS app, I used @executable_path/../Libraries. These can be specified in Xcode in the “Runtime Search Paths” build setting.

Now, the engine runs while properly linking and including ffmpeg! But that’s just the start of the battle - actually using ffmpeg effectively is… uhhh… not exactly straightforward. I’ll chronicle that experience in a future blog post. :P

Other Notes

Before wrapping up, I wanted to address a couple loose ends.

Why @rpath?

At first, it seemed to me that I could simply use @executable_path for library LC_ID_DYLIB values. But then I realized that this was problematic when the library needs to be located in different locations (relative to the executable) in different situations.

So @rpath is actually quite useful and shouldn’t be discounted, especially if the library may need to be used by more than one executable.

What about Linux?

This post explains how to perform dynamic library operations on a Mac. On Linux, the situation is similar, but names, tools, and syntaxes are different.

Dynamic libraries on Linux have a .so extension, rather than .dylib. Library formats are also different. Mac libraries use the Mach-O while Linux typically uses the ELF format. This leads to a need for different tools.

Field names for IDs and loader paths are different. I found this explanation of RPATH enlightening.

Instead of otool to view library/executable dependencies, you can use ldd to get a list of dependencies. The tools nm and objdump also provide helpful output.

Instead of install_name_tool, you can use patchelf to change paths of already built executables or libraries. There is also a tool called chrpath that is more limited in functionality, but may be more widely available.

I don’t have specific instructions for Linux, as I’m less familiar with it, and I haven’t (yet) had to use it in practice. But hopefully this gives you a starting point.

Verifying/Troubleshooting Loaded Libraries

If you want to view what libraries an executable is loading, you can set the environment variable DYLD_PRINT_LIBRARIES to 1 before running the program. For example:

DYLD_PRINT_LIBRARIES=1 ./GEngine-MacOS

This produces a long list of libraries that the executable is loading. Here’s a snippet:

dyld: loaded: <793D9643-CD83-3AAC-8B96-88D548FAB620> /usr/lib/libz.1.dylib
dyld: loaded: <9ECC9339-79E4-3539-B907-0EC66E522467> /usr/local/lib/libGLEW.2.1.dylib
dyld: loaded: <8E048273-192C-3E52-A3E4-625409FEF4B9> /Users/Clark/Library/Developer/Xcode/DerivedData/GEngine-bkvltqfkyvldabcaovheyyhxdmji/Build/Products/Debug/libfmod.dylib
dyld: loaded: <A53A7AD0-F7AC-3B8C-BF67-C20D17B80E32> /Users/Clark/Library/Developer/Xcode/DerivedData/GEngine-bkvltqfkyvldabcaovheyyhxdmji/Build/Products/Debug/libavutil.dylib
dyld: loaded: <2AEC4083-1BAC-33D5-A7CE-F6D8162ADB84> /Users/Clark/Library/Developer/Xcode/DerivedData/GEngine-bkvltqfkyvldabcaovheyyhxdmji/Build/Products/Debug/libavformat.dylib

This actually revealed something surprising to me: despite my intention to bundle zlib and GLEW with my executable, it is actually using the system-installed libraries!

Upon closer inspection, this was because the LC_ID_DYLIB fields for those two libraries was not set correctly. But it still worked “by chance” because I happened to have those two libraries installed on my machine.

It goes to show - it’s a good idea to verify loader paths before distributing an executable!

Conclusion

If you’re new to developing with dynamic libraries, all this complexity can come as a bit of a surprise, and though it is all documented in one way or another, it’s rather scattered and hard to find.

Hopefully, this post provides a reasonably clear explanation of dynamic libraries, their LC_ID_DYLIB fields, how they’re loaded using LC_LOAD_DYLIB loader commands, and a concrete example of it all coming together.

For further reading, I found these resources helpful: