Adventures in Text Encoding

Posted on August 3, 2026

For years, I’ve developed G-Engine and have paid zero attention to text encoding. When I’d store text in a C++ std::string or in a char array, it would just work. It turns out I was just lucky - I was writing text in English, and the default encoding just happened to work fine with English.

However, my luck ran out when I started dealing with Russian localized text and text input. Suddenly my text was garbled and illegible. Time to face the music and deal with the intricacies of text encoding!

This post dives into the history of text encodings and how you can write a program that has broad Unicode support (via UTF-8).

A Brief History of Text Encoding

Before diving into implementation details, I think a bit of historical context can help explain the core concepts of text encoding and also illustrate why we need to take certain actions today to achieve broad support for text in all human languages.

Note that this information isn’t necessarily 100% accurate with sources cited - but it is my understanding of the situation, and I think it’s accurate enough to effectively explain the situation to a programmer today. If you want the full story, dive down the Wikipedia rabbit hole!

The ASCII Standard

Computers only know how to store numbers, and one of the earliest problems programmers had to solve was how to represent human-readable text as numeric data. The solution was to create a mapping from specific numeric values to specific human-readable text characters. Such a mapping is known as a text encoding.

Way back in the 60s, early computers were already dealing with incompatible text encodings. Imagine if one government entity decides to map the letter A to the numeric value 1. But some other government entity decides to map the letter A to numeric value 100. Sharing computer data between those entities becomes a big problem!

As a result, the US established the ASCII standard text encoding in 1968. This encoding maps numeric values 0-127 to a standard set of characters including A-Z, a-z, 0-9, and common punctuation. The burgeoning US computer industry wisely adopted this standard, so virtually all computers used the same encoding. As long as you used English and only cared about American Dollars, you were all set.

Extended ASCII and Code Pages

ASCII does not include accented characters required to support many European languages nor does it include characters for Russian, Korean, Japanese, Mandarin, or other languages that don’t use the Latin alphabet. It also doesn’t support non-American currency symbols. Another system was needed to support these languages - and by “another system” I unfortunately mean “many different systems.”

Since a byte can store 256 unique values and ASCII only uses values 0-127, developers realized that they could map the remaining values 128-255 to other text characters to solve this problem. This was known as Extended ASCII.

But as many companies and entities rushed to do this, they created another problem: fragmentation and a lack of standardization. Many different encodings were invented that mapped different characters to different values. Each of these encodings were commonly referred to as a code page. Some examples:

  • Code Page 437: the original IBM PC character set.
  • ISO-8859-1: an ISO standard code page designed to support Western European languages.
  • Windows-1252: a widely used Microsoft Windows code page, similar to ISO-8859-1, also designed for Western European languages.
  • Mac OS Roman: a code page used by Macintosh computers.
  • Windows-1251: a code page for Cyrillic-based languages like Russian, Ukrainian, and Bulgarian.

Unfortunately, this was a messy situation. It was easy to accidentally interpret a text file using the wrong encoding, resulting in garbled or corrupted output (sometimes affectionately called “mojibake”). Text data typically did not specify which encoding to use, so you’d just have to know the correct encoding or use trial and error until the text looked correct.

(As an aside, even Extended ASCII was not enough to support Korean, Japanese, Mandarin, or other languages with more than 256 unique characters. In those regions, developers would create their own multi-byte encodings that only further complicated the situation.)

Here Comes Unicode

In an attempt to solve this fragmentation, Unicode was proposed in 1988 with the goal of providing a single encoding that could work for every human language. To do this, the initial proposal suggested using 2-byte characters so that 65,536 unique characters could be represented instead of just 256. Unfortunately, this concept was flawed - it turns out that there was a need to store many more than 65,536 unique characters!

Even more unfortunately, Microsoft was in a position where their Windows OS was extremely popular and they needed a globalized version of the OS quickly. So in 1993, they chose to support Unicode via the UCS-2 encoding, which uses 2 bytes per character.

  • The idea of wide characters (wchar_t) and wide strings (std::wstring) were added to C/C++. You needed to use these instead of the “narrow” char and std::string types to get Unicode support.
  • To handle both ASCII and Unicode strings, many Windows OS calls were branched to have ASCII (narrow) and Unicode (wide) variants - CreateFileA and CreateFileW for example.
  • The Microsoft version of the C runtime was also branched to have “narrow” and “wide” variants of common functions - strlen vs wcslen for example.

In hindsight, you could argue that this was all a mistake. Both Mac and Linux held off on adding Unicode support until the early 2000s and were able to avoid this misstep. They did so by embracing a new encoding that sidesteps all these issues AND was capable of supporting ANY unique character imaginable…

UTF-8

Unlike both ASCII and UCS-2, which are fixed-length encodings, the key to truly solving the encoding problem and supporting any number of unique characters was to switch to a variable-length encoding, the most popular of which is UTF-8.

UTF-8 works by simply adding more bytes when they are needed. The first byte encodes how many subsequent bytes are used. It’s fully backwards compatible with standard ASCII. For anything beyond, 2-4 bytes are used.

The beauty of UTF-8 is that it allows programmers to just continue using the normal char and std::string types in code. In other words, the code to support ASCII and UTF-8 are largely the same. The only time you need to be careful is when you are iterating individual characters within a string, since UTF-8 encodes some characters with 2-4 char instead of just one.

Both Mac and Linux switched directly from using code pages to supporting UTF-8. As a result, using and supporting UTF-8 on those platforms is pretty seamless.

However, due to the history of Unicode on Windows and Microsoft’s admirable insistence on backwards compatibility, Windows programming is more complex. Windows eventually switched from UCS-2 encoding to UTF-16 encoding (another variable length encoding that uses 2 or 4 bytes instead of 1, 2, 3, or 4 bytes). And then Windows also started to embrace UTF-8, but only if you specifically configure it to do so and follow specific rules.

These days, it’s recommended pretty much globally to use UTF-8 when writing a program or encoding text files, as this should allow you to support any human language or even a mix of human languages with a single encoding.

One More Windows Misstep

I don’t blame Microsoft for making some mistakes, since they were often trying to pioneer the best way to handle text encoding in an operating system at a time when they’re Windows was insanely popular and they needed solutions fast.

But when they did start supporting UTF-8 encoding, they made one more mistake: they chose to add a BOM (byte order marking) to the beginning of UTF-8 text files. This made it easier for their programs to detect whether a text file was using a legacy encoding or UTF-8. But it also complicated things because suddenly you could have files or text data that were UTF-8 encoded OR UTF-8 (with BOM) encoded! This could be annoying and had echoes of the code page chaos that occurred with Extended ASCII.

Fortunately, the BOM was retired in 2019. but UTF-8 with BOM encoding still exists in some Microsoft products (such as Visual Studio), so it’s important to be aware of it and how to deal with it.

Adding UTF-8 Support to Your Programs

Enough history, let’s get to some practical implementation! What do you need to do to support UTF-8 in your applications?

On Mac and Linux, the answer is fortunately…nothing! These platforms should already store your source code in UTF-8, they should already use UTF-8 encoding at runtime, and their OS APIs do not differ when using ASCII vs. UTF-8. You do still need to keep some things in mind when coding (we’ll discuss that later on), but as far as configuring your project goes, no steps need to be taken.

On Windows though…things are complicated! This entire section is devoted to Windows project configuration.

Force IDEs to Encode Source Files with UTF-8

Mac and Linux do this by default, but surprisingly Visual Studio does not. For legacy reasons, Visual Studio does a very strange thing:

  1. Visual Studio only assumes that your source code files are UTF-8 encoded if the file includes the BOM (byte order mark). If you are working cross-platform, your source code files should not include this. So Visual Studio thinks your files are not UTF-8 encoded.

  2. Visual Studio falls back on assuming that the file should be encoded in your OS’s current code page, which is likely Code Page 1252 if you are an American developer (but it could be some other code page if you are elsewhere in the world).

The result of this is that if you try to include Unicode characters in your source code files, Visual Studio will give you an error message about how the file can’t be saved with the current code page and would you like to instead use UTF-8 with BOM encoding? And no…you don’t really want that for cross-platform development!

Fortunately, you can resolve this whole debacle quite easily using EditorConfig. Simply ensure you have a file called .editorconfig at the root of your project repo and make sure it has this in it:

[*]
charset = utf-8

Visual Studio will detect this and just encode everything in UTF-8 (without the BOM), overriding the built-in behavior.

(By the way, this is also useful on Mac and Linux in case someone has configured their IDE to use a different encoding. Most modern IDEs support EditorConfig, and this file will override any IDE setting, ensuring that all project developers use the same encoding for all source files.)

Force Visual C++ Compiler to Use UTF-8

It isn’t enough for your source code files to be encoded with UTF-8 on the disk - you also must tell the Microsoft Visual C++ compiler to treat your source files as UTF-8 when compiling AND to store strings in the compiled binary as UTF-8.

To do this, you need to add the /utf-8 compiler flag. In Visual Studio, you can add this under Properties > C/C++ > Command Line and put the flag in the “Additional Options” area. You can also add it via CMake:

if(MSVC)
    target_compile_options(PROJECT_NAME PRIVATE
        /utf-8
    )
endif()

Force Windows Console Output to Use UTF-8

Even with all we’ve done so far, you may be surprised to find that if you try to log Unicode characters to the Windows console, it won’t work! Though we’ve configured the program to compile and run with UTF-8 encoded strings, we have not configured the console output to use UTF-8!

To do this, you must add the following function call somewhere in your program’s startup code:

// Include Windows.h for this.
SetConsoleOutputCP(CP_UTF8);

This tells the console to use the UTF-8 code page.

If you plan to accept input from the console, you also want to specify this:

SetConsoleCP(CP_UTF8);

Tell Windows to Interpret Narrow API Strings as UTF-8

Usually, when you call a “narrow” Windows API function, the passed string will be interpreted using Extended ASCII based on the currently active code page. However, as of Windows 10, you can tell the OS to set the active code page to UTF-8 when running your application. This allows you to pass UTF-8 encoded strings to “narrow” API functions and have them work correctly. In other words, you can avoid using the “wide” API functions and converting to/from wchar/std::wstring entirely.

To do this, you’ll need to create a file with the .manifest extension and with at least the following contents:

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity type="win32" version="1.0.0.0" name="YourAppName"/>

  <application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings>
      <activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
    </windowsSettings>
  </application>
</assembly>

Add this to your Visual Studio solution and ensure it’s hooked up under Properties > Manifest Tool > Inputs and Outputs > Additional Manifest Files. If you’re using CMake, you can just add the .manifest file as a target source file and CMake will automatically add the file to “Additional Manifest Files” for you.

You can verify that this is working by calling GetACP() at runtime and ensuring that it returns 65001 (the UTF-8 code page ID):

#include <iostream>
#include <Windows.h>

int main(int argc, char* argv[])
{
    std::cout << "Active Code Page: " << GetACP() << std::endl;
}

Note that this solution ONLY WORKS in Windows 10 Version 1903 (May 2019 Update) or later. If your application targets earlier Windows versions and you still want to support Unicode, you must instead use the “wide” API functions.

But even if you target earlier Windows versions, I’d recommend doing this to ensure the best possible support on Windows 10 and later.

Default to Windows Wide API

Since Windows has two versions of many functions (narrow/ASCII and wide/Unicode), the Windows header defines many macros that will remap to the appropriate variant depending on your compiler settings. For example, you can call CreateFile and this will be a macro that either maps to CreateFileA or CreateFileW depending on whether you are preferring the narrow or wide APIs respectively.

By default, the macro will map to the narrow version (like CreateFileA). To map to the wide version instead, you must add the compiler defines UNICODE and _UNICODE. The former tells the Windows API to use the wide variants while the latter tells the C runtime to use wide variants. You can also enable both defines by manually selecting the “Use Unicode character set” option in Visual Studio project settings.

Many developers mistakenly believe that you must add these compiler defines for a Windows program to support Unicode. In truth, the only thing these compiler defines do is remap certain macros to the narrow or wide function variants. If you code your program to explicitly call the appropriate version, then these macros become irrelevant.

But that being said, it is always recommended to add these compiler defines because it forces you to be explicit in your code:

  • If you intend to use the narrow API, you must call it explicitly.
  • If you use the remap macros, they map to the wide functions, and this ensures that you are properly converting to wchar_t or std::wstring first.

Coding with UTF-8

Holy cow, that was a ton of upfront configuration just to get Windows to use UTF-8 properly! Mac and Linux should just work out of the box. Now when writing code for your application, you can mostly just use char and std::string and not really worry about Unicode characters at all.

But that being said, there are some things to keep in mind, and some scenarios to be mindful of, when coding with UTF-8.

Use UTF-8 Everywhere

Throughout all of your application code, consider all strings to be UTF-8 encoded. The only time anything should not be UTF-8 encoded is at the boundaries of your application:

  • When reading data into your program from a file, an input channel, or from a Windows wide API, the data may not be UTF-8 encoded. If it isn’t, convert it to UTF-8 right away.
  • When writing data to a file, an output channel, or to a Windows wide API, the data may need to be some other encoding. Convert away from UTF-8 only right before the data leaves your program code.
  • It kind of goes without saying, but for every data store that you control (files on disk, databases, etc) encode strings in UTF-8.

By following these rules, your application logic can be fairly simple and just assume UTF-8 everywhere.

Use Explicit Windows API Calls

Whenever you call a Windows API function, explicitly use either the narrow or wide version depending on your needs. Do not use the remap macros.

  1. Use the narrow API if you only need to support the active code page. If you used the manifest solution to set the active code page to UTF-8, then you can still use the narrow API with Unicode, which is the best of all worlds. But keep in mind that this only works with Windows 10 or greater.

  2. Use the wide API if you want Unicode support and either a) want to support older versions of Windows or b) didn’t use the manifest solution to set the active code page to UTF-8.

If you are passing a string to the wide API, you’ll need to convert your strings from UTF-8 to UTF-16. Likewise, if you are retrieving string data from the wide API, you will need to convert from UTF-16 back to UTF-8.

Viewing UTF-8 in Visual Studio’s Debugger

Even with all we’ve done, the Visual Studio debugger will not show UTF-8 text by default. If you place a breakpoint and hover over an std::string or char* variable, the results will look garbled.

To fix this, you must manually watch the variable (usually right click and select “Add Watch”). Then, you must double click the watch and add “,s8” after the variable name. For example, if your variable is called “myText”, modify the watch window so that the variable name is “myText,s8”.

This is annoying to do over and over, but it seems to be the only solution. This allows you to view text in the debugger interpreted with UTF-8 encoding. Without the “s8” suffix, the active code page is used instead.

Add Support for Converting Between Text Encodings

For a modern program, there’s a good chance that, besides the Windows APIs, you won’t ever have to worry about data in your program being any encoding other than UTF-8. This is great and awesome if you can pull it off.

However, there may be scenarios where you unavoidably need to read in data that is not UTF-8. As an example, I was recently working with the Russian localization of Gabriel Knight 3, and I found that text in the data files was encoded using the Windows-1251 code page.

There is no built-in or automated way to convert data from other encodings to UTF-8 (at least not in C/C++). If you know some string data being read into your program is a different encoding, you must call the appropriate function to convert it to UTF-8.

There are several solutions out there to do this. From least complex to most complex:

  1. For converting UTF-8 to UTF-16 and back, you can pretty easily write utilities using MultiByteToWideChar and WideCharToMultiByte on Windows. An implementation of this that I wrote can be found here. The utf8-cpp library is also capable of doing this.

  2. For instances where you need to convert an Extended ASCII code page to UTF-8, you can manually write conversion code using a lookup table. For any char value 128-255, map the value to the appropriate UTF-8 byte sequence. An implementation of this that I wrote for Windows-1251 can be found here.

  3. If you don’t want to write lookup tables or if you need to convert more complex encodings for Korean, Japanese, or Mandarin, you might consider either using iconv or another library that focuses on the specific encodings you are interested in.

  4. The most comprehensive (but also complex and heavyweight) option is ICU. For some use cases, this can be overkill. But if you need to convert between a lot of encodings, it can make sense to use this.

Be Wary of String Length Functions

Functions that report string length, such as std::string::length and strlen, assume that one byte in a string equals one character. Since UTF-8 encoded characters can be 1-4 bytes in length, that assumption is wrong and these functions won’t necessarily give correct results.

Just think of these functions as providing the byte length of a string, not the character length. To get the character length, you must do one of two things:

  1. Write a function that iterates the bytes of the string, detects 2/3/4 byte UTF-8 encoded characters, and counts them as a single character.

  2. Use a third party library (such as utf8-cpp) that can count characters for you. Be sure to define the /Zc:__cplusplus compiler flag on Windows so that you can use all the features of this library!

If you go with option 2, you can easily query the number of characters or iterate the characters in a UTF-8 string:

// Get number of characters in a UTF-8 string.
size_t characters = utf8::distance(utf8Str.begin(), utf8Str.end());

// Iterate each character in a UTF-8 string.
auto it = utf8Str.begin();
while (it != utf8Str.end())
{
    // Advances 'it' and returns the 32-bit Unicode code point.
    uint32_t codePoint = utf8::next(it, utf8Str.end());
}

Think in Terms of Code Points Instead of Chars/Bytes

A lot of times, you just pass around strings and don’t worry about the individual characters of the string. But if you do ever need to write logic related to the characters in a string, think in terms of the code points as opposed to the chars that make up the string.

For example, for font rendering in Gabriel Knight 3, the basic algorithm is to map a char to a particular glyph inside of a bitmap. When that char is encountered in a string, we lookup the appropriate glyph and render it. This works well for ASCII or Extended ASCII, but fails when using UTF-8.

For UTF-8, we must instead do the mapping based on code points. And then when iterating a string to get each character, we must call something like utf8::next instead of indexing directly into the string like myStr[i].

(As an aside, there are also Unicode characters that are made up of multiple code points - these are called grapheme clusters. For example, an emoji and a skin tone are two code points that make up a single character. Handling grapheme clusters is more complex, and you only need to worry about it if your text strings may contain such complex multi-code-point characters.)

Write UTF-8 Helper Functions

It’s very easy to accidentally use an std::string function that breaks UTF-8 encoding - for example, substr and insert and erase can all corrupt your UTF-8 string if you aren’t careful!

To make this less complex and error-prone, I recommend writing some wrapper functions that safely perform these operations on UTF-8 encoded strings. For example, here’s a helper that safely implements UTF-8 substring logic with utf-cpp:

size_t Utf8::CharacterIndexToByteIndex(const std::string& string, size_t characterIndex)
{
    // Use utf8 library to advance x characters into the unicode string.
    auto it = string.begin();
    utf8::advance(it, characterIndex, string.end());

    // Use std::distance to get the byte offset at that character index.
    return std::distance(string.begin(), it);
}

size_t Utf8::CharacterCountToByteCount(const std::string& string, size_t characterIndex, size_t characterCount)
{
    return CharacterIndexToByteIndex(string, characterIndex + characterCount) - CharacterIndexToByteIndex(string, characterIndex);
}

std::string Utf8::Substring(const std::string& string, size_t startCharacterIndex, size_t characterCount)
{
    if(characterCount == std::string::npos)
    {
        return string.substr(CharacterIndexToByteIndex(string, startCharacterIndex));
    }
    else
    {
        return string.substr(CharacterIndexToByteIndex(string, startCharacterIndex), CharacterCountToByteCount(string, startCharacterIndex, characterCount));
    }
}

You can find a few more helpers that I wrote here.

“It Just Works”

A few years ago, I remember a colleague saying that UTF-8 was great because you can keep using char* and std::string and it just works.

Well, I’d argue that things are actually a bit more complex than that! But I hope the guidance in this post makes it easier for you to add UTF-8 support in your applications.

G-Engine  C++ 
comments powered by Disqus