UTF-8 Encode / Decode

UTF-8 bytes and text, converted both ways: why mojibake happens and how to diagnose it

What layer UTF-8 actually operates at

Unicode assigns each character a code point (such as U+4E2D). UTF-8 is the rule that turns a code point into a byte sequence. That rule is variable-length:

ASCII compatibility is the reason UTF-8 won and displaced UTF-16 and GBK as the de facto standard: English text takes exactly the same bytes under UTF-8 as under ASCII, so legacy systems reading a new file do not choke on it.

Three causes of garbled text

• Encoder and decoder used different charsets. Chinese encoded as GBK but decoded as UTF-8 comes back as 锟斤拷 or a run of question marks. This is the most common category by far.
• The original bytes were lost. For example, forcing non-UTF-8 bytes to be decoded as a UTF-8 string replaces illegal sequences with U+FFFD, and the original information is gone irreversibly — no later conversion can bring it back.
• A UTF-8 BOM was read as content. A file beginning with EF BB BF can be parsed as body text by some readers, leaving an invisible character at the start of the first line. Windows Notepad adds a BOM by default when saving UTF-8, which is a frequent source of cross-platform trouble.

How this tool works

It performs a reversible, byte-level conversion and makes no guesses about charset. You can encode text into a UTF-8 byte sequence shown as hex, decimal, binary, or escape form, or decode a byte sequence back to text as UTF-8.

To tell whether your data is already corrupted, convert the text to bytes and back again. If the result matches the original, the round trip is lossless and the problem lies elsewhere — most likely a charset assumption in the transport or storage layer. If the round trip already disagrees with the original, the data went through a lossy decode at an earlier step, and you need to go back to the source for the raw bytes.

The usual troubleshooting order runs: confirm the HTTP response Content-Type carries charset=utf-8 → confirm the database connection and column charset → confirm file reads and writes explicitly name an encoding. If any one of these three falls back to a platform default, it produces the same class of failure.