ReviseAlgo Logo

Strings

Unicode and Encoding

Working with Unicode in Python

Interview: Internationalization

Last Updated: June 12, 2026 • 7 min read

Python 3 strings are Unicode by default — every character is a Unicode code point. Understanding encoding (UTF-8, UTF-16, etc.) is essential for working with international text, file I/O, and network data.

Unicode Basics

  • Every character has a unique code point: 'A' = U+0041, 'é' = U+00E9, '' = U+1F600
  • ord(char) — get the code point of a character
  • chr(code_point) — get the character from a code point
  • len(s) counts Unicode code points, not bytes

Encoding & Decoding

  • Encoding (str → bytes): "café".encode("utf-8") → b'caf\xc3\xa9'
  • Decoding (bytes → str): b'caf\xc3\xa9'.decode("utf-8") → "café"
  • UTF-8 is the default encoding and the standard for web/JSON
  • UTF-16 and UTF-32 use fixed-width encoding

Common Encoding Issues

  • UnicodeDecodeError: Reading a file with wrong encoding (e.g., reading Latin-1 as UTF-8)
  • UnicodeEncodeError: Writing characters not supported by target encoding
  • Mojibake: Garbled text from double-encoding or wrong encoding
  • Some characters look identical but have different code points (e.g., Greek vs Latin 'A')

Unicode Normalization

Some characters have multiple representations (e.g., 'é' can be one code point or 'e' + combining acute accent). Use unicodedata.normalize('NFC', text) to normalize strings before comparison.

Use Cases

Internationalization (i18n) of applications

Reading/writing files with non-ASCII content

Working with JSON, XML, and web APIs that use Unicode

Text normalization for search and comparison

Common Mistakes

Not specifying encoding when opening files (open("f.txt", encoding="utf-8"))

Confusing character count (len) with byte count

Not normalizing Unicode strings before comparison

Assuming all characters are single-byte (UTF-8 uses 1-4 bytes per character)