Python Language Incompatibilities moving from Python 2 to Python 3 Strings: Bytes versus Unicode

30% OFF - 9th Anniversary discount on Entity Framework Extensions until December 15 with code: ZZZANNIVERSARY9


Python 2.x2.7

In Python 2 there are two variants of string: those made of bytes with type (str) and those made of text with type (unicode).

In Python 2, an object of type str is always a byte sequence, but is commonly used for both text and binary data.

A string literal is interpreted as a byte string.

s = 'Cafe'    # type(s) == str

There are two exceptions: You can define a Unicode (text) literal explicitly by prefixing the literal with u:

s = u'Café'   # type(s) == unicode
b = 'Lorem ipsum'  # type(b) == str

Alternatively, you can specify that a whole module's string literals should create Unicode (text) literals:

from __future__ import unicode_literals

s = 'Café'   # type(s) == unicode
b = 'Lorem ipsum'  # type(b) == unicode

In order to check whether your variable is a string (either Unicode or a byte string), you can use:

isinstance(s, basestring)
Python 3.x3.0

In Python 3, the str type is a Unicode text type.

s = 'Cafe'           # type(s) == str
s = 'Café'           # type(s) == str (note the accented trailing e)

Additionally, Python 3 added a bytes object, suitable for binary "blobs" or writing to encoding-independent files. To create a bytes object, you can prefix b to a string literal or call the string's encode method:

# Or, if you really need a byte string:
s = b'Cafe'          # type(s) == bytes
s = 'Café'.encode()  # type(s) == bytes

To test whether a value is a string, use:

isinstance(s, str)
Python 3.x3.3

It is also possible to prefix string literals with a u prefix to ease compatibility between Python 2 and Python 3 code bases. Since, in Python 3, all strings are Unicode by default, prepending a string literal with u has no effect:

u'Cafe' == 'Cafe'

Python 2’s raw Unicode string prefix ur is not supported, however:

>>> ur'Café'
  File "<stdin>", line 1
SyntaxError: invalid syntax

Note that you must encode a Python 3 text (str) object to convert it into a bytes representation of that text. The default encoding of this method is UTF-8.

You can use decode to ask a bytes object for what Unicode text it represents:

>>> b.decode()
Python 2.x2.6

While the bytes type exists in both Python 2 and 3, the unicode type only exists in Python 2. To use Python 3's implicit Unicode strings in Python 2, add the following to the top of your code file:

from __future__ import unicode_literals
# u'hi'
Python 3.x3.0

Another important difference is that indexing bytes in Python 3 results in an int output like so:

b"abc"[0] == 97

Whilst slicing in a size of one results in a length 1 bytes object:

b"abc"[0:1] == b"a"

In addition, Python 3 fixes some unusual behavior with unicode, i.e. reversing byte strings in Python 2. For example, the following issue is resolved:

# -*- coding: utf8 -*-
print("Hi, my name is Łukasz Langa.")
print(u"Hi, my name is Łukasz Langa."[::-1])
print("Hi, my name is Łukasz Langa."[::-1])

# Output in Python 2
# Hi, my name is Łukasz Langa.
# .agnaL zsakuŁ si eman ym ,iH
# .agnaL zsaku�� si eman ym ,iH

# Output in Python 3
# Hi, my name is Łukasz Langa.
# .agnaL zsakuŁ si eman ym ,iH
# .agnaL zsakuŁ si eman ym ,iH

Got any Python Language Question?