enc/learn/slides.md

Characters over the wire

Standards on sending, and parsing characters over the web.

Basic idea

• Assign a number to each character using a Character set.
• Encode the number to bytes using an encoding scheme.
• Transfer bytes over the internet

The terms character set is used interchangably with character encoding and code pages.

---

Common character sets

ASCII

• It assigns character to number mapping from 0-127 and covers english characters and some control codes (eg: new lines, tabs)
• Not everything from 0-127 is mapped.

Latin

• Also called ISO-8859-1 character set.
• This is an extension of ASCII and covers the Latin alphabet - À,ä...
• Number mappings upto 255.

Windows 1252

• Super set of Latin character set.
• Introduced by Microsoft.

Unicode

• Capable of defining a mapping for 1.1 million characters.
• Currently 150000 are defined.
• Each mapping is also called a unicode code point.
• Most languages - ஐ, ह
• Emojis 😮, 🤔
• Math ∫x.dx

---

Common encoding schemes

• An encoding scheme will encode the number to one or more bytes.

Single byte encoding schemes

• Uses up only one byte.
• Suitable for ASCII, Latin and Windows 1252 character sets.
• ASCII would only take up 7 bits, while Latin and Windows 1252 would take up 8 bits.
• Because Windows 1252 is a superset of Latin, which is also a super set of ASCII, for a very long time in the past, the most used encoding scheme was Windows 1252.
• Today, it only accounts for 1.4% of the internet traffic.

┌───────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec │ U+hex │ character │ byte │ hex │ dec │ bin      │
├───────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 97    │ 61    │ a         │ 0    │ 61  │ 97  │ 01100001 │
│ 98    │ 62    │ b         │ 1    │ 62  │ 98  │ 01100010 │
│ 99    │ 63    │ c         │ 2    │ 63  │ 99  │ 01100011 │
│ 100   │ 64    │ d         │ 3    │ 64  │ 100 │ 01100100 │
└───────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

---

Common encoding schemes

• *An encoding scheme will encode the number to one or more bytes.

Multi byte encoding schemes

UTF - 8

• Variable byte encoding scheme.
• 1 - 4 bytes to represent a unicode code point.
• Backward compatible with ASCII.
• Can represent a maximum number of 2097152 code points.
• 99% of the internet uses this encoding scheme.

Byte 1	Byte 2	Byte 3	Byte 4	Available bits
0xxxxxxx	-	-	-	7
110xxxxx	10xxxxxx	-	-	11
1110xxxx	10xxxxxx	10xxxxxx	-	16
11110xxx	10xxxxxx	10xxxxxx	10xxxxxx	21

┌────────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec  │ U+hex │ character │ byte │ hex │ dec │ bin      │
├────────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 97     │ 61    │ a         │ 0    │ 61  │ 97  │ 01100001 │
│ 98     │ 62    │ b         │ 1    │ 62  │ 98  │ 01100010 │
│ 2960   │ b90   │ ஐ         │ 2    │ e0  │ 224 │ 11100000 │
│        │       │           │ 3    │ ae  │ 174 │ 10101110 │
│        │       │           │ 4    │ 90  │ 144 │ 10010000 │
│ 2361   │ 939   │ ह         │ 5    │ e0  │ 224 │ 11100000 │
│        │       │           │ 6    │ a4  │ 164 │ 10100100 │
│        │       │           │ 7    │ b9  │ 185 │ 10111001 │
│ 129300 │ 1f914 │ 🤔        │ 8    │ f0  │ 240 │ 11110000 │
│        │       │           │ 9    │ 9f  │ 159 │ 10011111 │
│        │       │           │ 10   │ a4  │ 164 │ 10100100 │
│        │       │           │ 11   │ 94  │ 148 │ 10010100 │
└────────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

---

Common encoding schemes

• An encoding scheme will encode the number to one or more bytes.

Multi byte encoding schemes

UTF - 16

• Variable byte encoding scheme.
• 2 or 4 bytes to represent a unicode code point.

┌────────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec  │ U+hex │ character │ byte │ hex │ dec │ bin      │
├────────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 97     │ 61    │ a         │ 0    │ 00  │ 0   │ 00000000 │
│        │       │           │ 1    │ 61  │ 97  │ 01100001 │
│ 98     │ 62    │ b         │ 2    │ 00  │ 0   │ 00000000 │
│        │       │           │ 3    │ 62  │ 98  │ 01100010 │
│ 2960   │ b90   │ ஐ         │ 4    │ 0b  │ 11  │ 00001011 │
│        │       │           │ 5    │ 90  │ 144 │ 10010000 │
│ 2361   │ 939   │ ह         │ 6    │ 09  │ 9   │ 00001001 │
│        │       │           │ 7    │ 39  │ 57  │ 00111001 │
│ 129300 │ 1f914 │ 🤔        │ 8    │ d8  │ 216 │ 11011000 │
│        │       │           │ 9    │ 3e  │ 62  │ 00111110 │
│        │       │           │ 10   │ dd  │ 221 │ 11011101 │
│        │       │           │ 11   │ 14  │ 20  │ 00010100 │
└────────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

---

URL Encoding

• Applicable only for HTTP traffic.
• Some characters have a special meaning in the url string Eg: &, #, ?
• The url string should also be only in ASCII.
• These characters should be treated differently.

Steps to URL-encode a string

• Encode the string in one of the encoding schemes.
• If a particular character cannot appear in the url string, or is not ASCII, print the hex representation of the string, prefixed with a %.

┌───────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec │ U+hex │ character │ byte │ hex │ dec │ bin      │
├───────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 38    │ 26    │ &         │ 0    │ 26  │ 38  │ 00100110 │
│ 63    │ 3f    │ ?         │ 1    │ 3f  │ 63  │ 00111111 │
└───────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

• For example, if the url string p1&/pw? were to be url-encoded under utf-8 encoding, then it would be p1%26/pw%3f

┌───────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec │ U+hex │ character │ byte │ hex │ dec │ bin      │
├───────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 38    │ 26    │ &         │ 0    │ 00  │ 0   │ 00000000 │
│       │       │           │ 1    │ 26  │ 38  │ 00100110 │
│ 63    │ 3f    │ ?         │ 2    │ 00  │ 0   │ 00000000 │
│       │       │           │ 3    │ 3f  │ 63  │ 00111111 │
└───────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

• Under utf-16 encoding, it would be p1%00%26/pw%00%3f

---

What should be supported in applications?

• Support Unicode code points encoded as utf-8 characters.
• URL encode under utf-8.

---

What is a character?

• It is a group of unicode code points - also called a grapheme cluster.
• Eg: the character 'ப்' consists of 2 unicode code points as seen below.

┌───────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec │ U+hex │ character │ byte │ hex │ dec │ bin      │
├───────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 2986  │ baa   │ ப         │ 0    │ e0  │ 224 │ 11100000 │
│       │       │           │ 1    │ ae  │ 174 │ 10101110 │
│       │       │           │ 2    │ aa  │ 170 │ 10101010 │
│ 3021  │ bcd   │ ்         | 3    │ e0  │ 224 │ 11100000 │
│       │       │           │ 4    │ af  │ 175 │ 10101111 │
│       │       │           │ 5    │ 8d  │ 141 │ 10001101 │
└───────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

• Number of characters in a string is often different from string.Length.
• Some languages (eg: python) return the number of unicode code points.
• Some languages (eg: C#) will return the number of utf-16 bytes to encode the complete string.
• The below emoji is of length 1 in python and length 4 in c#.

┌────────┬───────┬───────────┬──────┬─────┬─────┬──────────┐
│ U+dec  │ U+hex │ character │ byte │ hex │ dec │ bin      │
├────────┼───────┼───────────┼──────┼─────┼─────┼──────────┤
│ 129300 │ 1f914 │ 🤔        │ 0    │ d8  │ 216 │ 11011000 │
│        │       │           │ 1    │ 3e  │ 62  │ 00111110 │
│        │       │           │ 2    │ dd  │ 221 │ 11011101 │
│        │       │           │ 3    │ 14  │ 20  │ 00010100 │
└────────┴───────┴───────────┴──────┴─────┴─────┴──────────┘

• Be careful about advertising character length limitations.