#1,007 – Getting Length of String that Contains Surrogate Pairs

January 9, 2014 1 Comment

You can use the string.Length property to get the length (number of characters) of a string.  This only works, however, for Unicode code points that are no larger than U+FFFF.  This set of code points is known as the Basic Multilingual Plane (BMP).

Unicode code points outside of the BMP are represented in UTF-16 using 4 byte surrogate pairs, rather than using 2 bytes.

To correctly count the number of characters in a string that may contain code points higher than U+FFFF, you can use the StringInfo class (from System.Globalization).

            // 3 Latin (ASCII) characters
            string simple = "abc";

            // 3 character string where one character
            //  is a surrogate pair
            string containsSurrogatePair = "A𠈓C";

            // Length=3 (correct)
            Console.WriteLine(string.Format("Length 1 = {0}", simple.Length));

            // Length=4 (not quite correct)
            Console.WriteLine(string.Format("Length 2 = {0}", containsSurrogatePair.Length));

            // Better, reports Length=3
            StringInfo si = new StringInfo(containsSurrogatePair);
            Console.WriteLine(string.Format("Length 3 = {0}", si.LengthInTextElements));

1007-001

Advertisement

Filed under Strings Tagged with , , , , ,

#999 – Some Examples of UTF-16 and UTF-8 Encoding

December 19, 2013 Leave a comment

Unicode maps characters into their corresponding code points, i.e. a numeric value that represents that character.  A character encoding scheme then dictates how each code point is represented as a series of bits so that it can be stored in memory or on disk.  UTF-16 and UTF-8 are the most commonly used encoding schemes for Unicode character data.

Below are some examples of how various characters would be encoded in UTF-16 and UTF-8.

  • Latin capital ‘A’, code point U+0041
    • UTF-16: 2 bytes, 00 41  (hex)
    • UTF-8: 1 byte, 41 (hex)
  • Latin lowercase ‘é’ with acute accent, code point U+00E9
    • UTF-16: 2 bytes, 00 E9 (hex)
    • UTF-8: 2 bytes, C3 A9 (hex)  [110x xxxx 10xx xxxx]
  • Mongolian letter A, U+1820
    • UTF-16: 2 bytes, 18 20 (hex)
    • UTF-8: 3 bytes, E1 A0 A0 (hex)  [1110 xxxx 10xx xxxx 10xx xxxx]
  • Ace of Spades playing card character, U+1F0A1
    • UTF-16: 4 bytes, D8 3C DC A1
    • UTF-8: 4 bytes, F0 9F 82 A1  [1111 0xxx 10xx xxxx 10xx xxxx 10xx xxxx]

Filed under Basics Tagged with , , , ,

#997 – UTF-16 Encoding, Part II

December 17, 2013 3 Comments

Unicode maps characters into their corresponding code points, i.e. a numeric value that represents that character.  A character encoding scheme then dictates how each code point is represented as a series of bits so that it can be stored in memory or on disk.

Recall that UTF-16 encoding uses either 2 or 4 bytes to represent each code point.

Code points larger than FFFF are represented using 4 bytes, known as a surrogate pair.

  • The first two bytes (lead surrogate) are in the range D800 – DBFF
  • The third and fourth bytes (trail surrogate) are in the range DC00 – DFFF

These surrogate pairs in turn represent code points in the range U+10000 – U+10FFFF.  This represents 1,048,576 additional values that can be encoded as surrogate pairs.  (Though the Unicode standard currently defines only a small subset of these values).

 

 

Filed under Basics Tagged with , , ,

#996 – UTF-16 Encoding, Part I

December 16, 2013 2 Comments

Unicode maps characters into their corresponding code points, i.e. a numeric value that represents that character.  A character encoding scheme then dictates how each code point is represented as a series of bits so that it can be stored in memory or on disk.

UTF-16 is one of the more common character encodings used to represent Unicode characters.  UTF-16 uses either 2 or 4 bytes to represent each code point.

All code points in the range of 0 to FFFF are represented directly as a 2-byte value.  This set of code points is known as the Basic Multilingual Plane (BMP).

Code points in the BMP are defined only within the following two ranges (hex values):

  • 0000 – D7FF
  • E000 – FFFF

This results in a total of 63,488 characters that can be represented.  This first set of values is known as the Basic Multilingual Plane (BMP).  In actuality, around 55,000 code points are currently defined in the BMP.

Filed under Basics Tagged with , , , , ,

#994 – Unicode Basics

December 12, 2013 7 Comments

To store alphabetic characters or other written characters on a computer system, either in memory or on disk, we need to encode each character so that we can store some numeric value that represents the character.  The numeric values are, ultimately, just bit patterns–where each bit pattern represents some character.

Unicode is a standard that specifies methods for encoding all characters from the written languages of the world.  This includes the ability to encode more than 1 million unique characters.  Unicode is the standard used for all web-based traffic (HTML and XML) and for storing character data on most modern operating systems (e.g. Windows, OS X, Unix).

The Unicode standard defines a number of different character encodings.  The most common are:

  • UTF-8 – Variable number of bytes used, from 1-4 bytes.  English characters use only 1 byte.
  • UTF-16 – Uses 2 bytes for most common characters, 4 bytes for other characters.

Filed under Basics Tagged with , , , ,

#993 – Some Examples of Characters

December 11, 2013 Leave a comment

Given that the System.Char data type (char) can store any Unicode character, encoded using UTF-16, below are some examples of valid characters.

Here is the list that we bind to, to display some characters:

            CharList = new ObservableCollection<char>
            {
                'a', 'Z', '.', '$', '\x00e1', '\x00e7', '\x0398', '\x03a9',
                '\x0429', '\x046c', '\x05d4', '\x05da', '\x0626', '\x0643',
                '\x0b92', '\x0caf', '\x0e0d', '\x1251', '\x13ca', '\x14cf',
                '\x21bb', '\x21e9', '\x222d', '\x2285', '\x2466', '\x2528',
                '\x2592', '\x265a', '\x265e', '\x2738', '\x3062', '\x3063',
                '\x3082', '\x3071', '\x3462', '\x3463', '\x3464', '\x3485',
                '\x5442', '\x54e1', '\x5494', '\xac21', '\xae25', '\xfc45',
                '\xfce8'
            };

993-001

Filed under Data Types Tagged with , , , , ,

#992 – The System.Char Data Type

December 10, 2013 Leave a comment

In C#, the char keyword is a synonym for the System.Char data type in the Base Class Library (BCL).  An instance of a char represents a single character, i.e. a single unit that is part of a written language.

Some examples of characters, all of which can be represented by a char:

  • Uppercase or lowercase alphabetic characters from the English (Latin) alphabet (a..z, A..Z)
  • Accented alphabetic characters
  • Alphabetic characters from other alphabets (e.g. Greek, Cyrillic, Hebrew, Chinese, Arabic, etc).
  • Punctuation marks  (e.g. ! , = [ {, etc).
  • Numeric digits (0..9)
  • Control or formatting characters (e.g. end-of-line, delete)
  • Mathematical and musical symbols
  • Special graphical glyphs (e.g. trademark symbol, smiley face)

A character stored in an instance of a char takes up 2 bytes (16 bits).  The values are encoded as Unicode characters, using the UTF-16 encoding.

Filed under Data Types Tagged with , , , , ,