Unicode equivalence is the specification by the Unicode character encoding standard that some sequences of code points represent essentially the same character. The feature was introduced in the standard to allow compatibility with pre-existing standard character sets, which often included similar or identical characters. Unicode provides two such notions, canonical equivalence and compatibility. Code point sequences that are defined as canonically equivalent are assumed to have the same appearance and meaning when printed or displayed. For example, the code point U+006E n LATIN SMALL LETTER N followed by U+0303 ◌̃ COMBINING TILDE is defined by Unicode to be canonically equivalent to the single code point U+00F1 ñ LATIN SMALL LETTER N WITH TILDE. Therefore, those sequences should be displayed in the same manner, should be treated in the same way by applications such as alphabetizing names or searching, and may be substituted for each other. Similarly, each Hangul syllable block that is encoded as a single character may be equivalently encoded as a combination of a leading conjoining jamo; a vowel conjoining jamo; and, if appropriate, a trailing conjoining jamo. Sequences that are defined as compatible are assumed to have possibly distinct appearances but the same meaning in some contexts. Thus, for example, U+FB00 ff LATIN SMALL LIGATURE FF, a typographic ligature, is defined to be compatible with, but not canonically equivalent to, the sequence U+0066 U+0066 (two Latin "f" letters). Compatible sequences may be treated the same way in some applications (such as sorting and indexing) but not in others, and they may be substituted for each other in some situations, but not in others. Sequences that are canonically equivalent are also compatible, but the opposite is not necessarily true. The standard also defines a text normalization procedure, called Unicode normalization, which replaces equivalent sequences of characters so that any two texts that are equivalent will be reduced to the same sequence of code points, called the normalization form or normal form of the original text. For both of the equivalence notions, Unicode defines two normal forms, one fully composed (where multiple code points are replaced by single points whenever possible) and one fully decomposed (where single points are split into multiple ones).
Sources of equivalence
Character duplication
For compatibility or other reasons, Unicode sometimes assigns two different code points to entities that are essentially the same character. For example, the letter "A with a ring diacritic above" is encoded as U+00C5 Å LATIN CAPITAL LETTER A WITH RING ABOVE (a letter of the alphabet in Swedish and several other languages) or as U+212B Å ANGSTROM SIGN. However, the symbol for angstrom is defined to be that Swedish letter, and most other symbols that are letters (such as ⟨V⟩ for volt) do not have a separate code point for each usage. In general, the code points of truly identical characters are defined to be canonically equivalent.
Combining and precomposed characters For consistency with some older standards, Unicode provides single code points for many characters that could be viewed as modified forms of other characters (such as U+00F1 for "ñ" or U+00C5 for "Å") or as combinations of two or more characters (such as U+FB00 for the ligature ff or U+0132 for the Dutch letter ij) For consistency with other standards and greater flexibility, Unicode also provides codes for many elements that are not used on their own but are meant instead to modify or combine with a preceding base character. Examples of those combining characters are U+0303 ◌̃ COMBINING TILDE and the Japanese diacritic dakuten (U+3099 ◌゙ COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK). In the context of Unicode, character composition is the process of replacing the code points of a base letter followed by one or more combining characters into a single precomposed character; and character decomposition is the opposite process. In general, precomposed characters are defined to be canonically equivalent to the sequence of their base letter and subsequent combining diacritic marks, whatever order they may occur.
Example
Typographical non-interaction Some scripts regularly use multiple combining marks that do not, in general, interact typographically, and do not have precomposed characters for the combinations. Pairs of such non-interacting marks can be stored in either order. Generally, the alternative sequences are canonically equivalent. The rules that define their sequencing in the canonical form also define whether they are considered to interact.
Typographic conventions Unicode provides code points for some characters or groups of characters that are modified only for aesthetic reasons (such as ligatures, the half-width katakana characters, or the full-width Latin letters for use in Japanese texts) or to add new semantics without losing the original one (such as digits in subscript or superscript positions, or the circled digits (such as "①") inherited from some Japanese fonts). Such a sequence is considered compatible with the sequence of original (individual and unmodified) characters for the benefit of applications where the appearance and added semantics are not relevant. However, the two sequences are not declared canonically equivalent since the distinction has some semantic value and affects the rendering of the text.
Encoding errors UTF-8 and UTF-16 (and also some other Unicode encodings) do not allow all possible sequences of code units. Different software will convert invalid sequences into Unicode characters using varying rules, some of which are very lossy (such as by turning all invalid sequences into the same character). That can be considered a form of normalization and can lead to the same difficulties as others.
Normalization A text processing software implementing the Unicode string search and comparison functionality must take into account the presence of equivalent code points. In the absence of that feature, users searching for a particular code point sequence would be unable to find other visually indistinguishable glyphs that have a different but canonically equivalent code point representation.
… excerpt ends here. Continue reading the full article.
