Joho the Blog » [eim] Alphabetical order explained in a mere 27,817 words

[eim] Alphabetical order explained in a mere 27,817 words

This is one of the most amazing examples I’ve seen of the complexity of even simple organizational schemes. “Unicode Collation Algorithm (Unicode Technical Standard #10)” spells out in precise detail how to sort strings in what we might colloquially call “alphabetical order.” But it’s way, way, way more complex than that.

Unicode is an international standard for how strings of characters get represented within computing systems. For example, in the familiar ASCII encoding, the letter “A” is represented in computers by the number 65. But ASCII is too limited to encode the world’s alphabets. Unicode does the job.

As the paper says, “Collation is the general term for the process and function of determining the sorting order of strings of characters” so that, for example, users can look them up on a list. Alphabetical order is a simple form of collation.

Sorting inconsistent alphabets is, well, a problem. But let Technical Standard #10 explain the problem:

It is important to ensure that collation meets user expectations as fully as possible. For example, in the majority of Latin languages, ø sorts as an accented variant of o, meaning that most users would expect ø alongside o. However, a few languages, such as Norwegian and Danish, sort ø as a unique element after z. Sorting “Søren” after “Sylt” in a long list, as would be expected in Norwegian or Danish, will cause problems if the user expects ø as a variant of o. A user will look for “Søren” between “Sorem” and “Soret”, not see it in the selection, and assume the string is missing, confused because it was sorted in a completely different location.

Heck, some French dictionaries even sort their accents in reverse order. (See Section 1.3.)

But that’s nothing. Here’s a fairly random paragraph from further into this magnificent document (section 7.2):

In the DUCET, characters are given tertiary weights according to Table 17. The Decomposition Type is from the Unicode Character Database [UAX44]. The Case or Kana Subtype entry refers either to a case distinction or to a specific list of characters. The weights are from MIN = 2 to MAX = 1F16, excluding 7, which is not used for historical reasons.

Or from section 8.2:

Users often find asymmetric searching to be a useful option. When doing an asymmetric search, a character (or grapheme cluster) in the query that is unmarked at the secondary and/or tertiary levels will match a character in the target that is either marked or unmarked at the same levels, but a character in the query that is marked at the secondary and/or tertiary levels will only match a character in the target that is marked in the same way.

You may think I’m being snarky. I’m not at all. This document dives resolutely into the brambles and does not give up. It incidentally exposes just how complicated even the simplest of sorting tasks is when looked at in their full context, where that context is history, language, culture, and the ambiguity in which they thrive.

7 Responses to “[eim] Alphabetical order explained in a mere 27,817 words”

  1. That is a fantastic example of complexity, a phenomenon which many are unwilling to acknowledge IMO.

    I’ll expose my ignorance – why is 7 not used in the Unicode database used in the example above?

  2. This retired programmer, who has not thought about writing code in 4 years, will not think about this as he floats in the pool reading a novel.

  3. No matter if some one searches for his vital thing, so he/she wishes to
    be available that in detail, therefore that thing
    is maintained over here.

    Visit my web site – drug rehab centers in Illinois

  4. I got this website from my pal who shared with me concerning this site and at the moment this time I am browsing this
    website and reading very informative posts at this time.

    My webpage … Massachusetts non profit drug rehab centers

  5. Thanks for your marvelous posting! I truly enjoyed reading it, you will be a great
    author. I will be sure to bookmark your blog
    and will often come back later on. I want to encourage continue your great writing, have a nice morning!

    My web page; stop drinking

  6. What’s up, all the time i used to check weblog posts here in the early
    hours in the daylight, as i love to learn more and more.

    Here is my web site: SouthBurlingtondrugrehab

  7. A fascinating discussion is definitely worth comment. There’s no doubt that that you need to publish more on this subject, it might not be a taboo matter but typically people do not discuss these subjects.
    To the next! Cheers!!

    Feel free to surf to my site; Lawrencedrugrehab

Leave a Reply


Web Joho only

Comments (RSS).  RSS icon