Encoding orders of Brahmic scripts

Norbert Lindenberg

November 2, 2023

This article documents the encoding orders that the OpenType Universal Shaping Engine assumes for the Brahmic scripts it supports. Understanding encoding orders is necessary when rendering or otherwise interpreting text in these scripts, as well as when entering text using input methods or otherwise generating text.

Contents

• Introduction
• Scripts covered in this document
• Derivation of the tables
• Caveats
• How to report issues
• Acknowledgments
• References
• 𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇
• Ahom
• Balinese
• Batak
• Bhaiksuki
• Brahmi
• Buginese (Lontara’)
• Buhid
• Chakma
• (Eastern) Cham
• Dives Akuru
• Dogra
• Grantha
• Gunjala Gondi
• Hanunoo
• Javanese
• Kaithi
• Kawi
• Kayah Li
• Kharoshthi
• Khojki
• Khudawadi
• Lepcha
• Limbu
• Mahajani
• Makasar
• Marchen
• Masaram Gondi
• Meetei Mayek
• Modi
• Multani
• Nag Mundari
• Nandinagari
• Newa
• Phags-pa
• Rejang
• Saurashtra
• Sharada
• Siddham
• Sinhala
• Soyombo
• Sundanese
• Syloti Nagri
• Tagalog
• Tagbanwa
• Tai Le
• Tai Tham
• Tai Viet
• Takri
• Tibetan
• Tirhuta
• Zanabazar Square

Introduction

Text written in a Brahmic script consists of orthographic syllables, two-dimensional visual arrangements of glyphs that form a unit. When encoding orthographic syllables, the Unicode characters corresponding to the glyphs should be arranged in a well-defined order, so that text can be rendered correctly, and compared, searched, or otherwise processed without ambiguities or missed matches. The Unicode Standard does not define this order for most Brahmic scripts, and does so incompletely or ambiguously for others. The shaping engines of OpenType font rendering systems have therefore become the de-facto definitions of the encoding orders of orthographic syllables.

The main tables in this document show the encoding orders for Brahmic scripts as defined by the OpenType Universal Shaping Engine (USE). This information can be used in several ways:

• Implementers of smart keyboards can use it to reorder characters input by the user into the correct order.
• Implementers of spelling checkers or input prediction engines can use it to ensure that their dictionaries and thus corrected text use the correct order.
• Font engineers can see in which order characters will be presented to fonts, and use it when implementing shaping logic.
• Engineers of fonts using technologies other than OpenType, such as Apple Advanced Typography, can use it to implement validation in their fonts.
• Implementers of other text-generating systems, such as speech input systems or chatbots, can use it to ensure they generate text that will render and be interpreted correctly.

The main tables show the classes into which the USE classifies the characters of each script, the order in which their characters can occur within a cluster (the USE’s approximation of an orthographic syllable), and how often they can occur. They may also show that some sections, such as consonant conjunct forms and their associated modifiers, can be repeated, and that after the initial consonant cluster a syllable may continue with either a final virama (for which the USE uses the Hindi term “halant”) or a more substantial sequence of medial consonants, vowels, vowel modifiers, and final consonants.

Supplementary tables show characters that are used in a script but can not be part of clusters, and for some scripts decompositions that the USE applies and character sequences that it does not allow because there are equivalent individual characters.

Scripts covered in this document

This document covers scripts that are encoded in Unicode 15.1, have the Unicode property Indic_Syllabic_Category defined for at least some of their characters, and are included in the list of scripts supported in the Universal Shaping Engine.

The Unicode property Indic_Syllabic_Category is generally defined for those characters in Brahmic scripts that participate in forming orthographic syllables. The Kharoshthi script, which was a contemporary rather than a descendant of Brahmi, is included because it forms orthographic syllables similar to Brahmic scripts. Some scripts that are descendants of Brahmi may not be included because their structure has been simplified to a simple linear sequence of glyphs.

Only scripts supported by the USE are included because it is the only OpenType shaping engine that provides a well-defined cluster structure for Brahmic scripts that enables largely interoperable implementations. The documentation for shaping engines supporting other Brahmic scripts (Bengali (Bangla), Devanagari, Gujarati, Gurmukhi, Kannada, Khmer, Lao, Malayalam, Myanmar, Odia (Oriya), Tamil, Thai, Telugu) is incomplete and has been abandoned by its owner, and implementations have diverged substantially, as documented for Devanagari and Khmer. New Tai Lue is missing because OpenType documentation does not assign it to any shaping engine.

Derivation of the tables

The first step in creating these tables was to determine which characters are used with each script. The Unicode Standard includes two properties, Script and Script_Extensions, that provide much of this information. However, the use of characters in the shared scripts Common and Inherited for specific scripts is generally not documented there, and some other information provided in script proposals may have been lost as well. Long-time Unicode contributor Roozbeh Pournader has therefore started a project to collect exemplar characters for each script, and that information was included here. In addition, U+25CC DOTTED CIRCLE was included for all scripts that use combining marks (except variation selectors) or characters with USE class REPHA.

In the classification of characters, some assumptions had to be made to work around USE bug 475, which causes ambiguous classifications for several characters. For characters where the USE overrides the Indic syllabic category, categorization was based on the override. U+00A0 NO-BREAK SPACE was assigned to the BASE_OTHER class to enable its use with combining marks, as recommended in the Unicode Standard. U+200D ZERO WIDTH JOINER was assigned to the ZWJ class. The remaining characters were assigned to the BASE_IND class to avoid creating expectations that may not be met. Another assumption had to be made to work around USE bug 928 by assigning U+1171E AHOM CONSONANT SIGN MEDIAL RA to subclass CONS_MED_PRE.

USE subclasses that the USE merges into one “sigla” are merged into one representative subclass. For example, VOWEL_PRE_ABOVE, VOWEL_PRE_ABOVE_POST, VOWEL_PRE_POST are merged into VOWEL_PRE.

The seven regular expressions defining cluster structures in the USE documentation were reduced to three. The USE’s independent cluster is not relevant because no characters of the scripts covered here that fall into the classes starting such clusters allow variation selectors. Instead, a list of characters that can not be part of clusters and are therefore treated as standalone is provided. The standard cluster and virama-terminated cluster were merged into one regular cluster, as they share a long common start, followed by a long or short tail. These tails show up as alternatives in the tables provided. The number-joiner terminated cluster and the numeral cluster were merged into one special cluster, as the only difference is an optional number-joiner. The symbol cluster remains as is, but is only shown for Balinese, the only script where the cluster can be longer than one character. For all other scripts, SYM characters are lumped into the list of characters that can not be part of clusters. The hieroglyph cluster is not relevant for Brahmic scripts.

For each script, all characters used are sorted into their USE classes and subclasses, and inserted into the three regular expressions. Empty classes and subclasses are removed, empty subexpressions are removed, and empty regular expressions are removed. The remainder is formatted into tables whose format is derived from that used to describe encoding orders in the Unicode Standard, but which are extended to preserve the structural information of the underlying regular expressions.

Caveats

The precise cluster structures used by implementations of the USE can differ somewhat. While the USE’s generic Brahmic cluster model works reasonably well for most of the scripts discussed here, some scripts require adjustments, which may have been made differently (or not at all) in different implementations.

As the USE serves many different scripts, the encoding order for any particular script will allow many character sequences that don’t make sense for that particular script. That’s OK – an encoding order is no substitute for a spelling checker.

The USE occasionally places characters into classes where they linguistically or graphically don’t belong. One example are visible viramas, which it classifies as dependent vowels because they occur instead of such vowels in the encoding order. Other cases are often motivated by the need to enable a character sequence that occurs in real life but wouldn’t be allowed by a strict interpretation of the USE cluster model and the underlying Unicode character data. One such example occurs in Chakma, where above-base and below-base vowels were swapped in order to match the canonical decomposition of two vowels.

Some of the tables in this document include characters that are used in the particular script, but belong to a different one, and classify them as the USE would do. However, before text reaches a shaping engine, OpenType systems will break it into script runs. How they do that is one of the undocumented mysteries of OpenType, but the result may be that such adopted characters can't form clusters together with characters from the script in whose table they appear.

The characters U+200D ZERO WIDTH JOINER and U+200C ZERO WIDTH NON-JOINER are used in numerous scripts, sometimes for specific purposes described in the Unicode Standard, sometimes for their generic purposes of requesting or breaking up ligatures (including conjunct forms). The USE allows them anywhere in a cluster, so they’re not documented in the cluster structure. See the USE sections Zero-width joiner and Zero-width non-joiner for details.

How to report issues

You might find some problem in the encoding orders documented in this article. As it combines information from multiple sources, issues should be reported where they can be fixed:

• If your script uses a character that is not encoded in Unicode yet, you can propose it for encoding.
• If your script uses a character that is encoded in Unicode, but not shown in the encoding order table for the script, you can propose to the UTC that it be added through the Script_Extensions property. A good example for such a proposal is the one for adding the Ardhavisarga character to the Oriya script (L2/18-330).
• If your script uses a character sequence which isn’t allowed by the encoding order, scroll to the bottom of the USE documentation to submit feedback.
• If you find issues in the text of this article, or can’t determine which of the three cases above applies, contact the author through the About page.

Acknowledgments

I’d like to thank: Andrew Glass for specifying the Universal Shaping Engine, without which this article would not have been possible. Roozbeh Pournader for collecting the script exemplars data, and for quickly addressing the issues I found. Many contributors for writing script proposals, and the few who turn these proposals into the Unicode Standard, including the character properties underlying the USE. Many type designers and font engineers for creating, and Google for sponsoring, the Noto font family, which makes it possible to show actual characters in this article. Marc Durdin, Muthu Nedumaran, and Simon Cozens for providing feedback on this article.

References

Norbert Lindenberg: Issues in Khmer syllable validation. Lindenberg Software LLC, 2019.

Norbert Lindenberg: Issues in Devanagari cluster validation. Lindenberg Software LLC, 2020.

Norbert Lindenberg: Implementing Javanese. The Unicode Consortium, 2022.

Microsoft Corporation: Creating and supporting OpenType fonts for the Universal Shaping Engine. Microsoft Corporation, dated 2022-10-01.

Microsoft Corporation: Microsoft Font-tools. GitHub, as of 2022-09-21. Includes data tables for the Universal Shaping Engine.

Roozbeh Pournader: unicode-data. GitHub, as of 2023-10-24. Includes script exemplar data.

The Unicode Consortium: The Unicode Standard, Version 15.1. The Unicode Consortium, 2023.

𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇𑽇𑽎𑽇

Ahom

Known bugs:

• Missing CONS_MED subclasses for some Indic_Positional_Category values (affects U+1171E)

Balinese

For the Balinese script, the USE supports two cluster structures: A regular one for the normal orthographic syllables used for writing normal languages, and a special one for the combinations of musical symbols and combining marks used for writing musical scores. Several notations using these symbols are described in Unicode Technical Note 51 Musical symbols and Sasak characters in the Balinese script.

Known bugs:

• Fully decomposed split vowels may occupy more than one position (affects U+1B3C)
• Problems in the “Split vowel handling” section (affects U+1B3C)

Batak

The Batak script has a feature that the USE does not support well: In a character sequence consonant-vowel-consonant-virama, the vowel and second consonant are displayed in reverse order. For example, the syllable stored as ᯖ ta, ◌ᯪ i, ᯇ pa, ◌᯲ virama has to be displayed as ᯖᯪᯇ᯲ tip. For the USE, however, this sequence consists of two clusters. For a way to implement this reordering in fonts, see Constructing fonts for the Batak script. Implementers of keyboards and other text producers should ensure that a Batak cluster in storage never contains both a vowel and a virama. If a user types both after the same consonant, the vowel has to be reordered before the consonant in storage.

Bhaiksuki

Brahmi

For the Brahmi script, the USE supports two cluster structures: a regular one for the normal orthographic syllables used for writing normal languages, and a special one for an old additive-multiplicative notation for numbers. This notation is described in section 14.1 “Brahmi” of The Unicode Standard.

Buginese (Lontara’)

Buhid

Chakma

(Eastern) Cham

Dives Akuru

Known bugs:

• Double REPHA in Dives Akuru (affects U+1193F, U+11941)

Dogra

Grantha

Gunjala Gondi

Hanunoo

Javanese

Kaithi

Kawi

Kayah Li

Kharoshthi

Khojki

Khudawadi

Lepcha

Known bugs:

• Unused new Indic_Syllabic_Category values (affects U+1C36)

Limbu

Mahajani

Makasar

Marchen

Masaram Gondi

Meetei Mayek

Modi

Multani

Nag Mundari

Nandinagari

Newa

Phags-pa

Rejang

Saurashtra

Sharada

Siddham

Known bugs:

• Fully decomposed split vowels may occupy more than one position (affects U+115B9)
• Problems in the “Split vowel handling” section (affects U+115B9)

Sinhala

Known bugs:

• Decomposed Sinhala Vowels (affects U+0DDA, U+0DDD, U+0DCA)

Soyombo

Sundanese

Syloti Nagri

Tagalog

Tagbanwa

Tai Le

Known bugs:

• USE cluster definition incompatible with non-Brahmic combining marks (affects U+0300..U+0301, U+0307..U+0308, U+030C)

Tai Tham

Tai Tham is not fully supported by the USE. Orthographic syllables in Tai Tham can be more complicated than those of most other scripts, and there’s no agreement yet on how to encode them in Unicode. See the Topical Document List: Tai Tham.

Known bugs:

• Reordering of multiple pre-base vowels not specified (affects U+1A6E..U+1A72)
• Consonant_Initial_Postfixed should be mapped to CONS_MED (affects U+1A5A)

Tai Viet

Takri

Tibetan

Known bugs:

• Several characters are in two USE classes each (affects U+0F01, U+0F04..U+0F06)

Tirhuta

Zanabazar Square

Known bugs:

• Double MBlw in Zanabazar Square (affects U+11A3B..U+11A3E)
• Several characters are in two USE classes each (affects U+11A3F, U+11A45)

𑽆𑽌𑽆𑽏