Towards Simplified Egyptian (March 2012)

I first mentioned Simplified Egyptian on this blog back in 2010, Simplified Egyptian: A brief Introduction, although the idea goes back several years before then.

The basis of Simplified Egyptian is to embed standard rules inside a font or functional equivalent (typically OpenType font) to instruct font rendering to combine hieroglyphs into predefined groups (much like the ligature ‘e’ with ‘acute’ normally combines into é e-acute).

So, why is it taking so long to implement? The first prerequisite to making Simplified Egyptian available in useful form was adding Egyptian Hieroglyphs to the Unicode standard (initiated 2006, finally released with Unicode 5.2 in October 2009). However a standard is necessary but not sufficient, popular Operating Systems (e.g. Windows, OS X, Linux, iOS, and Android) need to adequately support the standard. It is also essential that tools such as internet browsers, word processors and so on support standards compliant behaviour. Progress has been slow on all these fronts, disappointingly so.

Two positive developments during the last few months.

1. OpenType now (almost) formally recognizes the ‘egyp’ tag for fonts containing Egyptian Hieroglyphs (see http://www.microsoft.com/typography/otspec/scripttags.htm updated October 4, 2011 to bring the standard up to date, matching Unicode developments in recent years).
2. Firefox now recognizes ligatures in Egyptian fonts (not sure exactly when this happened but my test font was first seen to work correctly in Firefox 10 on Windows). Chrome, Internet Explorer, and Safari haven’t seen any progress in the 18 months since I published Egyptian Hieroglyphs on the Web (October 2010).

One minor negative two weeks ago. The Windows 8 consumer preview really ought to have worked since Unicode 6.x support is part of the design. However my Egyptian tests failed for Internet Explorer and several other programs and software interfaces. Hopefully those bugs will be shaken out before release and similar progress will be seen in OS X, iOS and Android before the end of 2012.

Looking at the trend it seems likely that by this time next year there will be enough pieces of the puzzle coming together to enable some useful Simplified Egyptian use scenarios on a variety of fully up to date systems with up to date software. Realistically, however, for most people 'up to date' is unlikely. Android phones are often stuck with obsolete versions of Android. Windows computers last for ages nowadays yet the cost of upgrading a version of Windows and Microsoft Office is normally prohibitively high for most people and organizations. As a rough estimate, it is going to take 3+ years before 25% of PCs (including tablets, excluding phones) are Simplified Egyptian capable.

Moving forward. Despite the apparently slow opportunities for uptake it seems worth starting the ball rolling now. Last year I created a list of about 1200 candidate rules for an implementation of Simplified Egyptian to test against the dictionary database and some sample documents. Further work is needed but I hope to provide a test implementation in InScribeX Web along with a Simplified Egyptian font this year, initially using a draft specification for the rules. IXW can be used to make sample web pages (using a downloadable version of the font) which we now know will be readable in Firefox at least. This work to be followed by ample time for comment and feedback on the specification from all, including experts in the subject. Timescales depend on when I can find some spare time (unless anyone knows of any funding!) but if all goes well, by around this time next year there will be a useable Simplified Egyptian system with at least some PC systems and a few standard apps that can use it. Then, as people upgrade systems and developers upgrade apps SE can reach out to an increasing audience.

Unfortunately a slow process with pitfalls e.g. web pages that only look correct on modern browsers unless we see a change in the habits of Apple, Google, Microsoft and other PC and device manufacturers. Most people working with hieroglyphs in software will continue to mainly use MdC-based solutions for some years to come on desktop/laptop systems. A topic I'll return to shortly.

InScribeX Web 3.5 available

The InScribeX Web software is now updated to preview version 3.5, replacing the previous version 3.4. The software now integrates about 30,500 references in the Egyptian dictionary database, and the accuracy level of the database has been substantially improved over 3.4. Apart from some minor performance oriented changes, there is no change to software functionality.

3.5 is the last of the 3.x series. I hope to continue the periodic update process every two to three months during 2012 as was done last year. I'm currently shooting for a 4.0 version in the Spring.

4.0 contains some usability changes and continues to support Windows and OSX platforms via Silverlight. It drops any attempt to support Linux desktop. My main objectives moving IXW forward this year are to 1. incorporate tools to enable use of Simplified Egyptian and 2. Make it easier to use Egyptian with Unicode in an up to date HTML5 Web browser context.

A note on Linux. When I started writing InScribeX Web the situation with Linux on desktop and laptop PCs looked fairly promising. Moonlight (the Linux equivalent of the Silverlight software used to deliver InScribeX in a web browser) was under active development by Novell. Since then, Linux has been very successful as the foundation of the proprietory Google Android operating system for mobile devices but interest in Linux desktop as an alternative to Microsoft Windows and Apple OSX has declined, not grown. The demise of Novell as an independent company caused Moonlight development to stop in an incomplete state meaning it is no longer realistic to take the current implementation of InScribeX Web onto the Linux desktop. This fact is only one consequence of the fact that the whole cross-platform perspective has changed in recent years thanks to low power microelectronics enabling interesting mobile devices. A topic I'd like to return to in the near future.

InScribeX Web 3.4 available

The InScribeX Web software is now updated to preview version 3.4, replacing the previous version 3.3. This uses the latest version of my word list/database for the dictionary feature. 3.4 contains over 30,000 references, representing an increase of about 2000 new entries since 3.3. There are also about 1500 corrections and clarifications to references present in the previous version. In short the database is getting close to my objectives for the first draft.

References in the dictionaries are as follows:

• AEM refers to Ancient Egyptian Medicine by J F Nunn (1996).


• DME refers to Concise Dictionary of Middle Egyptian by R O Faulkner (1961).


• EG refers to Egyptian Grammar by A Gardiner (Third edition, 1957).


• GHAD refers to Großes Handwörterbuch Ägyptisch-Deutsch by Rainer Hannig (2006).


• ME refers to Middle Egyptian by J P Allen (2000).


• Wb refers to Wörterbuch der Aegyptischen Sprach by A Erman and H Grapow (1926, 1971).

The largest single sub-collection of references is that from Wörterbuch which accounts for about 46% of the total number of references. Here my selection has been strongly guided by the ‘Beinlich word list’ although the German-English translations and transliterations into MdC machine coded hieroglyphs are my own.

The second largest sub-collection is from the Concise Dictionary, 38% of the total. The organisation of this Faulkner material largely follows the ‘Vygus word list’ though I’ve made a fairly large number of changes and MdC transliterations were done from scratch.

The smallest sub-collection currently is that from GHAD. This is only being used to add references relating to use of some of the rarer hieroglyphs.

I’d like to repeat and emphasise that InScribeX Web dictionaries are not intended to substitute for use of the various publications referenced. The user will want to refer directly to the Faulkner, Gardiner and other books to understand the context in which my dictionary entries are given. For instance Wörterbuch gives many alternative ‘spellings’ beyond those included in IXW at present. Egyptian Grammar has many instances of words not referenced in the current list, along with far more about the language and words beyond that of a simple list. Faulkner gives many references to the sources of his material.

To complete the first draft of the word list/database, I still have a parcel of work to tidy up references using rarer hieroglyphs so as to have a solid footing from which to tackle some practical issues of what to do with signs that are not in the Basic Egyptian Hieroglyphs set standardized in Unicode 5.2 (2009).

There is also a batch of references that needs re-checking and/or revised German-English translations.

I am therefore expecting several hundred more references and a bunch of corrections to the current set before drawing a line under this phase.

My current plan is still to incorporate these changes in a version 3.5 this winter to complete the IXW preview 3 developments. It also remains my intention to defer software changes (beyond anything of a minor technical nature) until this work is completed.

InScribeX Web directions (September 2011)

Next week Microsoft will be making announcements about Windows 8 and I expect this will raise speculation on software futures in the press. So now seems like a good time to forestall any questions about any implications for InScribeX Web directions over the next year or so.

In short, Silverlight/.Net has been a successful technical choice for development of IXW to date and I plan to continue to use Silverlight through 2012 to further evolve IXW and explore more aspects of Ancient Egyptian in Unicode as a cross-platform Mac OSX/Windows (XP and later) solution.

Nevertheless there are some other desirable developments in Ancient Egyptian on computer that don’t fit into the IXW cross-platform approach, for instance deeper integration with other applications and efficient support for a variety of low power and touch screen devices. I’d like to share some thinking on these topics here in the near future.

Meanwhile IXW is currently at Preview 3.3 (the third update this year) and I’m on track for a 3.4 release next month. The ‘preview 3’ series to wrap this Winter with 3.5 which completes the first draft of the word list/dictionaries and incorporates more analysis on sign lists.

I’ve been holding back user interface changes until Preview 4 which is intended to enable continuation of the step by step approach I’ve taken this year. Whereas the 2011 theme has been building a more comprehensive dictionary, the main 2012 theme is teasing out the relationships between ‘Simplified Egyptian’, hieroglyphs in Unicode, and MdC encodings and I hope to continue the dynamic of updates every 2/3 months in the 4.x framework. More on this closer to the time.

InScribeX Web 3.3 now available

I've just released the latest version of InScribeX Web. Version 3.3 is the third update this year following version 3.1 (April) and 3.2 (June). The dictionary now contains over 28,000 references, an increase of around 50% from the 3.0 version released last year.

There are no changes to system requirements to run InScribeX Web so virtually all Windows and Intel-based Mac machines are supported.

As far as the draft (EGPZ) word list used as the basis of the dictionaries is concerned, references to Egyptian Grammar (Gardiner) and Concise Dictionary of Middle Egyptian (Faulkner) are regarded as complete for the time being with this version 3.3 (aside from any remaining corrections required).

For development of the word list, I currently have a hit-list of 580 existing references needing further research and about 2500+ further references to be added before reaching my target of what to include in the first draft. In practical terms, I'm aiming for an updated 3.4 release in the Autumn as a stepping stone to a completed first draft.

Aside from the word list/dictionaries, there are other topics such as MdC and InScribe document editing, Simplified Egyptian, Egyptian in HTML, expanding the Unicode repertoire, and mathematical modelling of Egyptian. None of which are planned for 3.4 although I'm still actively prototyping in these areas.

Note. InScribeX Web is still based on Silverlight 3 (or later) in the (possibly forlorn) hope that 'Moonlight' (the Linux equivalent to Silverlight for Windows and Mac) will catch up this year. However the whole cross-platform question has moved on since the first 2009 version of InScribeX Web most visibly with the growth of the smartphone user base, and the profile of first generation mass-market Tablet/Slate devices. I'll try to address some of the questions of how this affects Ancient Egyptian in the digital world, and InScribeX in particular, in future blog posts.

Simplified Egyptian: Numerals

This is the second of a series of notes on a systematic way of working with Ancient Egyptian Hieroglyphs in Unicode, following up from Simplified Egyptian: A brief Introduction.

What follows makes a lot more sense if you have a hieroglyphic font installed, see my post Egyptian Hieroglyphs on the Web (October 2010).

Ancient Egyptian, in common with other early mathematical systems, had no notion of negative integers or the digit Zero. The Egyptian numeral system is not positional in the modern sense. It is nevertheless straightforward to decode. Examples:

Egyptian 𓎉𓏻 is 42 (𓎉 represents 40, 𓏻 represents 2).

Egyptian 𓆿𓍣𓎉𓏻 is 4,242 (𓆿 represents 4000, 𓍣 represents 200).

Our modern decimal system uses positional notation where the numerals 0, 1 … 9 are used to represent units, tens, hundreds etc. by virtue of position. The Ancient Egyptians used different symbols based on a tally system as should be obvious from the examples. Fortunately, one similarity to modern notation is that the higher magnitude quantities were normally written first (i.e. to the left in Simplified Egyptian, which is always written left to right).

Normalized forms of numerals

The following list gives the preferred representation of hieroglyphs in Unicode for numerals in Simplified Egyptian.

1 to 9: 𓏺, 𓏻, 𓏼, 𓏽, 𓏾,𓏿, 𓐀, 𓐁, 𓐂.
10 to 90: 𓎆, 𓎇,𓎈, 𓎉, 𓎊, 𓎋, 𓎌, 𓎍, 𓎎.
100 to 900: 𓍢, 𓍣, 𓍤, 𓍦, 𓍦, 𓍧, 𓍨, 𓍩, 𓍪.
1,000 to 9,000: 𓆼, 𓆽, 𓆾, 𓆿, 𓇀, 𓇁, 𓇂, 𓇃, 𓇄.
10,000 to 90,000: 𓂭, 𓂮, 𓂯, 𓂰, 𓂱, 𓂲, 𓂳, 𓂴, 𓂵.
100,000: 𓆐
1,000,000: 𓁨

Each of these forms is available in Unicode as a unique character. For instance hieroglyph 2 is the character U+133FB 𓏻 EGYPTIAN HIEROGLYPH Z015A. Use these ‘normal’ forms for basic writing of numbers in Simplified Egyptian and avoid practices such as repeating 𓏺 for 𓏻 unless there is a compelling reason.

Note that large numbers such as 𓁨𓁨𓆐𓆐𓂮𓆽𓍣𓎇𓏻 2,222,222 were not generally encountered in ancient texts so replicating the 𓁨 and 𓆐 is rather anachronistic. An alternative multiplicative notation evolved for large numbers although uses are apparently rare so I’ll defer this topic for now.

Alternative forms of numerals

The use of normalized forms as given above makes it easy to find a number such as 𓎉𓏻 (42) in web documents, word processor and spreadsheet documents, and so forth (so long as software is sufficiently up to date of course). Unicode provides some alternative forms such as U+13403 𓐃 EGYPTIAN HIEROGLYPH Z015I (numeral 5) but these alternates should be avoided for numerals in Simplified Egyptian where at all possible (𓐃 actually has a specific use as a fraction).

Other arrangements are found in Egyptian texts, such as the following form of 35 ( from Gardiner, Egyptian Grammar, p194).

Simplified Egyptian takes the position that these kinds of numeral groups are a matter for more elaborate treatments of hieroglyphs where it is not acceptable to take license and write the number as 𓎈𓏾.

Repeating numeral 1 twice may look very much like numeral 2 in a hieroglyphic font but this practice should be avoided in Simplified Egyptian unless there is a good reason. The rationale is because most Ancient Egyptian mathematics survives in hieratic rather than hieroglyphic writing and the numerals were often simplified into a less tally-like glyph appearance. The fact that modern discussion of the hieratic often uses a hieroglyphic presentation should not detract from the original character-like behaviour. There is the important practical point that web searches and text processing work far better with normalized forms.

Rotated versions of units (e.g. 𓐄, 𓐅 …) and tens (𓎭 and 𓎮) are used in hieratic (and sometimes hieroglyphic) to number days of the month. Simplified Egyptian also adopts this convention (I hope to return to this on a topic about calendars).

Confusables

The stroke hieroglyphs U+133E4 𓏤 EGYPTIAN HIEROGLYPH Z001 (representing unity and ideogram) U+133FB 𓏺 EGYPTIAN HIEROGLYPH Z015A (numeral 1) are distinguished in Unicode. Fonts usually make the numeral stroke taller then the ideogram stroke, reflecting Ancient Egyptian conventions. Texts encoded in MdC often do not make this distinction but it is strongly recommended to do so in Simplified Egyptian so as to enable accurate text processing.

Likewise, the plurality signs U+133E5 𓏥 EGYPTIAN HIEROGLYPH Z002 and U+133E6 𓏦 EGYPTIAN HIEROGLYPH Z002A should be distinguished from numeral 3 U+133E5 𓏼 EGYPTIAN HIEROGLYPH Z015B.

In some fonts, characters such as U+0131 ı LATIN SMALL LETTER DOTLESS I and U+006C l LATIN SMALL LETTER L may look very similar to the Egyptian stroke. There are various other opportunities for confusion, for instance numeral 10 𓎆 can look very similar to U+2229 ∩ INTERSECTION and some other characters.

Other examples are the special forms for 1, 2 and 3 used in dates potentially confusable with MINUS SIGN, HYPHEN and other dashes (1), EQUALS SIGN (2), and IDENTICAL TO (3) but should never appear in a context where the meaning is unclear. The special form of 10 looks rather like SUBSET OF.

Simplified Egyptian hieroglyphs should never be written with any non-Egyptian characters just beacause they look similar.

Mathematics beyond numerals

Cardinal numbers, fractions, weights, lengths, and other measurements are matters for future topics about Simplified Egyptian.

Update. Apparently, according to Google, this note is the first writing of 𓎉𓏻 on the web, a reminder it will be interesting to see how use of hieroglyphs grows in months and years to come.

Silverlight in the News

Silverlight made it onto the BBC News on Tuesday – Coders decry Silverlight change. Take an unfortunate choice of words by a senior executive or two; add the reactive and ill-informed commentators on some web message boards; then mix in some natural concerns from developers. Bang! Tempests in teacups, the media love them.

Personally speaking, I find it reassuring to observe that the amateur tradition is alive and well in Microsoft and at least one major multinational company is not self-wrapped in a cloak of PR and spin-doctoring. That being said, the last few minutes of Doctor Who The Christmas Invasion ought to be made compulsory viewing for all senior executives.

As a developer I'm happy so long as .Net is treated as a strategic family of products. Thanks to Novell it may become so on Unix/Linux too (even if the Linux ‘community’ is slow to recognize what the third wave of Unix is really about). Hey theres another tabloid headline: C/C++ is dead!

I hope I'm not alone in being pleased to learn Silverlight 5 is not being rushed out. Especially if it means some of the niggles are resolved and the SL/WP7/WPF portability model improved. And Unicode 6.0 of course! A Mix 2011 Beta with Summer release please.

Two real news stories for developers:

An interesting talk at PDC 2010 for C# developers: ‘The Future of C# and Visual Basic’ by Anders Hejlsberg – don’t be put off like I almost was by the Visual Basic tag, it is hardly mentioned so we are not subjected yet again to the irony implicit in the keyword Dim. The main theme is simplification of Asynch programming with the new await keyword for the next .Net revision. Along with parallel constructs, this pattern brings very useful ways of exploiting multi-core processors to .Net in a clean software design. The talk is summarised here.

Developers in the .Net/WPF/Silverlight space should also check out PDC 2010: 3-Screen Coding: Sharing code between Windows Phone, Silverlight, and .NET by Shawn Burke. I alluded to the value of portable code last month in Of Characters and Strings although I didn’t highlight the .Net 4 changes that enable sharing of binary assemblies (a topic in its own right). The new tooling for Visual Studio to assist in creating Portable Assemblies, as previewed by Shawn, should be very helpful in managing the shared assembly model. It should also help focus Microsoft development on removing some of the irritating incompatibilities between Silverlight and WPF.

I just can’t wait for await.

Bob Richmond

Windows: The 25th Anniversary

The first version of Microsoft Windows was released over 25 years ago.

The conventional release date quoted is that of the Microsoft Windows 1.0 retail product for ‘IBM compatible’ PCs launched on 20th November 1985. The truth, as usual, is a little different. In the beginning, as now, Windows was distributed by computer manufacturers (OEMs) and OEM releases were shipping weeks before the Microsoft retail set, possibly as early as September. In the pre-internet era product launches worked differently to nowadays.

Since blogging on My first Home PC – recollections of the RM Nimbus PC-186 I’ve browsed some of my notes surviving from 1985. There was a lot of interest in a pre-Beta of Windows I built for the BETT show in January to accompany the PC-186 launch. Windows was fairly stable by then, I'd been running prototypes during 1984 tracking alpha versions of Microsoft code. The new Intel 80186 CPU in combination with a larger memory space made Windows more fluid on the Nimbus compared with contemporary IBM PC models and their look-alikes. In May I built a Beta release for the PC-186 which had fairly widespread distribution in the UK with the early Nimbus computers sold to the RM education market of schools and colleges. After painstaking work on optimizations of the graphics driver and improvements to the Nimbus-specific DOS application switching system (all written in 16 bit assembly language), I mastered the first release candidate in late August. I don’t recall exactly when the first release version actually shipped; perhaps the answer is hidden in my attic or the RM archives.

Trivia. Windows 1 install could fit on two 3.5”/720K disks. I also created a version to run on a single 720K drive system with Windows Write, Notepad, Paint and a few other lightweight apps. However the majority of PC-186 systems ran off network servers or hard drive. Familiar features of Windows such as GDI graphics, the message pump, EXE/DLL architecture were present right at the beginning. However the original windowing system treated applications as tiles; full overlapping windows did not appear until version 2 (1987).

Windows 1 was not a commercial success for Microsoft. RM grew one of the larger installed bases among OEMs. Most PC manufacturers (including RM beyond the PC-186) embraced the quirky and limiting IBM PC hardware/BIOS compatibility design; application vendors usually worked with these primitive interfaces rather than use a hardware-independent API like Windows. A dismal state of affairs for some years. Fortunately the Apple Mac and to a lesser extent Windows and various non-Intel based machines continued to point to the future for personal computers although in late 1985, I didn’t expect it would take over four years before the best-selling Windows 3.0 (1990) established the long term shape of the Personal Computer.

A few web searches today revealed that many aspects of the evolution of personal computers appear to be well hidden. It was interesting to be involved in the emergence of the PC for a few years so I suppose I ought to return to this period occasionally to fill in some more gaps in the online record.

Postscript. The practical side of history is learning from the past. Windows itself originated at a time when the situation with early personal computers was chaotic with few standards. An array of incompatible machines faced the software developer and the early user of PC technology. Today we have a mix of new and old generation technologies with systems like iOS, Android, WP7, Kindle, Windows, OSX, Desktop Linux, Xbox, PS3, Wii etc. etc. operating in a complex connected world; each with its own different developer stories to tell. A new chaos has emerged and, I suspect, we are once again looking to redefine the meaning of personal computing.

Egyptian Hieroglyphs on the Web (October 2010)

One year after the release of Egyptian Hieroglyphs in Unicode 5.2 there has been some progress in making hieroglyphs usable on the web although it is still early days. I hope these notes are useful.

If you can see hieroglyphs 𓄞𓀁 in this sentence, good. Otherwise. A few notes, and you can decide whether it might be better to wait until things have moved forward a little.

Information on Egyptian Hieroglyphs in Unicode

For information on Unicode 6.0 (the latest version) see www.unicode.org/versions/Unicode6.0.0/. While the full text for 6.0 is being updated, refer to the 5.2 version www.unicode.org/versions/Unicode5.2.0/ch14.pdf section 14.17. The direct link to the chart is www.unicode.org/charts/PDF/U13000.pdf where signs are shown using the InScribe font.

The Wikipedia article en.wikipedia.org/wiki/Egyptian_hieroglyphs is fairly accurate as far as it goes, and contains hieroglyphs in Unicode which can be viewed given a suitable browser and font.

InScribeX Web still contains the largest set of material viewable online including sign list, dictionaries and tools. You need a Silverlight (or Moonlight) compatible system (the vast majority of PCs, Linux, Mac or Windows, are fine.). There is no requirement you install a font. I last updated InScribeX Web in May – yes it is about due for an update but time is the enemy (and I’d like to see Moonlight 3 released first anyway).

Browsers

Of the popular web browsers, only recent versions of Firefox display individual Unicode hieroglyphs correctly. I expect the situation will change over the next few months. Meanwhile, use Firefox if you want to explore the current state of the art.

Search

Right now, only Google search indexes Unicode hieroglyphs (and the transliteration characters introduced at Unicode 5.1 in 2008). I expect at some point next year Bing and Yahoo will be brought up to date but meanwhile stick with Google.

Fonts

A satisfactory treatment of hieroglyphs on the web really needs smart fonts installed on your computer. I’m on the case (see Simplified Egyptian: A brief Introduction) but it will take some time until all the pieces of the puzzle including browser support come together (see ISO/Unicode scripts missing in OpenType and other comments here).

Neither Apple nor Microsoft provide a suitable font at the moment as parts of, or add-ons to, iOS, OSX, or Windows.

Meanwhile, in general I can’t advise about basic free fonts to use (fonts sometimes appear on the internet without permission of copyright holders and I don't want to encourage unfair use of creative work).

I will note an ‘Aegyptus’ font is downloadable at http://users.teilar.gr/~g1951d/ – the glyphs are apparently copied from Hieroglyphica 2000. I’ve not analyzed this yet.

For InScribe 2004 users I currently have an intermediate version of the InScribe font available on request (email me on Saqqara at [Saqqara.org] with ‘InScribe Unicode Font’ in the message title – I get a lot of spam junk mail. That way I can let you know about updates.).

Asking a user to install a font to read a web page is in general a non-starter; I think the medium term future for web sites is the Web Open Font Format (WOFF) once the dust settles on the new web browser versions in development. I’ll post here about the InScribe font in this context and make examples available when the time is ripe.

Of Characters and Strings (in .Net, C#, Silverlight …): Part 2

"Your manuscript I’ve read my friend
And like the half you’ve pilfered best;
Be sure the piece you yet may mend –
Take courage man and steal the rest."

Anon (c. 1805)

As mentioned in Of Characters and Strings (in .Net, C#, Silverlight …): Part 1 many of the text processing features of .Net are such that code can be re-used over Silverlight, WPF, XNA, Mono, Moonlight etc. In this note I want to draw attention to a few key points and differences over platforms. It is far from a comprehensive analysis and I’ve skipped some important topics including collation, the variety of ways of displaying text ,and input methods.

A quick aside: I failed to highlight in Part 1 how 16-bit confusion in software is not limited to .Net, similar issues arise in Java (although Java has some helpful UTF-32 methods that .Net lacks) and the situation is far worse in the world of C/C++.

I’ll restrict attention to Silverlight 4, Silverlight 3 for Windows Phone 7 OS (WPOS7.0), and .Net 4. In case the significance of the opening quote passes the reader by, it is to be hoped that Silverlight 5 and any new branches on the .Net tree feature increased compatibility, deprecating certain .Net 4 elements where appropriate.

.Net Text in general

Many Silverlight (and WPF) programmers rely mainly on controls and services written by others to deal with text so are unfamiliar with the lower level implications – in this case the main thing is to make sure you test input, processing, and display components with text that contains non-BMP characters, combining character sequences, and a variety of scripts and fonts. You may also like to check out different culture settings. Relying on typing querty123 on a UK/US keyboard and using the default font is a sure-fire recipe for bugs in all but the simplest scenarios.

.Net text is oriented around UTF-16 so you will be using Unicode all or almost all the time. The dev. tools, e.g. Visual Studio 2010 and Expression Blend 4, work pretty well with Unicode and cope with the basics of non-BMP characters. I find the main nuisance with the VS 2010 code editor is that only one font is used at a given time, I can’t currently combine the attractive Consolas font with fall-back to other fonts when a character is undefined in Consolas.

For portability over .Net implementations, the best bet at present is to write the first cut in Silverlight 3 since for this topic SL3 is not far off being a subset of .Net 4. A good design will separate the text functionality support from the platform-specific elements, e.g. it’s not usually a good idea to mix text processing implementation in with the VM in a MVVM design or in the guts of a custom control. I often factor out my text processing into an assembly containing only portable code.

Text display on the various platforms is a large topic in its own right with a number of issues for portability once you go beyond using standard controls in SL, WPF etc. I’ll not go into this here. Likewise with text input methods.

Versions of Unicode

It would be really useful if the various flavours and versions of .Net were assertive on text functionality, e.g. Supports Unicode 6.0 (the latest version) or whatever. By support, I mean the relevant Unicode data tables are used (e.g. scripts, character info) not to imply that all or even most functionality is available for all related languages, scripts and cultures. Currently, the reality is much more fuzzy and there are issues derived from factors such as ISO/Unicode scripts missing in OpenType. Methods like Char.IsLetter can be useful in constructing a 'version detector' to figure out what is actually available at runtime.

Char structure

The .Net Char structure is a UTF-16 code unit, not a Unicode character. Current MSDN documentation still needs further updates by Microsoft to avoid this confusion (as noted in Part 1). Meanwhile the distinction should be kept in mind by developers using Char. For instance although the method Char.IsLetter(Char) returns true if the code unit is question is a letter-type character in the BMP, IsLetter(Char) is not in general a function you would use when looking for letters in a UTF-16 array.

It is therefore often a good idea to use strings or ‘string literal’ to represent characters, in preference to Char and ‘character literal’. Inexperienced programmers or those coming from a C/C++ background may find this odd to begin with, being familiar with patterns like

for (i=0; i < chars.Length; i++) { if (chars[i]=='A' { etc. }};

Fortunately, Char provides methods to work correctly with characters, for instance Char.InLetter(String, Int32). Char.IsLetter("A",0) returns true and the function works equally well for Char.IsLetter("\u10900",0) so long as your platform supports Unicode 5.1 (the version of Unicode that introduced U+10900 𐤀 PHOENICIAN LETTER ALF).

Char is largely portable apart from a few quirks. I find it especially puzzling that Char.ConvertFromUtf32 and Char.ConvertToUtf32 are missing from SL 3/4. Faced with this I wrote my own conversions and use these even on .Net 4.0 where the methods are available in Char, this way keeping my code portable and efficient.

I also wrote a CharU32 structure for use in UTF-32 processing (along with a StringU32) rather than extend Char to contain methods like IsLetter(int) for UTF-32 (as is done in Java). Makes for cleaner algorithms and something Microsoft may want to consider for .Net futures.

The CharUnicodeInfo class in SL 3/4 is a subset, missing GetDecimalDigit, a rather specialist helper for working with decimal numbers in non-Latin scripts and in most cases one would use GetNumericValue which is portable.

CharUnicodeInfo.GetUnicodeCategory and Char.GetUnicodeCategory have subtle differences.

String class

The .Net String is a sequential collection of Char objects whose value is immutable (i.e. its value is never modified but can be replaced. Informally, String encapsulates a read-only UTF-16 array. As with Char, the MSDN documentation tends to confuse Unicode character with UTF-16 code unit.

Comments above on use of string and string literal for Char also apply here. For instance the method IndexOf(Char) can only be used to find a Char, i.e. code unit. IndexOf(String) must be used to find an arbitrary Unicode character. If you try entering '\u10900' in C# in Visual Studio 2010, you will be warned “Too many characters in character literal” a reminder .NET character literals are not quite characters and "\u10900" is needed.

Much of String is portable. I’ll just mention a few differences:

Net 4.0 has extra String methods like StartsWith(String, Boolean, CultureInfo), a method which gives more flexibility in working with multiple cultures. SL 3/4 is oriented around current culture and invariant culture so not well suited to multilingual applications.

The whole model of culture and locale in software is way too complex to go into here, I’ll just say that the old fallacies like physical presence in Iceland means a user can speak Icelandic and never spends US dollars or that individuals use only one language have no place in the 21st Century.

SL 3/4 is missing the Normalize method, a very handy function when working with Unicode.

SL 3/4 has fewer constructors available to the programmer than Net 4 but some of those missing are error prone and wouldn’t normally be used in .Net 4.

StringBuilder class

Useful, gives measureable performance improvements constructing a lot of strings.

Very portable, just a few differences, I’ve not encountered these in real life code yet. Rather odd that Append(Decimal) is missing from SL 3/4 though.

StringInfo class

StringInfo is designed to work with actual Unicode characters so is very useful. I get the impression it is less well known among programmers than it ought to be.

Largely portable. However SL 3/4 is missing the SubstringByTextElements methods and LengthInTextElements both of which are handy for processing Unicode correctly in readable code. I’ve found myself simulating something similar to gain portability, although that being said, the portable enumeration methods make for a more common pattern.

Encoding classes

Encoding classes are often used while reading or writing text streams or converting from one text format to another, usually with a class derived from Encoding: SL 3/4 only provides the UnicodeEncoding (UTF-16) and UTF8Encoding classes, in contrast to .Net 4 which also has ASCIIEncoding, UTF7Encoding, and UTF32Encoding.

ASCII and UTF-7 are understandable, omission of UTF32Encoding is hard to see as anything other than an oversight given that UTF-32 is a useful weapon in the armoury of Unicode programming (although hardly ever a good idea for file streams). One of the first things I did in the early days of Silverlight 2 beta was to write conversion code. I hope Microsoft will add this to Silverlight 5, although for portability reasons it may be years before we can settle down with it.

The base Encoder class in .Net 4 has quite a number of extra methods and properties for Mail and Browser purposes as well as codepage handling. I tend to agree with the decision in SL to drop some of this complexity and discourage use of legacy stream formats even if that means occasionally custom code is required to convert ‘Windows ANSI’, ‘Mac OS Roman’ and the ISO/IEC 8859 series of 8 bit encodings.

The main features of Utf8Encoding such as BOM handling and exceptions are portable. Unicode normalization features are missing from SL 3/4 again, as with the String class.

Not the most portable part of .Net.

Of Characters and Strings (in .Net, C#, Silverlight …): Part 1

“The time has come,” the Walrus said,
“To talk of many things:
Of shoes—and ships—and sealing-wax—
Of characters—and strings—
And why the sea# is boiling hot—
And whether pigs have wings.”

(With apologies to Lewis Carroll, the Walrus, and the Carpenter).

During discussion of my comments ISO/Unicode scripts missing in OpenType on the Unicode mailing list, the point came up about desirability of greater understanding of Unicode among programmers and others involved with software development. For a start, there is one popular myth to dispel, the subject of this post which I hope to be the first of several notes on Unicode in .Net.

Myth debunk: a Unicode character is neither a cabbage nor a 16 bit code.

The origin of 16-bit confusion lies in the history of Unicode. Twenty years ago there were two initiatives underway to replace the already out-dated and problematic variety of 7/8-bit character encodings used to represent characters in modern scripts. A true Babel of ‘standard’ encodings back then made it impractical to write software to work with the worlds writing systems without a tremendous level of complexity. Unicode was originally conceived as a 16 bit coding to replace this mess. Meanwhile, the International Organization for Standardization (ISO) was working on ISO 10646 the ‘Universal Character Set’ UCS with space for many more characters than a 16-bit encoding has room for. The original ISO proposals for encoding were widely regarded as over complex so the ISO/Unicode approaches were merged by the time Unicode 2.0 was released in 1996. ISO 10646 now defines the Universal Character Set for Unicode. With unification, the notion of 16-bit characters became obsolete although a 16-bit encoding method remains (UTF-16) along with the popular 8-bit coding (UTF-8) and a 32-bit coding (UTF-32). Each encoding has its virtues. UTF stands for Unicode Transformation Format.

To understand what constitutes the Unicode notion of ‘character’, refer to http://www.unicode.org/versions/Unicode6.0.0/ (or the earlier version while the text of 6.0 is being completed). I will try to summarize briefly.

1. An abstract character is a unit of information for representation, control or organization of textual data. A Unicode abstract character is an abstract character encoded by the Unicode standard. Abstract characters not directly encoded in Unicode may well be capable of being represented by a Unicode combining character sequence. Each Unicode abstract character is assigned a unique name. Some combining sequences are also given names in Unicode, asserting their function as abstract characters.
2. A Unicode encoded character can be informally thought of as an abstract character along with its assigned Unicode code point (an integer in the range 0 to 10FFFF hexadecimal, the Unicode codespace). As noted above it is also assigned a unique name.
3. A Unicode character or simply character is normally used as shorthand for the term Unicode encoded character.

Here are two useful ways of describing Unicode characters:

U+006D LATIN SMALL LETTER M
U+13000 EGYPTIAN HIEROGLYPH A001
U+1F61C FACE WITH STUCK-OUT TONGUE AND WINKING EYE

And similar with the actual character displayed

U+006D – m – LATIN SMALL LETTER M
U+13000 – 𓀀 – EGYPTIAN HIEROGLYPH A001
U+1F61C – 😜 – FACE WITH STUCK-OUT TONGUE AND WINKING EYE

The first form is often preferable in scenarios where font support might not be present to display the actual character although on this blog I prefer to use the characters to encourage font diversity.

Note the conventional use of hexadecimal to state the value of the Unicode code point. This convention is different to that used in HTML where characters as numeric entities are written using decimal numbers rather than hexadecimal, e.g. &#77824; (13000 hexadecimal equals 77824 decimal).

From a programming perspective, the simplest way of representing Unicode is UTF-32 where each code point fits comfortably into a 32 bit data structure, e.g. in C# a uint or int (C/C++ programmers note C# defines as 32 bit, the size does not vary with CPU register size). Not entirely trivial because there may still be combining sequences. However UTF-32 is not used all that much in practice, not least because of memory cost.

Nowadays, most files containing Unicode text use UTF-8 encoding. UTF-8 uses 1 byte (octet) to encode the traditional 127 ASCII characters and up to 4 bytes to encode other characters. XML and HTML files are popular file formats that use Unicode (Mandatory in XML, optional in HTML where a surprising amount of the web, possibly 50%, still uses legacy encodings). I strongly recommend UTF-8 for text files rather than UTF-16 or legacy 8-bit encodings aka code pages etc. Having worked on several multilingual content-intensive projects, this is the golden rule, although I won’t expand further today on the whys and wherefores. [However I ought to mention the catch that is the ‘Byte order mark’, a byte sequence (0xEF, 0xBB, 0xBF) sometimes used at the start of a UTF-8 stream to assert UTF-8 not legacy text; this can confuse the novice particularly with ‘.txt’ files which can be Unicode or legacy. Windows Notepad uses BOM for Unicode text files. Visual Studio 2010 also uses BOM to prefix data in many file types including XML, XAML and C# code.]

UTF-16 is very popular with software writers working in C/C++ and .Net languages such as C#. A version of UTF-16 was the standard data format for Unicode 1.0. Unicode characters with character codes less than 0x10000 are said to belong to the Unicode BMP (Basic Multilingual Plane) and these are represented by one 16 bit number in UTF-16, other characters require two 16 bit numbers i.e. two UTF-16 codes from a range that do not encode characters, the so called surrogate code points dedicated to this purpose. As of Unicode 6.0, fewer than 50% of characters belong to the BMP but BMP characters account for a huge proportion of text in practice. This is by design; all popular modern languages have most script/writing system requirements addressed by the BMP and there are even specialist scripts such as Coptic defined here. Processing UTF-16 is often more efficient than UTF-8 and in most cases uses half the memory of UTF-32, all in all a good practical compromise solution.

Which brings me back to the 16-bit myth. The fact that so many popular characters belong to the BMP and only require one code unit in UTF-16 means it is easy to be mistaken into thinking most means all. The problem doesn’t even arise with UTF-8 and UTF-32 but the fact is much software uses UTF-16, indeed UTF-16 is essentially the native text encoding for Windows and .Net.

Example sources of 16-bit confusion:

The article on character sets at http://www.microsoft.com/typography/unicode/cs.htm is brazen:



This article is dated to 1997 but was probably written much earlier. Windows NT 3.1 (1993) was notable as the first computer operating system to use Unicode as its native text encoding and Microsoft deserves credit for this, alongside Apple who also did much to help early uptake of Unicode (but would not have a new operating system until OSX was released in 2001). I’m quoting this as an example of the fact that there are many old documents on the Web, confusing even when from reputable sources. I should mention, in contrast, much of MSDN (and indeed much of the relevant information on Wikipedia) is pretty up to date and reliable although not perfect on this subject.

The definition of the .Net Char structure on MSDN, http://msdn.microsoft.com/en-us/library/system.char.aspx, is much more recent.



Er, no. Char is not a Unicode character. It is a 16 bit Unicode code unit in UTF-16. Actually, this is explained later on in the Char documentation but the headline message is confusing and encourages programmers to use Char inappropriately.

The reasons I chose the Microsoft examples rather than the myriad of other confusing statements on the web are twofold. Firstly I'm focussing on .Net, C# etc. here. Secondly, Microsoft are generally ahead of the game with Unicode compared with other development systems which makes errors stand out more.

Fact is .Net actually works very well for software development with Unicode. The basic classes such as 'String' are Unicode (String is UTF-16) and it is almost true to say it is harder to write legacy than modern.

I had hoped to get a little further on the actual technicalities of working with Unicode characters and avoiding 16-bit pitfalls but time has proved the enemy. Another day.

Just three useful (I hope) points on .Net to conclude.

1. Code that works with String and Char should avoid BMP-thinking, e.g. if you want to parse a String, either avoid tests like IsLetter(Char) or wrap their usage in logic that also handles surrogates.

2. String, Char and the useful StringInfo class belong to the System namespaces and are pretty portable over the gamut of .Net contexts including Silverlight, WPF, XNA as well as the Novell parallel universe with Mono, MonoTouch, Moonlight etc. With a little care it can be straightforward to write text processing code that works across the board to target Windows, Mac, Linux, WP7 and whatever comes next.

3. Always test text-related code with strings that include non-BMP characters, and preferably also with data that includes combining sequences and usage instances of OpenType features such as ligatures.

Unicode 6.0 released: Let the challenge begin

Unicode 6.0.0 was released yesterday, October 12th 2010. This is a major update to Unicode - version 5.0 was released in 2006, followed by partial updates 5.1 (2008) and 5.2 (2009). Details are given at http://unicode.org/versions/Unicode6.0.0/, also see the Unicode, Inc. press release Unicode 6.0: Support for Popular Symbols in Asia. The Unicode character repertoire reflects the ISO/IEC 10646:2010 standard for characters, Unicode itself adding much of the technical information needed for implementation of writing systems.

All of which gobbeldy-gook masks the fact that Unicode is a rather wonderful thing, not only a valuable technology but also a work of art and beauty that eclipses much that passes for establishment and celebrity art of modern times in my personal opinion. Our world continues a path towards English as the lingua franca for Planet Earth with a clear decline in the relevance of traditional languages. Yet at the same time the technology that may be seen by some as a threat can also be the saviour. Unicode is the keystone. It is marvellous fact that 5000 years of diverse writing systems can become assessible to all for the first time in history and Unicode has played a pivotal role in making this happen during its 20 year evolution.

A specification is only a starting point. The complete text of Unicode 6.0 is still being revised for publication next year. I recently drew attention to ISO/Unicode scripts missing in OpenType and have since been informed that work is now underway to catch up on the missing scripts. Nevertheless it can be expected that it will take months and years before computer software and digital content catches up.

A fun addition to Unicode is the set of ‘Popular Symbols in Asia’ mentioned above. Emoticons. Here are four examples:

U+1F601 – 😁 – GRINNING FACE WITH SMILING EYES
U+1F63B – 😻 – SMILING CAT FACE WITH HEART-SHAPED EYES
U+1F64A – 🙊 – SPEAK NO EVIL MONKEY
U+1F64C – 🙌 – PERSON RAISING BOTH HANDS IN CELEBRATION

I suspect Emoticons will be the popular motivator for timely support of Unicode 6.0 by the usual corporate suspects (Apple, Google, Microsoft etc.). Meanwhile expect your web browser to show ‘unknown character’ for GRINNING FACE WITH SMILING EYES etc. above.

Search engines. When I checked this morning, neither Bing nor Google search were indexing the Emoticons, I’ll keep my eyes open to report on who wins that particular race.

Internet browser support. Internet Explorer is currently the most popular Internet browser and version 9 is currently in Beta. The standards based approach of IE9 and the promise of improved compatibility among new releases of all browsers though HTML5 support etc. is a very positive direction for the web. Firefox is the second most popular. Then Safari and Chrome, both webkit based. The level of Unicode 6.0 support in the major IE9 and Firefox 4 releases (expected in the first half of next year) may serve as one interesting predictor of directions in the Browser wars.

There are no especially strong motivators in the traditional Desktop software arena but the situation is different for newer device formats. Which of Android, Windows Phone 7, or iOS will support Emoticons first? What about eReaders? Silverlight, Flash. Traditionally, support for new versions of Uncode has been slow in coming but seems like the rules are different now.

Should make for an interesting 2011.

Footnote. Much of the content of Unicode is there because a large number of individuals have freely given their time simply because of the worthwhile nature of the project. (I don’t understand why the big picture has not captured the imagination of the wealthy people of our world.) In particular I’d like to mention Michael Everson (who I worked with on Egyptian Hieroglyphs and Transliteration) who deserves recognition for his many years of effort and a dogged determination to take Unicode beyond the short term requirements of commercial and national interests.

The Real Truth about 42

Today is Sunday, 10/10/10, an appropriate day to reflect on the number 42. I’d better explain for the sake of the sanity of non-mathematicians that binary 101010 is in fact the number 42 in disguise.

An obvious feature of 42 is its prime factorization: 2x3x7. Obvious can be boring so I'll add the more obscure fact that the sum 2x3 + 2x7 + 3x7 = 41, just one less than 42. I don’t know if anyone has named the class of numbers whose pair-wise sum of its prime factors plus 1 equals the number itself. That sum emerged en route on a visit to Hove so if anyone really needs a name how about a ‘Hove number’? Not an exceptional inspiration, but possibly a brand new observation to add to a large literature on the topic of 42.

More ‘fascinating’ facts about 42 can be found at Wikipedia - 42 (number) where I learned Charles Dodgson, writing as Lewis Carroll, was also fond of 42. Perhaps it’s in the Oxford tap water. Computer programmers may be amused by the fact that the wildcard character ‘*’ has character code 42 in ASCII and Unicode.

Truth is, the number 42 has been regarded as special for (probably) over 5000 years.

Traditionally, Ancient Egypt was divided into administrative districts, usually called ‘nomes’ nowadays (from the Greek word for ‘district’, Νομός; also Egyptian spꜣt/𓈈/etc.). Curiously when I placed ideograms for the 20 nomes of Lower Egypt and 22 nomes of Upper Egypt into the first draft for the (as then) proposed Unicode standard for Egyptian Hieroglyphs, it was only afterwards that 42 clicked ‘not that number AGAIN’. I expect the fame of 42 goes back to the dawn of writing and mathematics itself.

Thoth, a (male) Egyptian deity (Egyptian Ḏḥwty; 𓅝, 𓁟 etc.), was associated with wisdom, magic, writing, mathematics, astronomy and medicine. Maat, a (female) deity (Egyptian Mꜣꜥt; 𓁦, 𓁧 etc.) was associated with truth, equilibrium, justice and order. She represents a fundamental concept in Ancient Egyptian philosophy. In some later traditions which featured male-female pairing between deities, Thoth and Maat were linked together (although rarely in a romantic sense). Both deities are prominent in the judging of the deceased as featured in the ‘Book of the Dead’.

The Papyrus of Ani gives a list of 42 ‘negative confessions’ for the deceased – “I have not committed sin”, “I have not murdered” etc. The ‘Ten Commandments’ of the Old Testament can be thought of as a condensed version. Sometimes referred to as ‘the doctrine of Maat’. 42 associated deities, supervised by Thoth, were assigned to the judgment of the deceased during his or her passage through the underworld.

I can’t resist mentioning that the modern name “Book of the Dead” was invented by Karl Richard Lepsius (the Egyptian rw nw prt m hrw has been more literally translated as the ‘Spells of Coming Forth by Day’ or similar). It can be no more than coincidence that the publication in question, “Das Todtenbuch der Ägypter nach dem hieroglyphischen Papyrus in Turin mit einem Vorworte zum ersten Male Herausgegeben” was published in 1842. Lepsius was a major and influential figure during the emergence of the modern discipline of Egyptology as well as being responsible for the creation of the first hieroglyphic typeface as implemented by typographer Ferdinand Theinhardt, the “Thienhardt font”.

The ’42 Books of Thoth’ aka ’42 Books of Instructions’ were composed from around 3rd century BC supposedly based on earlier traditions. Only fragments remain from this Hermetic text which apparently contained books on philosophy, mathematics, magic, medicine, astronomy etc. A legendary source, highly influential in later traditions of mysticism, alchemy, occultism and magic. The 42 Books have been believed by some to contain the hidden key to the mysteries of immortality and the secrets of the Universe. A fruitful topic I guess for Dan Brown and other writers of fiction.

Trivia. Visiting the South Coast last December, I was amused to discover the return rail-fare from Oxford was £42. Got me thinking how often 42 has cropped up in my life. Coincidence can be good fun. I decided to keep an eye open for incidents involving near neighbours of 42: 40, 41, 43, and 44. A prospect so intriguing and exciting I’m surprised I woke up on the approach to a snow and ice encrusted Hove before the train rattled on its way to Worthing. I can now report the scientifically meaningless result after 10 months ‘research’. Those worthy siblings 40, 41, 43, 44 just don’t cut the mustard compared with their famous colleague. Perhaps it’s just me. Although when my son started at secondary school this September, there was a certain inevitability about his reply when asked in what number classroom his form was based. For a moment I thought he was kidding.

I can't really leave the topic without mentioning the obvious.

The writer most credited for the prominence of 42 in modern times is the late Douglas Adams. In his radio series “Hitchhikers Guide to the Galaxy” (BBC Radio 4, 1978), the “Answer to the Ultimate Question of Life, the Universe, and Everything” is calculated to be 42. The meme exploded. Adams later claimed to have picked 42 pretty much at random.

We will never know whether Adams knew of the antiquity of 42 as a profound and famous number, indeed as the answer to his very own ultimate question. Its easy to speculate that he must have held some knowledge, at least at some subconscious forgotten level. A remarkable coincidence otherwise, unless 42 is in fact the answer.

Yet not impossible. After all there is something rather cute and appealing about 42. She still looks good for her age. Don’t you think so too?

ISO/Unicode scripts missing in OpenType

Unicode 6.0 release is imminent (see www.unicode.org), a year after the release of Unicode 5.2 (October 2009). Version 6.0 introduces three new scripts: Mandaic, Batak, and Brahmi. There are extensions to other scripts and many other improvements and clarifications.

An aside to anyone involved in HTML5 standardisation. It would be a really good idea if Unicode 6.0 compatibility were specified as part of the formal standard for HTML, and included in conformance testing.

OpenType is the de-facto standard for font technology and as such an essential part of implementating a script. The latest set of script tags (codes) for OpenType is given at www.microsoft.com/typography/otspec/scripttags.htm (document last updated in January 2008 when checked today).

The current ISO-15924 list of script codes is given at www.unicode.org/iso15924/iso15924-codes.html.

Unfortunately, some Unicode scripts are missing from the OpenType script tag list. This is long overdue an update.

The fact that Unicode 5.2 has not been incorporated in OpenType specifications a year after release makes for an unsatisfactory situation. I am writing to those concerned and encourage others to do likewise.

The following 15 Unicode scripts are missing from OpenType:

Avestan (134, Avst, Unicode 5.2)
Bamum (435, Bamu, Unicode 5.2)
Batak (365, BatkUnicode 6.0)
Brahmi (300, Brah, Unicode 6.0)
Egyptian hieroglyphs (050, Egyp, Unicode 5.2)
Imperial Aramaic (124, Armi, Unicode 5.2)
Kaithi (317, Kthi, Unicode 5.2)
Lisu (Fraser) (399, Lisu, Unicode 5.2)
Mandaic, Mandaean (140, Mand, Unicode 6.0)
Old Turkic, Orkhon Runic (175, Orkh, Unicode 5.2)
Inscriptional Pahlavi (131, Phli, Unicode 5.2)
Inscriptional Parthian (230, Prti, Unicode 5.2)
Samaritan (123, Samr, Unicode 5.2)
Old South Arabian (105, Sarb, Unicode 5.2)
Tai Viet (359, Tavt, Unicode 5.2)

As a footnote. Not available in Unicode yet, but of interest to Egyptology are:

Meroitic Hieroglyphs (100, Mero, formal proposal with WG2)
Meroitic Cursive (101, Merc, formal proposal with WG2)
Egyptian Hieratic (060, Egyh, no formal proposal yet , contact me if you have any ideas)
Egyptian Demotic (070, Egyd, no formal proposal yet, contact me if you have any ideas)

There are also some desirable additions to be made to Egyptian Hieroglyphs (I'd like to see something with ISO/WG2 in 2012 if not before).

Simplified Egyptian: A brief Introduction

I coined the term ‘Simplified Egyptian’ several years ago as a technical approach to making Ancient Egyptian in hieroglyphs more useable in the modern digital world (see HieroglyphsEverywhere.pdf, Bob Richmond, 2006).

The snag in creating an implementation has long been external factors such as the status of web browsers and Word Processors, along with the associated de-facto or formal industry standards. The devil is in the detail and there are many idiosyncrasies in modern technology once one departs from the everyday. A notion like Simplified Egyptian would be no more than a curiosity if it were not widely accessible on personal computers and other digital devices.

One factor in the equation was the need to include Egyptian Hieroglyphs in the Unicode standard (published in Unicode 5.2, October 2009). Implementations of 5.2 are slowly becoming available; for instance Google web search now accepts hieroglyphs although Microsoft Bing and Yahoo search do not yet. Another key factor is support by Internet browsers. Firefox 3.6 looks viable now and I expect the latest versions of other popular browsers to support Egyptian to some degree within the next few months.

As various pieces of the technical puzzle appear to be coming together in the 2011 timeframe I thought it would be useful to summarise now what I see Simplified Egyptian being about. I envisage putting more flesh on the bones in future blog posts on a prototype implementation as leisure time permits (this is an unfunded project at present so time is the enemy).

Simplified Egyptian (SE) works as follows.

1. Define a subset of the Unicode 5.2 list of characters for Egyptian Hieroglyphs, avoiding variants and rarely used (in Middle Egyptian) characters.
2. Define fixed rules for combining hieroglyphs into groups so these rules can be implemented in TrueType/OpenType fonts or alternative rendering methods.
3. Use left to right writing direction.
4. Define data tables and algorithms for text manipulation and sorting.
5. Define ‘normalized forms’ for guidance on ‘correct’ ways of writing and processing Simplified Egyptian.

A more recent notion - Super-Simplified Egyptian (SSE) - takes these principles further by identifying an even more condensed subset of the Hieroglyphic script, a proper subset of SE, with a palette of fewer than 200 hieroglyphs.

There is no question that the SE method is highly anachronistic, SSE extremely so. Nevertheless, there is some utility in the approach.

I am also aware that superficially what I’m proposing suggests a flavour of modernised Egyptian at odds with the requirements of Egyptology for working with an ancient language with script usage that evolved over 3000 years. I will make no apologies for the fact that this is indeed one application and if SE encourages wider understanding of Egyptian albeit at a reduced technical level that is no bad thing in my opinion. Nevertheless, the most interesting aspect from my own interest is the question of how to use such a mechanism to enable improvements for academically sound publication and study of ancient texts in the context of 3000 years of language/script evolution. A non-trivial topic I shall not touch on further today.

My plan is to make available some small working examples of Simplified Egyptian on a series of web pages during the next few weeks. These examples use a WOFF (Web Open Font Format) font derived from my InScribe font. The reasons for doing this now are twofold.

1. A new generation of Internet Browsers pays greater attention to industry standards and should be capable of supporting Simplified Egyptian. Firefox 4 and Internet Explorer 9 are in Beta at the moment and I want to make the samples available for browser testing in case there is a need to shake out any browser bugs.

2. InScribe 3 for Windows will not use Simplified Egyptian. Originally it was my intention that SE would be a feature but it turned out to introduce too many complications in modes of use. Nevertheless, InScribe 3 retains some ‘SE-friendly’ characteristics and I want to be able to test these for real on the Web as I complete work on the software.

The samples will not work for users of older browser technology (right now this is a high ninety something per cent of internet capable devices). My short term concern is only that an elegant and simple to use implementation works as and when devices gain adequate support for internet standards.

That is not to say that workarounds can’t be contrived for devices whose manufacturers or users are not able or prepared to adapt to the new standards-based internet landscape. I'm happy to hear of any proposals.

Right now, this means use Firefox 3.6 or later to view samples as intended. I’m also tracking Chrome, Internet Explorer 9 Beta and Safari releases.

Browser of the month: Firefox

My post yesterday Quick test for Ancient Egyptian in web browsers (September 2010) actually exposed three bugs.

None of which involved Firefox. In fact Firefox 3.6 and later correctly display transliterations and hieroglyphs on a Windows system with a suitable Unicode 5.2 font containing hieroglyphs and the other characters.

The bugs are:
1. Latest releases of Chrome, Internet explorer (8 and 9 Beta) and Safari do not pick up that there is a local font with hieroglyphs. Basically a bug with Unicode 5.2 support I think. Attn: Apple, Google, Microsoft.
2. The same three browsers incorrectly process characters in the SMP given as character references e.g. &#55308;&#56639; Firefox is correct in displaying this pair as two bad characters per HTML specifications. This UTF-16 type of surrogate representation is not valid HTML: in my example the correct character reference is &#78143; Attn: Apple, Google, Microsoft.
3. The Blogspot post editing software gratuitiously changed my UTF-8 text into character references &#55308;&#56639; for no apparent reason. The editing software also is buggy when I try to re-edit the post. Attn: Google, Blogger

So a gold star to Mozilla/Firefox.

The wooden spoon ought to go to Google for hitting all three bugs but in mitigation I'll observe that hieroglyphs are now supported by Google search (unlike the situation with Microsoft Bing and Yahoo who haven't even caught up with Unicode 5.1 never mind 5.2. A tribute to corporate lethargy - wake up guys).

After discovering the Blogger bug, I've opened a secondary blog on WordPress - Journal of Total Obscurity 2. For the time being this remains my main blog but WordPress will be used for posts with hieroglyphs.

I've retained my original post here 'Quick test for Ancient Egyptian in web browsers (September 2010)' so bugs can be monitored but I've uploaded the correct version at Test page for Ancient Egyptian Hieroglyphs in Unicode (September 2010) on WordPress.

Quick test for Ancient Egyptian in web browsers (September 2010)

A quick test note to check Ancient Egyptian in Web browsers.

If you have a (Unicode 5.2 compatible) Egyptian font installed on your system, the next few lines ought to make sense:

ꜣꜢiIꜥꜤwWbBpPfFmMnNrRhHḥḤḫḪẖH̱sSšŠḳḲkKgGtTṯṮdDḏḎ

(in MdC this Egyptian transliteration reads +taa*AA*iIwWbBpPfFmMnNrRhh*HH*xx*XX*ss*SS*qq*kKgGtt*TT*dd*DD*)

𓄿𓇋𓏭𓂝𓅱𓃀𓊪𓆑𓅓𓈖𓂋𓉔𓎛𓐍𓄡𓋴𓈙𓈎𓎡𓎼𓏏𓍿𓂧𓆓

(in MdC these Egyptian hieroglyphs read +s-A-i-y-a-w-b-p-f-m-n-r-h-H-x-X-s-S-q-k-g-t-T-d-D)

In fact, this is a FAIL for hieroglyphs today on Windows for Chrome (6.0.472.63), Firefox (3.6.10) Internet Explorer 9 (Beta 9.0.7930), and Safari (5.0.2). Only Firefox successfully displays the transliteration.

Tantalizingly, the Firefox edit box does work:

Technically, all a browser needs to do is ennumerate all fonts on the host system and if the font implicit in the HTML is not present, use any font that supports the characters if available. Perhaps there needs to be some magic setting in the TrueType fonts for the browsers to work although this ought not to be necessary so I will count this as a multi-browser bug.

The lines should read:

Update. This site, Blogger, turned my HTML hieroglyph strings into entities, e.g. hieroglyphs in UTF-8 into  &#55308;&#56639; etc. Firebox has a bug in this case (entities in Unicode SMP) but not when raw UTF-8 is used in HTML so Firefox is very close to working, indeed it is good for many web pages. Blogger is a bit broken, the entities are simply confusing and bring nothing to the party.

Q. Why is my screen too small?

A. It’s a tradition.

Last month I mentioned the 25 year old RM Nimbus PC-186 and its 640x250 display. 250 was the number of lines displayable on a ‘CGA’ class CRT monitor of that time (more precisely at 50/60Hz non-interlaced). The 14" CGA was the only mass produced monitor available at a reasonable price in 1985 and it was that fact as much as the cost of the driver electronics that influenced the low resolution choice of display mode. By 1988, 14" ‘VGA’ type monitors were in mass production at 640x480 resolution and these soon gained higher definition 800x600.

During 1987-1993 one part of my job with RM involved working with a series of US Silicon Valley based companies who were growing the capabilities of PC graphics systems into the affordable market. Computer graphics has always been a personal interest so it was fun to be involved in bringing out the then emerging technology that is nowadays is taken for granted. My main role was writing device drivers for Windows and working with the chip designers to boost performance. During this period, the ‘holy grail’ was to reach 1024x768 24bit colour with an inexpensive design, a point we reached for the first time with a Cirrus Logic chip in 1993. This hit acceptable performance goals for Windows 3.1, removing the need for the transient 256 colour type displays popular for a while but problematic from an application programming point of view.

Two flies in the ointment. 1. Computer monitor manufacturers took a long time to come around to the obvious fact that 14"/15" CRT displays were too small for applications like word processing and spreadsheets. The sweet spot was 17"/19" but it seemed to take forever before it was accepted this was a volume market and the price benefits of mass production held sway (21" and above were cool but too unwieldy in CRT except for specialist applications such as CAD). 2. Most employers, schools and universities regarded it as acceptable to save a hundred dollars or so even if that meant seeing armies of highly paid employees and students hunched over small monitors peering at a fraction of a spreadsheet or page of text.

So much for history, though I’ll repeat the point that it seems to be a well-established tradition to use displays that are too small for purpose. Eventually things get better so nowadays good flat screen displays for desktop computers are very affordable. Although last year I visited the newsroom of a popular newspaper and it was almost laughable to see journalists and typesetters using displays that were obviously too small to efficiently work with a tabloid format. All to save the cost of a lunch or two!

Moores Law in the twenty first century. Electronics shrinking to give high functionality with reduced power consumption, and the consequential growth of small format computing: laptops, netbooks, smartphones, tablets, eBook readers. In each case the same pattern. Early devices have less than usable screen sizes and not just for reasons of manufacturing cost. Product marketing tries to avoid the fact that the emperor has no clothes. Keen leading edge users, in denial, claim its all ok. Markets learn and devices gradually move to something more ergonomic and pleasant to use.

This topic came to mind while I was tweaking an InScribe design for netbooks (typically a usable 1024x600 10" display nowadays after that unfortunate early fad for the 90s-retro 800x480 resolution on 7" in 2008). Reading today’s announcement of the upcoming RIM ‘PlayBook’ device (7" LCD, too small for its aspirations in my opinion. See the, I expect ill-fated, Dell Streak.). Not that 7"/8" is a bad format for many purposes (Note to Amazon with the 6” Kindle. And Sony. Try measuring a paperback book!).

Incidentally whatever the flaws in the first generation iPad, 9.7" is not dramatically smaller than the optimum size for purpose so kudos to Apple for bucking the usual pattern (although I personally think 11-12" touchscreen hits the right compromise between portability and function).

So if you find yourself peering at the internet through a 3.5" supposedly state of the art smartphone remember that for users to suffer for a while is a tradition, you are paying the price of being a part of history in the making, and things will soon get better (better for smartphones I suspect means about a 4.2-4.5" with narrow bezel compromise in current tech).

PS. Inches not metric; another tradition.

Document embedding and OLE

There are a number of technical issues concerned with what I'm attempting to accomplish with the InScribe software. The biggest thorn in my side is the issue of embedding, visualising, and editing embedded data in compound documents. This is therefore something of a background note on the topic.

The notion of embedding or linking an ‘object’ in a document is commonplace. Web pages incorporate pictures, videos and specialized objects such as Flash or Silverlight interactive components. Word processing documents likewise contain objects, sometimes interactive objects, alongside the text. The idea of ‘compound document’ goes back to the 1980s.

The current situation with embedded objects is chaotic. Examples. Microsoft Office and Open Office, the two most popular office suites, have different schemes for add-ins. Even limiting attention to one vendor, Microsoft Office has added features in the 2003, 2007 then 2010 editions: all well and good but makes it difficult to support a diverse user base (Office 2003 is still widely used). Firefox, Chrome, Internet Explorer each has its own plugin approach and the story gets more complicated when considering non-Windows Internet browsers.

We are sorely missing standard, flexible, open approaches to embedding. On the web side of the coin, HTML5 is a move in the right direction though in itself no panacea despite what some less technically minded commentators may say on the subject (a topic for a future blog entry!).

A concrete example. There is no bottom line hardware-related reason nowadays that a simple photo editor plug-in software component could not operate on devices as diverse as eBook readers, smartphones, tablet computers, as well as notebook and desktop computers. Such a plug-in could enrich many applications, not only web browsing but word processors and camera management tools. In our Babel like world of 2010, to accomplish such a thing in software would involve writing versions for iOS, OS X, Android, Windows XP, Windows 7, Symbian, Kindle, Blackberry, Gnome, Wii... the list goes on. The developer then needs to tackle application-specific rules for plug-ins (if such exist). A relatively simple piece of software is almost impossible to deploy widely.

I am not advocating one ring to rule them all, simply highlighting the severe lack of standard ways for applications and components to interoperate and deploy.

In fact, for Microsoft Windows applications there has been one solution for almost 20 years. OLE (Object Linking and embedding) was introduced to Windows in 1990 and expanded substantially to version 2 in 1993. Parts of the OLE system were renamed ActiveX controls in 1996. I’ve exploited OLE with the InScribe software to enable in-place editing of Ancient Egyptian in Word Processing and other applications.

OLE has never been an ideal technology solution. Parts are overcomplicated and error prone during development. OLE design is too specific to classic Windows architecture. There was originally insufficient attention given to security issues although this is now largely addressed. Nevertheless, for Windows applications, OLE provided some solutions to the big problem of making applications and components work together by defining standard rules for interoperability of functions like embedding and compound documents.

Unfortunately, OLE development by Microsoft pretty much stopped well over a decade ago, more a victim of fashion rather than any logical reason I suspect. One side effect is that something as fundamental as how copy and paste works between Windows applications is still stuck in a 1990s time-warp.

As the internet grew and new technologies such as Java and .Net became available, inasmuch as there was any attempt to address application interoperability, solutions tended to be product specific. Microsoft themselves targetted ‘the enterprise’ with less focus on the general personal computer user. Rather than an OLE philosophy where third parties can expand the capabilities of Windows and its applications in a general way, Microsoft Office became focussed on enterprise oriented Office specific add-ons. Linux and other alternatives failed to rise to the challenge of developing a more open and flexible approach.

OLE continues to be supported in Windows and Office, but pretty much maintenance only. OpenOffice and other products continue likewise. I have not looked at the latest Adobe Creative Suite (CS5) but recall an earlier move around CS3 to remove some OLE functionality. Microsoft Office Word 2010 runs OLE embedding in a compatibility mode. I don’t expect OLE to disappear in the next few years but it is certainly becoming less usable.

If anyone reading this understands why the issue of application interoperability and OLE type functionality is missing from .Net and WPF I'd be delighted to find out.

This slow decline of OLE and the lack of practical modern alternatives proved a stumbling block in my development of a new version of the InScribe for Windows software. I expect other developers are in a similar situation of having to make some undesirable compromises in order to get a product released. On a positive note, the problem has stimulated a number of interesting ideas for future directions and I hope to touch on these here during the next few months.

Meanwhile a call for anyone working on document embedding to please learn from the past. Lets try to ensure whatever is latest and greatest at least accomplishes what OLE did 20 years ago (and still almost does).

Mangaglyphics

Apparently the Japanese word manga (katakana マンガ, kanji 漫画, hiragana まんが) can be loosely translated to English as “whimsical pictures”. Distinctive manga styles have seen growing popularity outside Japan during the last few decades predominantly through comic book, cartoon, and video game formats.

Some time ago I had the crazy notion that there are some interesting ways to combine the tradition of ancient Egyptian Hieroglyphs with manga styles in an entertaining way, mangaglypic seemed like the word rather than the equally obvious hieromanga. Fun, possibly with some educational value.
However with the work that needs to be done on improving accessibility to non-whimsical applications of ancient Egyptian on personal computers and other devices mangaglyphic is pretty low on my software to do list.

So why mention the term right now? Partly because it looks like a mangaglyph or two are creeping unasked into the InScribeX Web user interface. Partly because I’d be delighted to hear from artists or others experimenting with this style of image. However what actually stimulated my writing today was discovering the search engine bing.com still returns zero results for mangaglyphic or related words and google.com returns only one result. So in the unlikely event the term catches on at all I wanted to state mangaglyphic is meant to be a generic word. No attempts to register trademarks etc. please.