Silverlight made it onto the BBC News on Tuesday – Coders decry Silverlight change. Take an unfortunate choice of words by a senior executive or two; add the reactive and ill-informed commentators on some web message boards; then mix in some natural concerns from developers. Bang! Tempests in teacups, the media love them.
Personally speaking, I find it reassuring to observe that the amateur tradition is alive and well in Microsoft and at least one major multinational company is not self-wrapped in a cloak of PR and spin-doctoring. That being said, the last few minutes of Doctor Who The Christmas Invasion ought to be made compulsory viewing for all senior executives.
As a developer I'm happy so long as .Net is treated as a strategic family of products. Thanks to Novell it may become so on Unix/Linux too (even if the Linux ‘community’ is slow to recognize what the third wave of Unix is really about). Hey theres another tabloid headline: C/C++ is dead!
I hope I'm not alone in being pleased to learn Silverlight 5 is not being rushed out. Especially if it means some of the niggles are resolved and the SL/WP7/WPF portability model improved. And Unicode 6.0 of course! A Mix 2011 Beta with Summer release please.
Two real news stories for developers:
An interesting talk at PDC 2010 for C# developers: ‘The Future of C# and Visual Basic’ by Anders Hejlsberg – don’t be put off like I almost was by the Visual Basic tag, it is hardly mentioned so we are not subjected yet again to the irony implicit in the keyword Dim. The main theme is simplification of Asynch programming with the new await keyword for the next .Net revision. Along with parallel constructs, this pattern brings very useful ways of exploiting multi-core processors to .Net in a clean software design. The talk is summarised here.
Developers in the .Net/WPF/Silverlight space should also check out PDC 2010: 3-Screen Coding: Sharing code between Windows Phone, Silverlight, and .NET by Shawn Burke. I alluded to the value of portable code last month in Of Characters and Strings although I didn’t highlight the .Net 4 changes that enable sharing of binary assemblies (a topic in its own right). The new tooling for Visual Studio to assist in creating Portable Assemblies, as previewed by Shawn, should be very helpful in managing the shared assembly model. It should also help focus Microsoft development on removing some of the irritating incompatibilities between Silverlight and WPF.
I just can’t wait for await.
Bob Richmond
Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts
Thursday, 4 November 2010
Friday, 22 October 2010
Of Characters and Strings (in .Net, C#, Silverlight …): Part 2
"Your manuscript I’ve read my friend
And like the half you’ve pilfered best;
Be sure the piece you yet may mend –
Take courage man and steal the rest."
Anon (c. 1805)
As mentioned in Of Characters and Strings (in .Net, C#, Silverlight …): Part 1 many of the text processing features of .Net are such that code can be re-used over Silverlight, WPF, XNA, Mono, Moonlight etc. In this note I want to draw attention to a few key points and differences over platforms. It is far from a comprehensive analysis and I’ve skipped some important topics including collation, the variety of ways of displaying text ,and input methods.
A quick aside: I failed to highlight in Part 1 how 16-bit confusion in software is not limited to .Net, similar issues arise in Java (although Java has some helpful UTF-32 methods that .Net lacks) and the situation is far worse in the world of C/C++.
I’ll restrict attention to Silverlight 4, Silverlight 3 for Windows Phone 7 OS (WPOS7.0), and .Net 4. In case the significance of the opening quote passes the reader by, it is to be hoped that Silverlight 5 and any new branches on the .Net tree feature increased compatibility, deprecating certain .Net 4 elements where appropriate.
.Net Text in general
Many Silverlight (and WPF) programmers rely mainly on controls and services written by others to deal with text so are unfamiliar with the lower level implications – in this case the main thing is to make sure you test input, processing, and display components with text that contains non-BMP characters, combining character sequences, and a variety of scripts and fonts. You may also like to check out different culture settings. Relying on typing querty123 on a UK/US keyboard and using the default font is a sure-fire recipe for bugs in all but the simplest scenarios.
.Net text is oriented around UTF-16 so you will be using Unicode all or almost all the time. The dev. tools, e.g. Visual Studio 2010 and Expression Blend 4, work pretty well with Unicode and cope with the basics of non-BMP characters. I find the main nuisance with the VS 2010 code editor is that only one font is used at a given time, I can’t currently combine the attractive Consolas font with fall-back to other fonts when a character is undefined in Consolas.
For portability over .Net implementations, the best bet at present is to write the first cut in Silverlight 3 since for this topic SL3 is not far off being a subset of .Net 4. A good design will separate the text functionality support from the platform-specific elements, e.g. it’s not usually a good idea to mix text processing implementation in with the VM in a MVVM design or in the guts of a custom control. I often factor out my text processing into an assembly containing only portable code.
Text display on the various platforms is a large topic in its own right with a number of issues for portability once you go beyond using standard controls in SL, WPF etc. I’ll not go into this here. Likewise with text input methods.
Versions of Unicode
It would be really useful if the various flavours and versions of .Net were assertive on text functionality, e.g. Supports Unicode 6.0 (the latest version) or whatever. By support, I mean the relevant Unicode data tables are used (e.g. scripts, character info) not to imply that all or even most functionality is available for all related languages, scripts and cultures. Currently, the reality is much more fuzzy and there are issues derived from factors such as ISO/Unicode scripts missing in OpenType. Methods like Char.IsLetter can be useful in constructing a 'version detector' to figure out what is actually available at runtime.
Char structure
The .Net Char structure is a UTF-16 code unit, not a Unicode character. Current MSDN documentation still needs further updates by Microsoft to avoid this confusion (as noted in Part 1). Meanwhile the distinction should be kept in mind by developers using Char. For instance although the method Char.IsLetter(Char) returns true if the code unit is question is a letter-type character in the BMP, IsLetter(Char) is not in general a function you would use when looking for letters in a UTF-16 array.
It is therefore often a good idea to use strings or ‘string literal’ to represent characters, in preference to Char and ‘character literal’. Inexperienced programmers or those coming from a C/C++ background may find this odd to begin with, being familiar with patterns like
for (i=0; i < chars.Length; i++) { if (chars[i]=='A' { etc. }};
Fortunately, Char provides methods to work correctly with characters, for instance Char.InLetter(String, Int32). Char.IsLetter("A",0) returns true and the function works equally well for Char.IsLetter("\u10900",0) so long as your platform supports Unicode 5.1 (the version of Unicode that introduced U+10900 𐤀 PHOENICIAN LETTER ALF).
Char is largely portable apart from a few quirks. I find it especially puzzling that Char.ConvertFromUtf32 and Char.ConvertToUtf32 are missing from SL 3/4. Faced with this I wrote my own conversions and use these even on .Net 4.0 where the methods are available in Char, this way keeping my code portable and efficient.
I also wrote a CharU32 structure for use in UTF-32 processing (along with a StringU32) rather than extend Char to contain methods like IsLetter(int) for UTF-32 (as is done in Java). Makes for cleaner algorithms and something Microsoft may want to consider for .Net futures.
The CharUnicodeInfo class in SL 3/4 is a subset, missing GetDecimalDigit, a rather specialist helper for working with decimal numbers in non-Latin scripts and in most cases one would use GetNumericValue which is portable.
CharUnicodeInfo.GetUnicodeCategory and Char.GetUnicodeCategory have subtle differences.
String class
The .Net String is a sequential collection of Char objects whose value is immutable (i.e. its value is never modified but can be replaced. Informally, String encapsulates a read-only UTF-16 array. As with Char, the MSDN documentation tends to confuse Unicode character with UTF-16 code unit.
Comments above on use of string and string literal for Char also apply here. For instance the method IndexOf(Char) can only be used to find a Char, i.e. code unit. IndexOf(String) must be used to find an arbitrary Unicode character. If you try entering '\u10900' in C# in Visual Studio 2010, you will be warned “Too many characters in character literal” a reminder .NET character literals are not quite characters and "\u10900" is needed.
Much of String is portable. I’ll just mention a few differences:
Net 4.0 has extra String methods like StartsWith(String, Boolean, CultureInfo), a method which gives more flexibility in working with multiple cultures. SL 3/4 is oriented around current culture and invariant culture so not well suited to multilingual applications.
The whole model of culture and locale in software is way too complex to go into here, I’ll just say that the old fallacies like physical presence in Iceland means a user can speak Icelandic and never spends US dollars or that individuals use only one language have no place in the 21st Century.
SL 3/4 is missing the Normalize method, a very handy function when working with Unicode.
SL 3/4 has fewer constructors available to the programmer than Net 4 but some of those missing are error prone and wouldn’t normally be used in .Net 4.
StringBuilder class
Useful, gives measureable performance improvements constructing a lot of strings.
Very portable, just a few differences, I’ve not encountered these in real life code yet. Rather odd that Append(Decimal) is missing from SL 3/4 though.
StringInfo class
StringInfo is designed to work with actual Unicode characters so is very useful. I get the impression it is less well known among programmers than it ought to be.
Largely portable. However SL 3/4 is missing the SubstringByTextElements methods and LengthInTextElements both of which are handy for processing Unicode correctly in readable code. I’ve found myself simulating something similar to gain portability, although that being said, the portable enumeration methods make for a more common pattern.
Encoding classes
Encoding classes are often used while reading or writing text streams or converting from one text format to another, usually with a class derived from Encoding: SL 3/4 only provides the UnicodeEncoding (UTF-16) and UTF8Encoding classes, in contrast to .Net 4 which also has ASCIIEncoding, UTF7Encoding, and UTF32Encoding.
ASCII and UTF-7 are understandable, omission of UTF32Encoding is hard to see as anything other than an oversight given that UTF-32 is a useful weapon in the armoury of Unicode programming (although hardly ever a good idea for file streams). One of the first things I did in the early days of Silverlight 2 beta was to write conversion code. I hope Microsoft will add this to Silverlight 5, although for portability reasons it may be years before we can settle down with it.
The base Encoder class in .Net 4 has quite a number of extra methods and properties for Mail and Browser purposes as well as codepage handling. I tend to agree with the decision in SL to drop some of this complexity and discourage use of legacy stream formats even if that means occasionally custom code is required to convert ‘Windows ANSI’, ‘Mac OS Roman’ and the ISO/IEC 8859 series of 8 bit encodings.
The main features of Utf8Encoding such as BOM handling and exceptions are portable. Unicode normalization features are missing from SL 3/4 again, as with the String class.
Not the most portable part of .Net.
And like the half you’ve pilfered best;
Be sure the piece you yet may mend –
Take courage man and steal the rest."
Anon (c. 1805)
As mentioned in Of Characters and Strings (in .Net, C#, Silverlight …): Part 1 many of the text processing features of .Net are such that code can be re-used over Silverlight, WPF, XNA, Mono, Moonlight etc. In this note I want to draw attention to a few key points and differences over platforms. It is far from a comprehensive analysis and I’ve skipped some important topics including collation, the variety of ways of displaying text ,and input methods.
A quick aside: I failed to highlight in Part 1 how 16-bit confusion in software is not limited to .Net, similar issues arise in Java (although Java has some helpful UTF-32 methods that .Net lacks) and the situation is far worse in the world of C/C++.
I’ll restrict attention to Silverlight 4, Silverlight 3 for Windows Phone 7 OS (WPOS7.0), and .Net 4. In case the significance of the opening quote passes the reader by, it is to be hoped that Silverlight 5 and any new branches on the .Net tree feature increased compatibility, deprecating certain .Net 4 elements where appropriate.
.Net Text in general
Many Silverlight (and WPF) programmers rely mainly on controls and services written by others to deal with text so are unfamiliar with the lower level implications – in this case the main thing is to make sure you test input, processing, and display components with text that contains non-BMP characters, combining character sequences, and a variety of scripts and fonts. You may also like to check out different culture settings. Relying on typing querty123 on a UK/US keyboard and using the default font is a sure-fire recipe for bugs in all but the simplest scenarios.
.Net text is oriented around UTF-16 so you will be using Unicode all or almost all the time. The dev. tools, e.g. Visual Studio 2010 and Expression Blend 4, work pretty well with Unicode and cope with the basics of non-BMP characters. I find the main nuisance with the VS 2010 code editor is that only one font is used at a given time, I can’t currently combine the attractive Consolas font with fall-back to other fonts when a character is undefined in Consolas.
For portability over .Net implementations, the best bet at present is to write the first cut in Silverlight 3 since for this topic SL3 is not far off being a subset of .Net 4. A good design will separate the text functionality support from the platform-specific elements, e.g. it’s not usually a good idea to mix text processing implementation in with the VM in a MVVM design or in the guts of a custom control. I often factor out my text processing into an assembly containing only portable code.
Text display on the various platforms is a large topic in its own right with a number of issues for portability once you go beyond using standard controls in SL, WPF etc. I’ll not go into this here. Likewise with text input methods.
Versions of Unicode
It would be really useful if the various flavours and versions of .Net were assertive on text functionality, e.g. Supports Unicode 6.0 (the latest version) or whatever. By support, I mean the relevant Unicode data tables are used (e.g. scripts, character info) not to imply that all or even most functionality is available for all related languages, scripts and cultures. Currently, the reality is much more fuzzy and there are issues derived from factors such as ISO/Unicode scripts missing in OpenType. Methods like Char.IsLetter can be useful in constructing a 'version detector' to figure out what is actually available at runtime.
Char structure
The .Net Char structure is a UTF-16 code unit, not a Unicode character. Current MSDN documentation still needs further updates by Microsoft to avoid this confusion (as noted in Part 1). Meanwhile the distinction should be kept in mind by developers using Char. For instance although the method Char.IsLetter(Char) returns true if the code unit is question is a letter-type character in the BMP, IsLetter(Char) is not in general a function you would use when looking for letters in a UTF-16 array.
It is therefore often a good idea to use strings or ‘string literal’ to represent characters, in preference to Char and ‘character literal’. Inexperienced programmers or those coming from a C/C++ background may find this odd to begin with, being familiar with patterns like
for (i=0; i < chars.Length; i++) { if (chars[i]=='A' { etc. }};
Fortunately, Char provides methods to work correctly with characters, for instance Char.InLetter(String, Int32). Char.IsLetter("A",0) returns true and the function works equally well for Char.IsLetter("\u10900",0) so long as your platform supports Unicode 5.1 (the version of Unicode that introduced U+10900 𐤀 PHOENICIAN LETTER ALF).
Char is largely portable apart from a few quirks. I find it especially puzzling that Char.ConvertFromUtf32 and Char.ConvertToUtf32 are missing from SL 3/4. Faced with this I wrote my own conversions and use these even on .Net 4.0 where the methods are available in Char, this way keeping my code portable and efficient.
I also wrote a CharU32 structure for use in UTF-32 processing (along with a StringU32) rather than extend Char to contain methods like IsLetter(int) for UTF-32 (as is done in Java). Makes for cleaner algorithms and something Microsoft may want to consider for .Net futures.
The CharUnicodeInfo class in SL 3/4 is a subset, missing GetDecimalDigit, a rather specialist helper for working with decimal numbers in non-Latin scripts and in most cases one would use GetNumericValue which is portable.
CharUnicodeInfo.GetUnicodeCategory and Char.GetUnicodeCategory have subtle differences.
String class
The .Net String is a sequential collection of Char objects whose value is immutable (i.e. its value is never modified but can be replaced. Informally, String encapsulates a read-only UTF-16 array. As with Char, the MSDN documentation tends to confuse Unicode character with UTF-16 code unit.
Comments above on use of string and string literal for Char also apply here. For instance the method IndexOf(Char) can only be used to find a Char, i.e. code unit. IndexOf(String) must be used to find an arbitrary Unicode character. If you try entering '\u10900' in C# in Visual Studio 2010, you will be warned “Too many characters in character literal” a reminder .NET character literals are not quite characters and "\u10900" is needed.
Much of String is portable. I’ll just mention a few differences:
Net 4.0 has extra String methods like StartsWith(String, Boolean, CultureInfo), a method which gives more flexibility in working with multiple cultures. SL 3/4 is oriented around current culture and invariant culture so not well suited to multilingual applications.
The whole model of culture and locale in software is way too complex to go into here, I’ll just say that the old fallacies like physical presence in Iceland means a user can speak Icelandic and never spends US dollars or that individuals use only one language have no place in the 21st Century.
SL 3/4 is missing the Normalize method, a very handy function when working with Unicode.
SL 3/4 has fewer constructors available to the programmer than Net 4 but some of those missing are error prone and wouldn’t normally be used in .Net 4.
StringBuilder class
Useful, gives measureable performance improvements constructing a lot of strings.
Very portable, just a few differences, I’ve not encountered these in real life code yet. Rather odd that Append(Decimal) is missing from SL 3/4 though.
StringInfo class
StringInfo is designed to work with actual Unicode characters so is very useful. I get the impression it is less well known among programmers than it ought to be.
Largely portable. However SL 3/4 is missing the SubstringByTextElements methods and LengthInTextElements both of which are handy for processing Unicode correctly in readable code. I’ve found myself simulating something similar to gain portability, although that being said, the portable enumeration methods make for a more common pattern.
Encoding classes
Encoding classes are often used while reading or writing text streams or converting from one text format to another, usually with a class derived from Encoding: SL 3/4 only provides the UnicodeEncoding (UTF-16) and UTF8Encoding classes, in contrast to .Net 4 which also has ASCIIEncoding, UTF7Encoding, and UTF32Encoding.
ASCII and UTF-7 are understandable, omission of UTF32Encoding is hard to see as anything other than an oversight given that UTF-32 is a useful weapon in the armoury of Unicode programming (although hardly ever a good idea for file streams). One of the first things I did in the early days of Silverlight 2 beta was to write conversion code. I hope Microsoft will add this to Silverlight 5, although for portability reasons it may be years before we can settle down with it.
The base Encoder class in .Net 4 has quite a number of extra methods and properties for Mail and Browser purposes as well as codepage handling. I tend to agree with the decision in SL to drop some of this complexity and discourage use of legacy stream formats even if that means occasionally custom code is required to convert ‘Windows ANSI’, ‘Mac OS Roman’ and the ISO/IEC 8859 series of 8 bit encodings.
The main features of Utf8Encoding such as BOM handling and exceptions are portable. Unicode normalization features are missing from SL 3/4 again, as with the String class.
Not the most portable part of .Net.
Tuesday, 19 October 2010
Of Characters and Strings (in .Net, C#, Silverlight …): Part 1
“The time has come,” the Walrus said,
“To talk of many things:
Of shoes—and ships—and sealing-wax—
Of characters—and strings—
And why the sea# is boiling hot—
And whether pigs have wings.”
(With apologies to Lewis Carroll, the Walrus, and the Carpenter).
During discussion of my comments ISO/Unicode scripts missing in OpenType on the Unicode mailing list, the point came up about desirability of greater understanding of Unicode among programmers and others involved with software development. For a start, there is one popular myth to dispel, the subject of this post which I hope to be the first of several notes on Unicode in .Net.
Myth debunk: a Unicode character is neither a cabbage nor a 16 bit code.
The origin of 16-bit confusion lies in the history of Unicode. Twenty years ago there were two initiatives underway to replace the already out-dated and problematic variety of 7/8-bit character encodings used to represent characters in modern scripts. A true Babel of ‘standard’ encodings back then made it impractical to write software to work with the worlds writing systems without a tremendous level of complexity. Unicode was originally conceived as a 16 bit coding to replace this mess. Meanwhile, the International Organization for Standardization (ISO) was working on ISO 10646 the ‘Universal Character Set’ UCS with space for many more characters than a 16-bit encoding has room for. The original ISO proposals for encoding were widely regarded as over complex so the ISO/Unicode approaches were merged by the time Unicode 2.0 was released in 1996. ISO 10646 now defines the Universal Character Set for Unicode. With unification, the notion of 16-bit characters became obsolete although a 16-bit encoding method remains (UTF-16) along with the popular 8-bit coding (UTF-8) and a 32-bit coding (UTF-32). Each encoding has its virtues. UTF stands for Unicode Transformation Format.
To understand what constitutes the Unicode notion of ‘character’, refer to http://www.unicode.org/versions/Unicode6.0.0/ (or the earlier version while the text of 6.0 is being completed). I will try to summarize briefly.
1. An abstract character is a unit of information for representation, control or organization of textual data. A Unicode abstract character is an abstract character encoded by the Unicode standard. Abstract characters not directly encoded in Unicode may well be capable of being represented by a Unicode combining character sequence. Each Unicode abstract character is assigned a unique name. Some combining sequences are also given names in Unicode, asserting their function as abstract characters.
2. A Unicode encoded character can be informally thought of as an abstract character along with its assigned Unicode code point (an integer in the range 0 to 10FFFF hexadecimal, the Unicode codespace). As noted above it is also assigned a unique name.
3. A Unicode character or simply character is normally used as shorthand for the term Unicode encoded character.
Here are two useful ways of describing Unicode characters:
U+006D LATIN SMALL LETTER M
U+13000 EGYPTIAN HIEROGLYPH A001
U+1F61C FACE WITH STUCK-OUT TONGUE AND WINKING EYE
And similar with the actual character displayed
U+006D – m – LATIN SMALL LETTER M
U+13000 – 𓀀 – EGYPTIAN HIEROGLYPH A001
U+1F61C – 😜 – FACE WITH STUCK-OUT TONGUE AND WINKING EYE
The first form is often preferable in scenarios where font support might not be present to display the actual character although on this blog I prefer to use the characters to encourage font diversity.
Note the conventional use of hexadecimal to state the value of the Unicode code point. This convention is different to that used in HTML where characters as numeric entities are written using decimal numbers rather than hexadecimal, e.g. 𓀀 (13000 hexadecimal equals 77824 decimal).
From a programming perspective, the simplest way of representing Unicode is UTF-32 where each code point fits comfortably into a 32 bit data structure, e.g. in C# a uint or int (C/C++ programmers note C# defines as 32 bit, the size does not vary with CPU register size). Not entirely trivial because there may still be combining sequences. However UTF-32 is not used all that much in practice, not least because of memory cost.
Nowadays, most files containing Unicode text use UTF-8 encoding. UTF-8 uses 1 byte (octet) to encode the traditional 127 ASCII characters and up to 4 bytes to encode other characters. XML and HTML files are popular file formats that use Unicode (Mandatory in XML, optional in HTML where a surprising amount of the web, possibly 50%, still uses legacy encodings). I strongly recommend UTF-8 for text files rather than UTF-16 or legacy 8-bit encodings aka code pages etc. Having worked on several multilingual content-intensive projects, this is the golden rule, although I won’t expand further today on the whys and wherefores. [However I ought to mention the catch that is the ‘Byte order mark’, a byte sequence (0xEF, 0xBB, 0xBF) sometimes used at the start of a UTF-8 stream to assert UTF-8 not legacy text; this can confuse the novice particularly with ‘.txt’ files which can be Unicode or legacy. Windows Notepad uses BOM for Unicode text files. Visual Studio 2010 also uses BOM to prefix data in many file types including XML, XAML and C# code.]
UTF-16 is very popular with software writers working in C/C++ and .Net languages such as C#. A version of UTF-16 was the standard data format for Unicode 1.0. Unicode characters with character codes less than 0x10000 are said to belong to the Unicode BMP (Basic Multilingual Plane) and these are represented by one 16 bit number in UTF-16, other characters require two 16 bit numbers i.e. two UTF-16 codes from a range that do not encode characters, the so called surrogate code points dedicated to this purpose. As of Unicode 6.0, fewer than 50% of characters belong to the BMP but BMP characters account for a huge proportion of text in practice. This is by design; all popular modern languages have most script/writing system requirements addressed by the BMP and there are even specialist scripts such as Coptic defined here. Processing UTF-16 is often more efficient than UTF-8 and in most cases uses half the memory of UTF-32, all in all a good practical compromise solution.
Which brings me back to the 16-bit myth. The fact that so many popular characters belong to the BMP and only require one code unit in UTF-16 means it is easy to be mistaken into thinking most means all. The problem doesn’t even arise with UTF-8 and UTF-32 but the fact is much software uses UTF-16, indeed UTF-16 is essentially the native text encoding for Windows and .Net.
Example sources of 16-bit confusion:
The article on character sets at http://www.microsoft.com/typography/unicode/cs.htm is brazen:
This article is dated to 1997 but was probably written much earlier. Windows NT 3.1 (1993) was notable as the first computer operating system to use Unicode as its native text encoding and Microsoft deserves credit for this, alongside Apple who also did much to help early uptake of Unicode (but would not have a new operating system until OSX was released in 2001). I’m quoting this as an example of the fact that there are many old documents on the Web, confusing even when from reputable sources. I should mention, in contrast, much of MSDN (and indeed much of the relevant information on Wikipedia) is pretty up to date and reliable although not perfect on this subject.
The definition of the .Net Char structure on MSDN, http://msdn.microsoft.com/en-us/library/system.char.aspx, is much more recent.
Er, no. Char is not a Unicode character. It is a 16 bit Unicode code unit in UTF-16. Actually, this is explained later on in the Char documentation but the headline message is confusing and encourages programmers to use Char inappropriately.
The reasons I chose the Microsoft examples rather than the myriad of other confusing statements on the web are twofold. Firstly I'm focussing on .Net, C# etc. here. Secondly, Microsoft are generally ahead of the game with Unicode compared with other development systems which makes errors stand out more.
Fact is .Net actually works very well for software development with Unicode. The basic classes such as 'String' are Unicode (String is UTF-16) and it is almost true to say it is harder to write legacy than modern.
I had hoped to get a little further on the actual technicalities of working with Unicode characters and avoiding 16-bit pitfalls but time has proved the enemy. Another day.
Just three useful (I hope) points on .Net to conclude.
1. Code that works with String and Char should avoid BMP-thinking, e.g. if you want to parse a String, either avoid tests like IsLetter(Char) or wrap their usage in logic that also handles surrogates.
2. String, Char and the useful StringInfo class belong to the System namespaces and are pretty portable over the gamut of .Net contexts including Silverlight, WPF, XNA as well as the Novell parallel universe with Mono, MonoTouch, Moonlight etc. With a little care it can be straightforward to write text processing code that works across the board to target Windows, Mac, Linux, WP7 and whatever comes next.
3. Always test text-related code with strings that include non-BMP characters, and preferably also with data that includes combining sequences and usage instances of OpenType features such as ligatures.
“To talk of many things:
Of shoes—and ships—and sealing-wax—
Of characters—and strings—
And why the sea# is boiling hot—
And whether pigs have wings.”
(With apologies to Lewis Carroll, the Walrus, and the Carpenter).
During discussion of my comments ISO/Unicode scripts missing in OpenType on the Unicode mailing list, the point came up about desirability of greater understanding of Unicode among programmers and others involved with software development. For a start, there is one popular myth to dispel, the subject of this post which I hope to be the first of several notes on Unicode in .Net.
Myth debunk: a Unicode character is neither a cabbage nor a 16 bit code.
The origin of 16-bit confusion lies in the history of Unicode. Twenty years ago there were two initiatives underway to replace the already out-dated and problematic variety of 7/8-bit character encodings used to represent characters in modern scripts. A true Babel of ‘standard’ encodings back then made it impractical to write software to work with the worlds writing systems without a tremendous level of complexity. Unicode was originally conceived as a 16 bit coding to replace this mess. Meanwhile, the International Organization for Standardization (ISO) was working on ISO 10646 the ‘Universal Character Set’ UCS with space for many more characters than a 16-bit encoding has room for. The original ISO proposals for encoding were widely regarded as over complex so the ISO/Unicode approaches were merged by the time Unicode 2.0 was released in 1996. ISO 10646 now defines the Universal Character Set for Unicode. With unification, the notion of 16-bit characters became obsolete although a 16-bit encoding method remains (UTF-16) along with the popular 8-bit coding (UTF-8) and a 32-bit coding (UTF-32). Each encoding has its virtues. UTF stands for Unicode Transformation Format.
To understand what constitutes the Unicode notion of ‘character’, refer to http://www.unicode.org/versions/Unicode6.0.0/ (or the earlier version while the text of 6.0 is being completed). I will try to summarize briefly.
1. An abstract character is a unit of information for representation, control or organization of textual data. A Unicode abstract character is an abstract character encoded by the Unicode standard. Abstract characters not directly encoded in Unicode may well be capable of being represented by a Unicode combining character sequence. Each Unicode abstract character is assigned a unique name. Some combining sequences are also given names in Unicode, asserting their function as abstract characters.
2. A Unicode encoded character can be informally thought of as an abstract character along with its assigned Unicode code point (an integer in the range 0 to 10FFFF hexadecimal, the Unicode codespace). As noted above it is also assigned a unique name.
3. A Unicode character or simply character is normally used as shorthand for the term Unicode encoded character.
Here are two useful ways of describing Unicode characters:
U+006D LATIN SMALL LETTER M
U+13000 EGYPTIAN HIEROGLYPH A001
U+1F61C FACE WITH STUCK-OUT TONGUE AND WINKING EYE
And similar with the actual character displayed
U+006D – m – LATIN SMALL LETTER M
U+13000 – 𓀀 – EGYPTIAN HIEROGLYPH A001
U+1F61C – 😜 – FACE WITH STUCK-OUT TONGUE AND WINKING EYE
The first form is often preferable in scenarios where font support might not be present to display the actual character although on this blog I prefer to use the characters to encourage font diversity.
Note the conventional use of hexadecimal to state the value of the Unicode code point. This convention is different to that used in HTML where characters as numeric entities are written using decimal numbers rather than hexadecimal, e.g. 𓀀 (13000 hexadecimal equals 77824 decimal).
From a programming perspective, the simplest way of representing Unicode is UTF-32 where each code point fits comfortably into a 32 bit data structure, e.g. in C# a uint or int (C/C++ programmers note C# defines as 32 bit, the size does not vary with CPU register size). Not entirely trivial because there may still be combining sequences. However UTF-32 is not used all that much in practice, not least because of memory cost.
Nowadays, most files containing Unicode text use UTF-8 encoding. UTF-8 uses 1 byte (octet) to encode the traditional 127 ASCII characters and up to 4 bytes to encode other characters. XML and HTML files are popular file formats that use Unicode (Mandatory in XML, optional in HTML where a surprising amount of the web, possibly 50%, still uses legacy encodings). I strongly recommend UTF-8 for text files rather than UTF-16 or legacy 8-bit encodings aka code pages etc. Having worked on several multilingual content-intensive projects, this is the golden rule, although I won’t expand further today on the whys and wherefores. [However I ought to mention the catch that is the ‘Byte order mark’, a byte sequence (0xEF, 0xBB, 0xBF) sometimes used at the start of a UTF-8 stream to assert UTF-8 not legacy text; this can confuse the novice particularly with ‘.txt’ files which can be Unicode or legacy. Windows Notepad uses BOM for Unicode text files. Visual Studio 2010 also uses BOM to prefix data in many file types including XML, XAML and C# code.]
UTF-16 is very popular with software writers working in C/C++ and .Net languages such as C#. A version of UTF-16 was the standard data format for Unicode 1.0. Unicode characters with character codes less than 0x10000 are said to belong to the Unicode BMP (Basic Multilingual Plane) and these are represented by one 16 bit number in UTF-16, other characters require two 16 bit numbers i.e. two UTF-16 codes from a range that do not encode characters, the so called surrogate code points dedicated to this purpose. As of Unicode 6.0, fewer than 50% of characters belong to the BMP but BMP characters account for a huge proportion of text in practice. This is by design; all popular modern languages have most script/writing system requirements addressed by the BMP and there are even specialist scripts such as Coptic defined here. Processing UTF-16 is often more efficient than UTF-8 and in most cases uses half the memory of UTF-32, all in all a good practical compromise solution.
Which brings me back to the 16-bit myth. The fact that so many popular characters belong to the BMP and only require one code unit in UTF-16 means it is easy to be mistaken into thinking most means all. The problem doesn’t even arise with UTF-8 and UTF-32 but the fact is much software uses UTF-16, indeed UTF-16 is essentially the native text encoding for Windows and .Net.
Example sources of 16-bit confusion:
The article on character sets at http://www.microsoft.com/typography/unicode/cs.htm is brazen:
This article is dated to 1997 but was probably written much earlier. Windows NT 3.1 (1993) was notable as the first computer operating system to use Unicode as its native text encoding and Microsoft deserves credit for this, alongside Apple who also did much to help early uptake of Unicode (but would not have a new operating system until OSX was released in 2001). I’m quoting this as an example of the fact that there are many old documents on the Web, confusing even when from reputable sources. I should mention, in contrast, much of MSDN (and indeed much of the relevant information on Wikipedia) is pretty up to date and reliable although not perfect on this subject.
The definition of the .Net Char structure on MSDN, http://msdn.microsoft.com/en-us/library/system.char.aspx, is much more recent.
Er, no. Char is not a Unicode character. It is a 16 bit Unicode code unit in UTF-16. Actually, this is explained later on in the Char documentation but the headline message is confusing and encourages programmers to use Char inappropriately.
The reasons I chose the Microsoft examples rather than the myriad of other confusing statements on the web are twofold. Firstly I'm focussing on .Net, C# etc. here. Secondly, Microsoft are generally ahead of the game with Unicode compared with other development systems which makes errors stand out more.
Fact is .Net actually works very well for software development with Unicode. The basic classes such as 'String' are Unicode (String is UTF-16) and it is almost true to say it is harder to write legacy than modern.
I had hoped to get a little further on the actual technicalities of working with Unicode characters and avoiding 16-bit pitfalls but time has proved the enemy. Another day.
Just three useful (I hope) points on .Net to conclude.
1. Code that works with String and Char should avoid BMP-thinking, e.g. if you want to parse a String, either avoid tests like IsLetter(Char) or wrap their usage in logic that also handles surrogates.
2. String, Char and the useful StringInfo class belong to the System namespaces and are pretty portable over the gamut of .Net contexts including Silverlight, WPF, XNA as well as the Novell parallel universe with Mono, MonoTouch, Moonlight etc. With a little care it can be straightforward to write text processing code that works across the board to target Windows, Mac, Linux, WP7 and whatever comes next.
3. Always test text-related code with strings that include non-BMP characters, and preferably also with data that includes combining sequences and usage instances of OpenType features such as ligatures.
Thursday, 23 September 2010
Document embedding and OLE
There are a number of technical issues concerned with what I'm attempting to accomplish with the InScribe software. The biggest thorn in my side is the issue of embedding, visualising, and editing embedded data in compound documents. This is therefore something of a background note on the topic.
The notion of embedding or linking an ‘object’ in a document is commonplace. Web pages incorporate pictures, videos and specialized objects such as Flash or Silverlight interactive components. Word processing documents likewise contain objects, sometimes interactive objects, alongside the text. The idea of ‘compound document’ goes back to the 1980s.
The current situation with embedded objects is chaotic. Examples. Microsoft Office and Open Office, the two most popular office suites, have different schemes for add-ins. Even limiting attention to one vendor, Microsoft Office has added features in the 2003, 2007 then 2010 editions: all well and good but makes it difficult to support a diverse user base (Office 2003 is still widely used). Firefox, Chrome, Internet Explorer each has its own plugin approach and the story gets more complicated when considering non-Windows Internet browsers.
We are sorely missing standard, flexible, open approaches to embedding. On the web side of the coin, HTML5 is a move in the right direction though in itself no panacea despite what some less technically minded commentators may say on the subject (a topic for a future blog entry!).
A concrete example. There is no bottom line hardware-related reason nowadays that a simple photo editor plug-in software component could not operate on devices as diverse as eBook readers, smartphones, tablet computers, as well as notebook and desktop computers. Such a plug-in could enrich many applications, not only web browsing but word processors and camera management tools. In our Babel like world of 2010, to accomplish such a thing in software would involve writing versions for iOS, OS X, Android, Windows XP, Windows 7, Symbian, Kindle, Blackberry, Gnome, Wii... the list goes on. The developer then needs to tackle application-specific rules for plug-ins (if such exist). A relatively simple piece of software is almost impossible to deploy widely.
I am not advocating one ring to rule them all, simply highlighting the severe lack of standard ways for applications and components to interoperate and deploy.
In fact, for Microsoft Windows applications there has been one solution for almost 20 years. OLE (Object Linking and embedding) was introduced to Windows in 1990 and expanded substantially to version 2 in 1993. Parts of the OLE system were renamed ActiveX controls in 1996. I’ve exploited OLE with the InScribe software to enable in-place editing of Ancient Egyptian in Word Processing and other applications.
OLE has never been an ideal technology solution. Parts are overcomplicated and error prone during development. OLE design is too specific to classic Windows architecture. There was originally insufficient attention given to security issues although this is now largely addressed. Nevertheless, for Windows applications, OLE provided some solutions to the big problem of making applications and components work together by defining standard rules for interoperability of functions like embedding and compound documents.
Unfortunately, OLE development by Microsoft pretty much stopped well over a decade ago, more a victim of fashion rather than any logical reason I suspect. One side effect is that something as fundamental as how copy and paste works between Windows applications is still stuck in a 1990s time-warp.
As the internet grew and new technologies such as Java and .Net became available, inasmuch as there was any attempt to address application interoperability, solutions tended to be product specific. Microsoft themselves targetted ‘the enterprise’ with less focus on the general personal computer user. Rather than an OLE philosophy where third parties can expand the capabilities of Windows and its applications in a general way, Microsoft Office became focussed on enterprise oriented Office specific add-ons. Linux and other alternatives failed to rise to the challenge of developing a more open and flexible approach.
OLE continues to be supported in Windows and Office, but pretty much maintenance only. OpenOffice and other products continue likewise. I have not looked at the latest Adobe Creative Suite (CS5) but recall an earlier move around CS3 to remove some OLE functionality. Microsoft Office Word 2010 runs OLE embedding in a compatibility mode. I don’t expect OLE to disappear in the next few years but it is certainly becoming less usable.
If anyone reading this understands why the issue of application interoperability and OLE type functionality is missing from .Net and WPF I'd be delighted to find out.
This slow decline of OLE and the lack of practical modern alternatives proved a stumbling block in my development of a new version of the InScribe for Windows software. I expect other developers are in a similar situation of having to make some undesirable compromises in order to get a product released. On a positive note, the problem has stimulated a number of interesting ideas for future directions and I hope to touch on these here during the next few months.
Meanwhile a call for anyone working on document embedding to please learn from the past. Lets try to ensure whatever is latest and greatest at least accomplishes what OLE did 20 years ago (and still almost does).
The notion of embedding or linking an ‘object’ in a document is commonplace. Web pages incorporate pictures, videos and specialized objects such as Flash or Silverlight interactive components. Word processing documents likewise contain objects, sometimes interactive objects, alongside the text. The idea of ‘compound document’ goes back to the 1980s.
The current situation with embedded objects is chaotic. Examples. Microsoft Office and Open Office, the two most popular office suites, have different schemes for add-ins. Even limiting attention to one vendor, Microsoft Office has added features in the 2003, 2007 then 2010 editions: all well and good but makes it difficult to support a diverse user base (Office 2003 is still widely used). Firefox, Chrome, Internet Explorer each has its own plugin approach and the story gets more complicated when considering non-Windows Internet browsers.
We are sorely missing standard, flexible, open approaches to embedding. On the web side of the coin, HTML5 is a move in the right direction though in itself no panacea despite what some less technically minded commentators may say on the subject (a topic for a future blog entry!).
A concrete example. There is no bottom line hardware-related reason nowadays that a simple photo editor plug-in software component could not operate on devices as diverse as eBook readers, smartphones, tablet computers, as well as notebook and desktop computers. Such a plug-in could enrich many applications, not only web browsing but word processors and camera management tools. In our Babel like world of 2010, to accomplish such a thing in software would involve writing versions for iOS, OS X, Android, Windows XP, Windows 7, Symbian, Kindle, Blackberry, Gnome, Wii... the list goes on. The developer then needs to tackle application-specific rules for plug-ins (if such exist). A relatively simple piece of software is almost impossible to deploy widely.
I am not advocating one ring to rule them all, simply highlighting the severe lack of standard ways for applications and components to interoperate and deploy.
In fact, for Microsoft Windows applications there has been one solution for almost 20 years. OLE (Object Linking and embedding) was introduced to Windows in 1990 and expanded substantially to version 2 in 1993. Parts of the OLE system were renamed ActiveX controls in 1996. I’ve exploited OLE with the InScribe software to enable in-place editing of Ancient Egyptian in Word Processing and other applications.
OLE has never been an ideal technology solution. Parts are overcomplicated and error prone during development. OLE design is too specific to classic Windows architecture. There was originally insufficient attention given to security issues although this is now largely addressed. Nevertheless, for Windows applications, OLE provided some solutions to the big problem of making applications and components work together by defining standard rules for interoperability of functions like embedding and compound documents.
Unfortunately, OLE development by Microsoft pretty much stopped well over a decade ago, more a victim of fashion rather than any logical reason I suspect. One side effect is that something as fundamental as how copy and paste works between Windows applications is still stuck in a 1990s time-warp.
As the internet grew and new technologies such as Java and .Net became available, inasmuch as there was any attempt to address application interoperability, solutions tended to be product specific. Microsoft themselves targetted ‘the enterprise’ with less focus on the general personal computer user. Rather than an OLE philosophy where third parties can expand the capabilities of Windows and its applications in a general way, Microsoft Office became focussed on enterprise oriented Office specific add-ons. Linux and other alternatives failed to rise to the challenge of developing a more open and flexible approach.
OLE continues to be supported in Windows and Office, but pretty much maintenance only. OpenOffice and other products continue likewise. I have not looked at the latest Adobe Creative Suite (CS5) but recall an earlier move around CS3 to remove some OLE functionality. Microsoft Office Word 2010 runs OLE embedding in a compatibility mode. I don’t expect OLE to disappear in the next few years but it is certainly becoming less usable.
If anyone reading this understands why the issue of application interoperability and OLE type functionality is missing from .Net and WPF I'd be delighted to find out.
This slow decline of OLE and the lack of practical modern alternatives proved a stumbling block in my development of a new version of the InScribe for Windows software. I expect other developers are in a similar situation of having to make some undesirable compromises in order to get a product released. On a positive note, the problem has stimulated a number of interesting ideas for future directions and I hope to touch on these here during the next few months.
Meanwhile a call for anyone working on document embedding to please learn from the past. Lets try to ensure whatever is latest and greatest at least accomplishes what OLE did 20 years ago (and still almost does).
Saturday, 7 August 2010
Missing in Silverlight 4: a functional GlyphTypeface class
Warning. Obscurity level: HIGH.
This note is primarily aimed at the Silverlight development team in Microsoft Redmond. Other Silverlight developers may also want to understand a limitation of Silverlight 4.
Background
Applications that require superscripts, subscripts, and other rich text functionality need control of character/glyph positional placement. Advanced typography is also useful in applications such as e-book readers where it is often desirable to accurately represent the look and feel of the book. Specialist applications that do mathematical typography (and my ancient Egyptian work) need this kind of precision. From a developer perspective, it is the GlyphTypeface class in the .Net/WPF System.Windows.Media namespace that provides much of required functionality for WPF applications.
The problem
The Silverlight 4 documentation available from Microsoft (see http://msdn.microsoft.com/en-us/library/system.windows.media.glyphtypeface(VS.95).aspx) states
Er no! The WPF 4 version of GlyphTypeface indeed does this. However Silverlight 4 only supports reading the name of the font and its version number. All the useful functionality is missing. The documentation quoted only applies to WPF.
It is therefore impossible in general to implement advanced typography in Silverlight. A big hole - typography has been possible since Windows 3.1, the first release (1992) to incorporate scalable (TrueType) fonts. [Note: sure there are clumsy workarounds in very special circumstances but I won’t go into those today].
The solution
Expand the GlyphTypeface in Silverlight 5 to provide all missing functionality except where this conflicts for some reason with the Silverlight security model. In particular, discovery of the black box for a glyph is essential, as is CharacterToGlyphMap (without which the ‘Glyphs’ class has only limited use). A fairly small amount of straightforward work in the Silverlight runtime yields a big benefit to third party developers and should also help functional enhancement to controls such as RichTextBox.
Note: Windows Phone 7 is also lacking functionality here.
This note is primarily aimed at the Silverlight development team in Microsoft Redmond. Other Silverlight developers may also want to understand a limitation of Silverlight 4.
Background
Applications that require superscripts, subscripts, and other rich text functionality need control of character/glyph positional placement. Advanced typography is also useful in applications such as e-book readers where it is often desirable to accurately represent the look and feel of the book. Specialist applications that do mathematical typography (and my ancient Egyptian work) need this kind of precision. From a developer perspective, it is the GlyphTypeface class in the .Net/WPF System.Windows.Media namespace that provides much of required functionality for WPF applications.
The problem
The Silverlight 4 documentation available from Microsoft (see http://msdn.microsoft.com/en-us/library/system.windows.media.glyphtypeface(VS.95).aspx) states
| “The GlyphTypeface object is a low-level text object that corresponds to a single face of a font family as represented by an OpenType font file, or serialized as a block of memory in a document. Each glyph defines metrics that specify how it aligns with other glyphs. The correct GlyphTypeface to use for a run of characters in a given logical font is normally determined by the Silverlight font system. The GlyphTypeface object provides properties and methods for the following: · Obtaining font face common metrics, such as the ratio of ascent and descent to em size. · Obtaining metrics, outlines, and bitmaps for individual glyphs.” |
Er no! The WPF 4 version of GlyphTypeface indeed does this. However Silverlight 4 only supports reading the name of the font and its version number. All the useful functionality is missing. The documentation quoted only applies to WPF.
It is therefore impossible in general to implement advanced typography in Silverlight. A big hole - typography has been possible since Windows 3.1, the first release (1992) to incorporate scalable (TrueType) fonts. [Note: sure there are clumsy workarounds in very special circumstances but I won’t go into those today].
The solution
Expand the GlyphTypeface in Silverlight 5 to provide all missing functionality except where this conflicts for some reason with the Silverlight security model. In particular, discovery of the black box for a glyph is essential, as is CharacterToGlyphMap (without which the ‘Glyphs’ class has only limited use). A fairly small amount of straightforward work in the Silverlight runtime yields a big benefit to third party developers and should also help functional enhancement to controls such as RichTextBox.
Note: Windows Phone 7 is also lacking functionality here.
Tuesday, 3 August 2010
Ace Computers (Pilot and Jupiter)
In the interests of the online documentation of obscure connections.
Several years ago, I attended a celebration of the life of Alan Turing (1912-1954) at King's College Cambridge. Turing is recognized nowadays as an influential pioneer in the history of Computer Science and Computers. His key contributions to British code breaking work at Bletchley Park during World War II have historical significance. At the King's event, it was fascinating to meet several of the people who had worked with Turing; a reminder of how young computer technology really is.
This recollection came to mind recently while reading an article (How Alan Turing's Pilot ACE changed computing) on the BBC website. The Pilot Ace was an early computer designed by Turing and developed at the British National Physical Laboratory (NPL) from 1946 to its release in 1950. For several years, the Pilot Ace was used for commercial applications and can now be seen in the London Science Museum. The BBC article refers to a radio interview with Tom Vickers, operations manager on the Pilot Ace project (As of writing, the interview is still available via Harriet Vickers blog where the interview starts 11 minutes into the podcast).
An early home computer called the Jupiter Ace was released in 1982, an untypical device of its time based around the FORTH programming system, an approach that yielded a little more efficiency than similar machines of that era which were mostly programmed using interpreted BASIC.
Computing technology had advanced considerably since the era of the Pilot Ace 30 years earlier but the two devices in their own way illustrate the challenge of trying to work with hardware not quite ready for prime time yet interesting nevertheless. A basic Jupiter Ace came with 1024 bytes (1K) of main memory, expandable to 49x1024 bytes (49K). The Pilot Ace originally had 512 bytes main memory, later expanded to 1408 bytes (implemented using vacuum tubes!) with a 16K byte drum memory peripheral.
Now for the obscure part. The Jupiter Ace ROM software was written by Steve Vickers. Steve had already written much of the ROM software for the Sinclair ZX home computers, popular hobbyist type machines in the UK of the early 1980s. The name "Jupiter Ace" was inspired by the work of his father Tom Vickers on the Pilot Ace. Generations.
It is now 60 years since the Pilot Ace release. Almost 30 since the Jupiter Ace and the computing landscape has changed far more dramatically in the last three decades than in the first three following Turing's work. A myriad of observations could be made but I'll simply note that commonplace telephones nowadays have a billion times more memory than the Pilot, and hundreds of thousands more than the Jupiter.
Perhaps it is time for another Ace computer.
Several years ago, I attended a celebration of the life of Alan Turing (1912-1954) at King's College Cambridge. Turing is recognized nowadays as an influential pioneer in the history of Computer Science and Computers. His key contributions to British code breaking work at Bletchley Park during World War II have historical significance. At the King's event, it was fascinating to meet several of the people who had worked with Turing; a reminder of how young computer technology really is.
This recollection came to mind recently while reading an article (How Alan Turing's Pilot ACE changed computing) on the BBC website. The Pilot Ace was an early computer designed by Turing and developed at the British National Physical Laboratory (NPL) from 1946 to its release in 1950. For several years, the Pilot Ace was used for commercial applications and can now be seen in the London Science Museum. The BBC article refers to a radio interview with Tom Vickers, operations manager on the Pilot Ace project (As of writing, the interview is still available via Harriet Vickers blog where the interview starts 11 minutes into the podcast).
An early home computer called the Jupiter Ace was released in 1982, an untypical device of its time based around the FORTH programming system, an approach that yielded a little more efficiency than similar machines of that era which were mostly programmed using interpreted BASIC.
Computing technology had advanced considerably since the era of the Pilot Ace 30 years earlier but the two devices in their own way illustrate the challenge of trying to work with hardware not quite ready for prime time yet interesting nevertheless. A basic Jupiter Ace came with 1024 bytes (1K) of main memory, expandable to 49x1024 bytes (49K). The Pilot Ace originally had 512 bytes main memory, later expanded to 1408 bytes (implemented using vacuum tubes!) with a 16K byte drum memory peripheral.
Now for the obscure part. The Jupiter Ace ROM software was written by Steve Vickers. Steve had already written much of the ROM software for the Sinclair ZX home computers, popular hobbyist type machines in the UK of the early 1980s. The name "Jupiter Ace" was inspired by the work of his father Tom Vickers on the Pilot Ace. Generations.
It is now 60 years since the Pilot Ace release. Almost 30 since the Jupiter Ace and the computing landscape has changed far more dramatically in the last three decades than in the first three following Turing's work. A myriad of observations could be made but I'll simply note that commonplace telephones nowadays have a billion times more memory than the Pilot, and hundreds of thousands more than the Jupiter.
Perhaps it is time for another Ace computer.
Thursday, 17 June 2010
C++ vs. C# 2010 ... and the winner is ...
As far as I know there is no World Cup for programming languages, quadrennial or otherwise. No FIFA++ or FIFA# either. But, if there was...
I have been helping for several months with a project aiming to commercialise an optimising technique for microelectronic circuits discovered as part of some academic research at a local university. Who would predict that the mathematics of Galois polynomials would have an industrial application?! Well, this is my journal of total obscurity.
Writing the software. Apart from user interface, visualisation and usability all good stuff, a key element of the project is of course the algorithms, some of which require a lot of computational processing by current standards and can be time consuming (potentially days rather than hours or minutes).
The original research software used a C++ with STL implementation and much of the time I continued using C++ for algorithm work as an adaption of the original despite the fact I’m far more C# oriented nowadays. Mental exercise is a good thing. My Windows software design was a hybrid with the user interface and control functionality in C#/.Net/WPF with algorithms packaged in a DLL implemented in C++. Standard .Net interop linked the two components together.
I suspect most developers with strong experience in C++ and C# will agree that a modern language such as C# can be far more readable than C++. However a common concern is the efficiency of C# on the .Net and Mono environments using the Common Library Runtime (CLR) compared with C++ static compilation into native machine code. There is no question that C and C++ work closer to the metal and their compilers generate very efficient code in the right hands (C has been around since 1973 and C++ since 1983). The C/C++ combination remains the most efficient programming choice for some applications.
Question. What happened when I rewrote a key algorithm from C++ into C# using the same logic?
Answer. Performance improved by an order of magnitude. In fact my initial benchmarks were so close to exactly 10x faster I felt obliged to check and recheck. Memory usage for the C# version was about 5% less than C++ for this specific problem.
[In both instances C++ and C# I used Microsoft Visual Studio 2010 and .Net 4 runtimes but have since replicated similar figures on Linux using GNU CPP for C++ and MonoDevelop 2.2 with Mono 2.6.4 for C#.].
Understanding the why. Partly down to inefficiencies in the C++ heap and STL templates - software uses hashset/unordered_map/vector objects. Partly benefits of automatic lifecycle management of C# objects with a built in garbage collector. A little benefit from C# language features such as the foreach construction.
OK, I’m sure that by adjusting algorithms and replacing STL constructs I could bring C++ up to a similar or higher speed than the C# version. Plenty can be said on that topic. But life is too short for such antics especially when the C# version is more concise and readable that a C++ version could ever be.
Back in 2006 Italy lifted the trophy, C# and CLR implementations were not as mature, C++ held sway. Times change.
Now, Summer 2010, to my mind C# takes the top of the podium. Weighing the evidence, fact not opinion. Performance and elegance, it is so often a trade off but this, my recent experiment, sealed the question for me.
Postscript. I wrote this note to encourage other developers to look again at the rationale of choosing C++ over a modern programming language even for computationally intensive tasks. The usual caveats. Especially 1. existing systems where language change is impractical and interop inappropriate and 2. the limited class of problem where C++ still rules. Personally speaking. I'll continue to choose C or C++ where there is a clear benefit but the problem domain where this is true seems to have shrunk dramatically.
I have been helping for several months with a project aiming to commercialise an optimising technique for microelectronic circuits discovered as part of some academic research at a local university. Who would predict that the mathematics of Galois polynomials would have an industrial application?! Well, this is my journal of total obscurity.
Writing the software. Apart from user interface, visualisation and usability all good stuff, a key element of the project is of course the algorithms, some of which require a lot of computational processing by current standards and can be time consuming (potentially days rather than hours or minutes).
The original research software used a C++ with STL implementation and much of the time I continued using C++ for algorithm work as an adaption of the original despite the fact I’m far more C# oriented nowadays. Mental exercise is a good thing. My Windows software design was a hybrid with the user interface and control functionality in C#/.Net/WPF with algorithms packaged in a DLL implemented in C++. Standard .Net interop linked the two components together.
I suspect most developers with strong experience in C++ and C# will agree that a modern language such as C# can be far more readable than C++. However a common concern is the efficiency of C# on the .Net and Mono environments using the Common Library Runtime (CLR) compared with C++ static compilation into native machine code. There is no question that C and C++ work closer to the metal and their compilers generate very efficient code in the right hands (C has been around since 1973 and C++ since 1983). The C/C++ combination remains the most efficient programming choice for some applications.
Question. What happened when I rewrote a key algorithm from C++ into C# using the same logic?
Answer. Performance improved by an order of magnitude. In fact my initial benchmarks were so close to exactly 10x faster I felt obliged to check and recheck. Memory usage for the C# version was about 5% less than C++ for this specific problem.
[In both instances C++ and C# I used Microsoft Visual Studio 2010 and .Net 4 runtimes but have since replicated similar figures on Linux using GNU CPP for C++ and MonoDevelop 2.2 with Mono 2.6.4 for C#.].
Understanding the why. Partly down to inefficiencies in the C++ heap and STL templates - software uses hashset/unordered_map/vector objects. Partly benefits of automatic lifecycle management of C# objects with a built in garbage collector. A little benefit from C# language features such as the foreach construction.
OK, I’m sure that by adjusting algorithms and replacing STL constructs I could bring C++ up to a similar or higher speed than the C# version. Plenty can be said on that topic. But life is too short for such antics especially when the C# version is more concise and readable that a C++ version could ever be.
Back in 2006 Italy lifted the trophy, C# and CLR implementations were not as mature, C++ held sway. Times change.
Now, Summer 2010, to my mind C# takes the top of the podium. Weighing the evidence, fact not opinion. Performance and elegance, it is so often a trade off but this, my recent experiment, sealed the question for me.
Postscript. I wrote this note to encourage other developers to look again at the rationale of choosing C++ over a modern programming language even for computationally intensive tasks. The usual caveats. Especially 1. existing systems where language change is impractical and interop inappropriate and 2. the limited class of problem where C++ still rules. Personally speaking. I'll continue to choose C or C++ where there is a clear benefit but the problem domain where this is true seems to have shrunk dramatically.
Subscribe to:
Posts (Atom)