Why are char bits portable?

[Kent M Pitman <KMP@SCRC-STONY-BROOK.ARPA>]

[This message was sent last Friday but got returned due to some 
 network problems. Apologies to anyone who gets duplicate copies.]

    Date: 19 Jun 86 12:32 PDT
    From: Masinter.pa@Xerox.COM
    Subject: Re: Why aren't char bits portable?
    To: common-lisp@su-ai.ARPA
    In-Reply-to: KMP%SCRC-STONY-BROOK:ARPA's message of Monday, June 9, 1986
     12:23 pm 
    Message-ID: <860619-123848-1153@Xerox>

    Char-bits aren't portable because they are a property of the physical
    keyboard of the machine, and, most keyboards don't have bits that
    correspond to them. ("Most" when enumerating all keyboards, computer
    keyboards, keyboards on machines that run lisp, or even accepted
    standards (DIN, ANSI) for keyboards.)

    The best thing that a "portable" program can do to cope with the variety
    of keyboards is twofold:

    a) stick as much as possible within the set of standard characters

I assume the Japanese and IBM folks monitoring this discussion got a good
chuckle out of this one. I'm not sure what you mean by the set of standard
characters. Much of the CL spec is explicitly vague on the issue in order
to not preclude major changes in either character set or number of shift
chars provided.

    b) mark the mapping of keys to characters as a clear
    implementation-dependent part of programs that have to go beyond the
    standard characters.

It's important to distinguish between implementation dependent behavior
and unpredictable behavior. Many things in Common Lisp are conceptually
well-defined even though they vary from implementation to implementation.
For example, the behavior of (1+ MOST-POSITIVE-FIXNUM) varies in detail 
from implementation to implementation, but conceptually it does not. I 
don't think that it's inappropriate to describe a program as having 
useful commands bound to keys and to tell the user to type "?" to see a
list of the commands available. 

You may have to do this for more reasons than just char bits. Maybe some
commands won't turn themselves on for other reasons. Eg, maybe some 
setup code will not enable a certain command because MOST-POSITIVE-FIXNUM 
is too small, or SINGLE-FLOAT-EPSILON is too big, or MACHINE-INSTANCE
returns NIL, or whatever. 

The issue is much bigger than you make it out to be and if you claim it
to be fatal, then I claim you're attacking the whole notion of whether
CL can be portable... and I don't know if CL programs can ever be truly 
portable, but I believe we can fix the language such that they can be
usefully portable.

    Char-bits have no more place in the "standard" than does
    double-bucky-coke-bottle.

[For those who don't know the term, a bucky bit is a generalized shift bit.
Control, Meta, Super, and Hyper are examples of such bits.]

This just isn't so. ITS Teco and ITS Emacs (which was ported to TOPS-20)
uses char bits quite portably, even though huge numbers of Emacs users 
have only ASCII keyboards. The compatibility there works remarkably 
gracefully. I had written a ton of programs which presumed that there
was a control bit. Teco's reader just took care of knowing what kind of 
keyboard users were on, automatically translating ascii 001 to an A with 
the Control bit set, etc. In the case of ;, which cannot be controllified
in ASCII, Emacs provided c-^ as an prefix which would put a control bit
on the next char typed and pass that through. That way, I could type
c-^ A and get a Control-A.

I also note that I use my Lisp Machine from home (on an Ambassador terminal)
and it does exactly the same trick. There's a front end processor which
takes c-^ char into c-char, ESC char into m-char, c-] char into s-char, etc.
It even lets me type Resume, Abort, End etc. by c-_ R, c-_ A, c-_ E, etc.
The point is that Lisp itself manages this, and any program I write sees
"normal" chars with bucky bits properly attached.

CL makes no statements about keyboards at all. There's nothing to say that
you don't have a tablet or voice input device managing your I/O. All we
say is that there are functions which read characters from streams. How
those characters end up being placed on the stream is pretty much left
up to the particular implementation.

So I just don't believe this argument.

As long as users of a portable program can get to all the relevant 
commands, they don't tend to complain that their keyboard (which perhaps
has no META key) cannot type META-X. They just don't use META-X -- 
presumably they get the same functionality some other way. But users
who have a META key -do- complain when there's functionality available
and a free key that the functionality could be attached to. The current
CL strategy for bucky bits allows me the ability to deal with exactly
this problem -- attaching things to chars when they're available and not
when they're not. Why would you want to deny me the opportunity to do that?
Presumably if it wouldn't be a natural thing on your machine, then you
just don't implement the bits for your implementation, and my programs
will never try to use them, and your users won't think to complain 
because they won't have funny shift keys that suggest they -should-
be using them.

Also, the presence of bucky bits in CL may help encourage some keyboard 
vendors to add bits. If we say "let's not use these bits because no 
keyboard vendors provide them", then I fear that at their next design
meeting, they'll say "let's not provide these bits because no languages
seem to use them". Someone's got to take the first step. We already
have. I see no reason to back off on it if people (eg, me) are using
it productively.