Function streams

[Daniel L. Weinreb <dlw at SCRC-TENEX at MIT-ML>]

Having had a great deal of experience with streams, the stream protocol,
and the issues of inheritance and sharing for streams, I can assure you
that without some kind of inheritance, it is very hard to create a
stream protocol that is at all satisying.  As a simple example, it is
nice to allow a FORCE-OUTPUT message to be handled by some streams, but
if someone wants to implement a simple stream with no buffering, he has
to explicitly implement FORCE-OUTPUT as a null method.  This makes it
intolerably difficult to write a very simple stream.

In the early days of the Lisp Machine, we had a function called
STREAM-DEFAULT-HANDLER that any stream could call with an unrecognized
message; the stream would pass ITSELF in as an argument, so that there
could be a STRING-OUT operation.  The default handler for the STRING-OUT
operation would just call the TYO operation for each character in the
string, but any stream that wanted to have hairy, efficient multi-char
output could handle STRING-OUT itself.  This mechanism was a very simple
form of inheritance, with only one superclass in the world, and the only
form of method-combination being shadowing.  This predates flavors or
even the earlier Lisp Machine class mechanism; the default handler was
just a function that stream functions all called when given an unknown
keyword.

This worked pretty well for us, for a while.  One problem was that all
stream functions had lambda-lists like (MESSAGE &OPTIONAL ARG1 ARG2 ...)
because the meanings of ARG1 and ARG2 depended on the value of the
message.  ARG1 and ARG2 are pretty unclear names.  The immediate
solution to this problem was the implementation of DEFSELECT (see the
Lisp Machine manual).  (Also, DEFSELECT produced functions that did the
big dispatch with a microcode assist, but that is not relevant to this
discussion.)

However, as we wrote more advanced I/O software, this mechanism soon
showed itself as being quite deficient.  Each stream that wanted to
support hairy multi-char output and buffered input and so on would have
to implement those concepts itself.  It turns out that it is not very
easy to write code that can accept an input buffer full of text and dish
it out in pieces of the right size.  This code got implemented many
different times by different people, in order to implement editor buffer
streams, file system streams, streams to DMA devices, network streams,
and so on.  Most of the implementations had subtle fencepost errors, and
many of them did not implement QUITE the same protocol regarding what to
do at the end-of-file (even though the protocol was completely
documented).

To deal with this problem, we came up with a set of flavors that
implemented all this stuff in one place: very efficiently, and without
bugs.  Now, if you want to write a stream that is capable of doing
STRING-OUT and LINE-IN and LINE-OUT operations, all you have to do is
create a new flavor, teach it how to handle a very small number of
messages (get next input buffer, advance input pointer, here is an
output buffer, etc.).  The flavors provide all of the actual
stream-protocol interfaces such as TYI and TYO and STRING-OUT and
LINE-IN and FORCE-OUTPUT, by sending the stream these few messages that
you provide.  Everything in the system was changed to use these flavors,
and a whole class of bugs finally vanished.  (This is what Moon was
referring to when he mentioned the STREAM flavors in his earlier
message.)

(By the way, I don't think that these flavors make much use of :BEFORE
and :AFTER daemons (except the ASCII-TRANSLATING-MIXINs, which have
daemons to translate between character sets), but they do use
non-hierarchical inheritance.  How anybody can get useful work done when
restricted to hierarchical inheritance is beyond me; the world just
doesn't work hierarchically.  But anyway.)

The point of all of this is that non-trivial message receiving is needed
if you really want to make a good I/O system.

Now, one thing to keep in mind about your proposal is that all it
discusses is a message sending protocol.  To wit, it says that messages
are sent to an object by applying that object to a symbol that specifies
the message name, and the rest of the arguments to the method.  It
doesn't say anything about how someone might receive such a message.  In
Common Lisp, usually you'd have to create a closure over a function that
has a big CASE-like construct on its first argument and had arguments
named ARG1 and ARG2 and so on; this is what we used to do many years ago
in the Lisp Machine, and it does work even though it is not as easy and
elegant as what we hvae now.  However, any particular Common Lisp
implementation could choose to provide more advanced message receiving
facilities, such as DEFSELECT, classes, or flavors.

In fact, this is the way that message-sending is currently defined to
work in the Lisp Machine.  We are planning to extend that definition
someday, so that it would be possible to send messages to primitive
objects (numbers, symbols) as well as to instances of flavors.  To that
end, we have stopped using FUNCALL to send messages and now use SEND.
For the time being, SEND just does FUNCALL, but someday it will be made
more clever and it will be able to deal with non-functions as its first
argument.

Because of this, if you put in your stream proposal, it would be somewhat
nicer for us if it were defined to call SEND to send the message, and
Common Lisp SEND were defined to be like FUNCALL when the argument is
a function and be undefined otherwise.

The other problem with the proposal is that we already have a very
similar mechanism but it uses different names for the operations.  In
particular, all of the operation names are keyword symbols.  Typical
symbols are :TYI, :UNTYI, :TYO, :STRING-OUT, :LINE-IN, :LINE-OUT,
;FORCE-OUTPUT, :CLOSE, :CLEAR-OUTPUT, :CLEAR-INPUT, and :TYI-NO-HANG.
It would be a shame if we had to implement two I/O systems, one for
Common Lisp and one for internal use.  It would be a lot of work to
completely change all of our message names, and it would not fit into
the rest of our system nicely if they were not keywords.  (Of course,
adding :INCH and :OUCH to our system is very easy, since we just add
them to one particular flavor and everybody automaticaly gets them.)

It might possibly be better to just forget about the whole thing as far
as the white pages are concerned, though.  I'm not sure that this
ability is really important for definition of portable software, and I
think it might be a lot of work to figure out how to design this in such
a way that we'll all be happy with it.