Further remarks about portability

[Kent M Pitman <KMP@SCRC-STONY-BROOK.ARPA>]

This message contains replies to and remarks about several of the messages
which have crossed the list since my message a couple of days ago...

    Date: Thu, 22 May 86 12:44:25+0900
    From: yuasa@kurims.kurims.kyoto-u.junet

    Having heard of the problems on the LISP package extensions, I got 
    tempted to add the following constant into our system.

    THOSE-SYMBOLS-THAT-ARE-ADDED-IN-THE-LISP-PACKAGE             [Constant]

    ... (The name does not matter.) With this constant, implementors of 
    portable code could do the shadowing easily.

       (SHADOW THOSE-SYMBOLS-THAT-ARE-ADDED-IN-THE-LISP-PACKAGE)

    I know this would not solve all the problem, but it is better than nothing.
    At least, Gregor's PCL code might become clearer, if all CL implementations
    support this kind of constant.  Any comments?

If you implement such a thing without the rest of us standardizing on it, it's 
of little value since portable code won't know to look for it. I mention this
only because you sounded like your message sounds like you're looking for an
interim solution.

Also, -all- portable programs would have to begin with:
       (SHADOW THOSE-SYMBOLS-THAT-ARE-ADDED-IN-THE-LISP-PACKAGE)
because they wouldn't stand a real chance of running correctly without it. This
seems a little silly since the point of CL was not to agree on a package that was
incompatible and then to patch it with a way to get around that agreed-upon
incompatibility. The point of CL was to give users who cared about portability
something they could cling to so they don't get swept away by the whims of
designers of specific implementations. If we don't give people at least that,
we're not doing very much for them.

I think it was Jonathan Rees who pointed out that the problem is more severe
because the home package of a symbol severely affects the portability of code
which has been read by READ and printed by PRINT in a given implementation.
This problem fails to be addressed by your shadow technique. Consider the 
following program:

 (IN-PACKAGE 'PROCESSOR)
 (USE-PACKAGE 'LISP)
 (SHADOW THOSE-SYMBOLS-THAT-ARE-ADDED-IN-THE-LISP-PACKAGE)

 (DEFUN PROCESS-FILE (INPUT-FILE OUTPUT-FILE)
   (LET ((*PACKAGE* (FIND-PACKAGE "PROCESSOR")))
     (WITH-OPEN-FILE (INPUT-STREAM INPUT-FILE :DIRECTION :INPUT)
       (WITH-OPEN-FILE (OUTPUT-STREAM OUTPUT-FILE :DIRECTION :OUTPUT)
         (PRINT (READ INPUT-STREAM) OUTPUT-STREAM)))))

and a file A.LISP which contains the expression:

 (FOO LISP:FOO)

If (PROCESS-FILE "A.LISP" "B.LISP") is done, then B.LISP is not portable if
the LISP package can vary its contents because the output may contain either
(FOO LISP:FOO) or (FOO FOO) depending upon whether FOO was on the LISP package
and therefore got shadowed in this implementation. In another implementation,
different results will be produced. If the results of such processing are 
shared back and forth between different implementations (as I could easily
imagine wanting to happen in Macsyma's save files, for example), the results
could be disastrous.

Finally, I'd like to remark that I couldn't help getting a chuckle out of 
the following message from John Diamant.  He's obviously in agreement with me
on this, but unfortunately (as I've learned the hard way), the full magnitude
of the problem is worse than he describes it...

    Date: Wed, 21 May 86 14:03:49 pdt
    From: hpfclp!diamant@hplabs.ARPA
    
    ... The problem with making the default environment the extended one is that it
    is IMPOSSIBLE to write portable programs!  If implementation X makes an
    incompatible extension to COMPILE, and I want the CL-only environment so
    I can use the CLtL version of COMPILE, I must say something like:
    
    #+X (progn
	  (unuse-package 'lisp)
	  (use-package 'cl-only))  ; or some other non-standard package name
    
    I realize that someday we may agree on the name of the this other package, but
    that isn't an upward compatible change.  It requires everyone trying to write
    portable code based on the first edition of CLtL to change their code.  The
    piece of code above has made the inherently portable program no longer
    portable (since if I didn't know about implementation X, I couldn't have
    written it to run on implementation X). 

Not only is all this too true, but even worse, we don't even require
implementations to put any distinguishing feature in their implementation. Eg,
VAXLISP 1.2 comes up with a features list that includes things like VAX, VMS,
etc. but nothing that says DIGITAL or DEC or anything. The result is that if I
knew someone else (eg, LUCID) had implemented a lisp that ran under VAX VMS,
I'd not be able to tell from the features list whose Lisp I was running under.
To further aggravate things, we don't specify what package things in the
features list should be in, so while on the LispM you do 
(PUSH ':FOO LISP:*FEATURES*)  and later #+FOO in any package works, in VAXLISP 
you have to do (PUSH 'WHATEVER:FOO LISP:*FEATURES*) and #+FOO only works in 
the WHATEVER package. (I assume #+WHATEVER:FOO works elsewhere but have never
actually constructed the test. Perhaps I shouldn't be so sure, though, since
#+3600 and #-3600 don't do the obvious thing -- they both get read errors.
No reason they shouldn't since CLtL doesn't say they'll work.)

My point is that we failed to standardize just enough even on #+ and #- so that
it's hard even when you've bitten the bullet and decided to just write
implementation specific code in some places. This is a terrible shame. In
retrospect, I think that if there's one thing we should have gotten right in
the first release of CL, it was #+ and #- so that people would have a foothold
when other things didn't work.

Right along side that should have been to make the LISP package contain all and
only the things mentioned in CLtL. There just isn't a substitute for this.
Anything less just screws those people who care about portability.

It's silly to say that there shouldn't be a standard package which contains
only CL stuff because allowing that to happen lets open the situation where I
could create a system which took all symbols having n letters or less which
were not defined in CLtL and defined them to be system constants that couldn't
be assigned, bound, etc. and which had completely silly values. Would we want
to admit that such a language was a Common Lisp? Fahlman is arguing that we
should.

It's also silly to say that the point of CL was to define a lot of interesting
meanings but finally not to put them in some known place, which is effectively
what Fahlman is suggesting.

I'm not trying to cast blame on anyone about past decisions. Most of us had the 
power to speak up early on and failed to notice the impending problem. As they say,
hindsight is always 20/20. If we made mistakes, they were arguably not intentional.
CL is obviously a highly ambitious experiment in portability. There were so many
things to deal with the first time around that it wasn't possible to know what was
going to matter in the end. But now we're at the crossroads to the future with
time our side and better access to experience and knowledge than we had the last
time around. If we make any new CL spec and it doesn't correct these problems,
lack of proper experience and time isn't going to be so good a defense. Users are
going to think that either we don't care about portability or that we don't 
understand it. I don't know which I think would be worse.