More about packages, from DLW and BSG

[Daniel L. Weinreb <dlw%SCRC-TENEX%MIT-MC@SU-DSN>]

Actual problems with the proposal:

In your example, have you considered what would happen if you were to
run the Lisp compiler on Newton's init file?  As far as I can tell, the
compiler would signal an error upon seeing the reference to the symbol
"relativity:speed-of-light", because that package would not exist, at
compile time.  You need an eval-when around the load form.

As the proposal is written, use-package will consider its arguments from
left-to-right, and pushnew each one.  This will effectively reverse the
order of the list, which seems confusing.

If you import an uninterned symbol, or otherwise cause a symbol to exist
that is present in some package but has no home package, the rules for
how to print the symbol (assume that package is the current package)
both say that you should print it without any qualifier and that you
should print it with a leading #:.  Presumably the rule for how to print
symbols should be amended to say that it only applies to interned
symbols (those with a home package).  The consistency rules are not
violated, since they are careful to only talk about interned symbols.

If an interned symbol is removed from its home package then the
consistency rules will be violated (the symbol might be printed as
foo:bar or foo#:bar even though it's not in that package).  I don't know
what to do about this.

Places where the proposal is unclear:

There is a fundamental question about the semantics of packages whose
answer I was not sure of until I had read the proposal most carefully,
namely, what does it mean for a symbol to be "external"?  The real
answer is that a given symbol can be interned in many packages, and in
each of these packages it can either be internal or external.  You can
only ask whether a symbol is external in a certain package, not whether
it is external.  Let me assure you that this point is not at all obvious
unless you read the proposal extremely carefully, and it caused us some
confusion at various points.  The first paragraph about external versus
internal gives you the impression that it would be meaningful to have a
function called externalp that just takes a symbol and returns a boolean
value, by its use of language.

An explicit definition of the algorithm for looking up an unqualified
name was given, but no corresponding definition was given of the
algorithm for looking up a name qualified with a colon.  Suppose we are
looking up the string "FOO:BAR", and FOO has an internal symbol named
BAR, and FOO inherits from FIE, and FIE has an external symbol named
BAR.  Does it return the external BAR symbol in FIE, or does it signal
an error because the symbol that it finds isn't external?  It is not
until the function find-external-symbol is defined that we are actually
told the answer to this question (the former).  I had initially assumed
that "FOO:BAR" follows the standard lookup algorithm, and then signals
an error if it finds that the symbol it has found is not an "external
symbol".  (Of course, there's no such thing as being an external symbol,
but I still wasn't sure of that.)  This point also needs more
clarification earlier in the description, possibly by putting in a
separate algorithm just to make things completely clear.  (In
particular, the wording "refers to the external symbol named @b[buffer]
in the package named @b[editor]" is what threw me off most severely.  I
asked myself "what if @b[buffer] is an internal symbol?" and it wasn't
clear what the answer was.  The phrasing suffers from "present King of
France" syndrome.)

I am also unhappy with the sentence in the description of use-package
that says "These symbols become available as @i[internal] symbols".
This is non-strict phrasing, since the term "available" is only defined
in such a way that you can ask whether a symbol is available or not; you
can't ask whether it's available "as" an internal symbol.  The sentence
as phrased deserves to be followed by "so to speak"; it isn't literally
meaningful, and gives me the impression that they "become internal
symbols of the package", which they don't.

The declare-package/error-checking issue:

The statement that "no two symbols of the same name may be accessible
from the same package" is trivially true, from the definition of
"accessible"; after all, find-symbol can only return one symbol.  What
Moon is really saying here, I presume, is that no symbol should shadow
any other symbol unless some explicit mechanism is used that says "hey,
I'm really trying to shadow something".  Moon said elsewhere that "the
order of searching ancestors cannot be allowed to make a difference if
we are to remain sane", which implies the same thing, more or less.  I
presume he's saying that every operation that might set up an implicit
shadowing must be error-checked to make sure that it doesn't do so.

This clearly means that the concept of "shadowed" must be added to the
proposal and defined clearly.  I would presume that shadowing, like
external/internal, is a property of an "interning"; that is, a symbol
can be interned in several packages, and in each package it might or
might not be "shadowing".  Someone must propose what functions exist to
cause an interning to be shadowing, go through some examples of how
you'd use shadowing, and figure out the algorithm for checking for name
conflicts whenever use-package, inherit-package, export, or other
functions are called given that some internings are shadowing and some
are not.  I think Dill's example (the P1, S1 one) shows that the
algorithm is not immediately obvious, particularly when shadowing is
involved.  Until then it's hard to judge whether it all works or not.
By the way, I also noticed the problem of "when do you know that you're
finished loading?".

My analysis of Moon's BLORPH example is that the situation that the
various participants have asked for is that there should be one symbol,
BLORPH, external in the A package, and packages B and C should be using
this symbol.  If BLORPH is created and exported in A before B and C are
loaded, this happens, and otherwise an error gets signalled by Moon's
proposed error-checking stuff.  The existing Zetalisp package system
deals with this by "locking" any package that is used by any other
package; i.e. it never lets you add a new symbol into any ancestor
package.  In either case, Moon's scenario is not really a "screw case",
since there is only one situation that can occur without an error being
signalled.  The official screw cases are where two different things can
happen without any error being signalled.  Moon's example shows how the
current proposal can create this kind of screw.  SEF's comment about
"the old LispM package system, which sometimes does seem like it's out
to screw people in all possible ways" is interesting in this light.

In Dill's new proposal, the same problem is solved by using "copy
semantics".  If you export a symbol in P2 after P1 asked to use P2, then
P1 just doesn't get the new symbol.  So ancestors have to get set up
before they're used.  This is just like the Zetalisp philosophy, except
that you don't get an error message if you do things out of order
(because in the new definition, it's not an error).  The order dependence
is still there.

In the Zetalisp package system, we did what we considered the simple and
safe thing.  We stated at the outset that every package must be declared
before it is used, and the declaration sets up all relevant information
about the package, saying what it inherits from, exports, shadows, etc.
The package declaration must be evaluated (i.e. loaded) before anything
else can even THINK about using the package.

Enough rambling: the major decision to be made at this juncture is
whether we want to try to let you use a package before its owner has
declared it, or whether we want to make sure that the owner declares a
package before anybody tries to use it.  The present proposal attempts
to do the former.  One problem has been pointed out already (by Moon):
declaring a package that is constructed by inclusion of other packages
requires the the declarer know about this structure.  I.e. if I want to
use package foo, and package foo inherits from package bar, I have to
know that.  This is clearly unmodular.  Here's something even worse
about declaring a package that includes some other package.  Consider
(declare-package "foo"
        :include "bar
        :export '("a" "b" "c"))
Are "a", "b", and "c" interned in "foo" or "bar"?  The syntax of
declare-package has to be extended to allow the caller to say so,
explicitly.  Are you grossed out yet?

I think the current favorite counterproposal around here is to say that
for every package, there is a file that sets up that package, specifying
what it uses and includes and exports, and programs that want to access
those packages would have something like Dill's load-package function to
read in that file if it doesn't exist already.  For small packages, the
entire program that makes up that package could be in the file; for
large packages, you have to get the rest of the files loaded yourself,
but this one file has all the relevant forms that set up the package
structure.