One cool thing about Lisp is that you can easily embed new languages in it, and those languages can be small and beautiful. For example, I have a Python-esque FOR macro that uses an iterator protocol, and a universal binding macro that subsumes all of Common Lisp's binding constructs (LET, LET*, LABELS, FLET, MULTIPLE-VALUE-BIND, etc.) So for me, Common Lisp has actually shrunk without losing any functionality. This is not possible in languages without macros. Such languages are indeed doomed to either grow forever, or change in non-backwards-compatible ways (e.g. Python3). But with macros you can shrink a language as well as grow it. This is one of the reasons Common Lisp continues to thrive.
Also, I forgot to mention another language-shrinker included in that library: REF. REF is a universal de-referencer that subsumes NTH, ELT, SLOT-VALUE, GETHASH and probably a few other things that I can't remember right now. It also lets you build abstract associative maps (a.k.a. dictionaries) with interchangeable implementations (see the DICTIONARY module), which lets you get rid of ASSOC and GETF.
I don't know Lisp, so correct me if I am wrong. The whole idea of embedding your own language sound pretty much the same as a growing the language, except that you are doing it yourself, in a non standard way.
If you build one new feature that subsumes N>1 old features then you're effectively shrinking the language because you never need to use those N old features any more. You can then support legacy code with macros that compile the old features into the new feature. At that point you've actually shrunk the language. You can even have those legacy support macros give compile-time warnings that the old, now-deprecated feature is being used and that the code ought to be changed. In fact, you can even easily write tools that will do this translation automatically. So yes, you really can shrink the language.
But even then, you have only shrunk the language for you - unless you can get everyone else to use your new feature. But if several such things are proposed, typically none of them gains traction, and so they all stay as just individual solutions.
Which is still fine - for the individuals who write them. But the language, as generally used, does not shrink.
Obviously anyone who chooses not to use a particular macro will not reap the benefits that macro provides. That doesn't change the fact that it's possible to write language-shrinking macros in Common Lisp, and so it's ironic to choose Common Lisp as the poster child for how languages inevitably grow without bound once they reach a certain level of complexity.
Indeed. Lisp is a poster child for how you can have one standard put out in 1994, and everything is reasonably fine 21 years later.
I really respect that about Common Lisp: the committee had the good sense (lacking in C and C++ and their ilk) of going their separate ways when they got the job done.
Good sense in a way that is related to good taste, I should add.
The way Common Lisp is generally used is by writing macros that shrink the language at higher levels. It's used that way because it can be. Whether one considers the lack of such capability in other languages a handicap or feature doesn't change that it is common Common Lisp practice. There aren't Common Lisp shops that don't use macros since Common Lisp includes standard macros.
>unless you can get everyone else to use your new feature.
When you are writing functions, classes you are extending language in similar way, and everyone else have to use this extension. You just not used to macros and your opinion is based only on your habits.
There are many articles on lisp that explain that in lisp you can try out many things in a few evenings and find suitable solution. In this way you shouldn't be smart enough to make an ideal, general design in one turn, but you will find it through many iterations. And this is what makes lisp different - these iterations are very cheap.
The 'standard' thing is both a gift and an issue. You're free to fulfill your needs, but everyone will do so. The community needs to be mature and sensitive, sharing good ideas, not building silos.
Both rigidly defined and flexibly defined languages appear to have their issues. I'm not much of a historian of Lisp but my basic understanding is that it's great for lone developers, but a pain for larger teams, and the flexibility of the language is the reason behind both of these.
In the Go language the tool gofmt formats the source code to certain standards. I haven't heard a single developer complain about gofmt, developers seem to appreciate that it takes away the bikeshedding over style, there's one established way to lay out your Go code and that's to use gofmt.
Now if bikeshedding can happen over something as trivial as code layout, imagine what's going to happen if you make it trivial to change the language. Someone didn't design a regex library the way you like it? Code your own. So will everyone else. You end up with thousands of developers reinventing the wheel, all building their own versions of the same thing, that aren't necessarily compatible.
CL now has its own package manager (Quicklisp), I'd be curious to know if this has had an impact on fragmentation, that's CL's best bet for reestablishing itself as a language that's growing in use.
I'm calling out this nonsense; "larger teams" of Common Lisp programmers are a myth. If such a thing happens, they will make it a priority to get along so that their good luck persists as long as possible. :) :)
> there's one established way to lay out your Go code and that's to use gofmt.
There is basically one established way to lay out Lisp, and even Vim's Lisp mode gets it about 95% right. (The main issue is not quite having the correct vocabulary, out of the box, of what symbols denote forms that have a body requiring sub-form indentation rather than list element alignment.)
(operator (with maybe some args)
(and a)
;; here is a header comment
(body of forms) ; margin comment
(indented by one level such as two spaces))
(function call with many arguments
split across several
lines lined up with the first argument)
'(quoted list
split across lines
lining up with first element
(oh here comes a nested
list))
;;;
;;; three semi's for block comment
;;;
#(vector literal
follows quoted list)
The above represents the majority of what you need for writing nice looking Lisp.
You've missed my point. I used gofmt as an example of a tool that exists because programmers like to do things multiple ways, even when it can be beneficial to do things in a standardised way.
The formatting of Lisp is not what I was referring to, it was the flexibility of the language. Lisp encourages writing DSLs. DSLs by their nature are opinionated. The only way to stop the language going in lots of directions at once is to have a canonical implementation of supporting features to work against. My point was that without a package manager, this focus was not as strong as it should be, but there's a hope that with standardisation on packages the language would grow with a sense of direction.
To use another example, look at how many implementations of CL and Scheme there are, why does all this fragmentation exist? What barriers are in place for coders working on extending a smaller subset of implementations?
For what it's worth, I'm not saying this is a Lisp-only trait (for another example in the computing world, just look at how many Linux distros exist), but the simplicity and flexibility of Lisp does lend itself to fragmentation. Whether this is a good thing or not is a matter of debate.
Out of interest, what impact have you noted since Quicklisp began being used?
> To use another example, look at how many implementations of CL and Scheme there are, why does all this fragmentation exist?
For Common Lisp the reasons are: competition, different licenses, create proprietary property to sell, different runtime implementation strategies, different compilation strategies.
Basically the same reason why there are several different Java implementations.
Common Lisp was created, because it was clear that there is a fragmentation of Lisp, but there shouldn't one for the basic language. Originally it was also thought to standardize on some libraries, but the whole thing ran out of steam. But you can read the Common Lisp standardization groups on various topics.
Quicklisp discourages people from re-inventing the wheel because it makes the pre-existing wheels trivial to access. No one in their right mind would try to reinvent regexps in Common Lisp (except perhaps as an exercise) because CL-PPCRE is just so fucking awesome. There's a CL library for just about everything nowadays, and most of them are very high quality.
That's what I meant about being mature. Lisp literature spends a lot of time explaining when to or not to use macros. Don't be silly and write something that won't be a game changer just because you want regexes literals like expressed #r#.... instead of (rx ...). I'm pretty positive any seasoned lisper can sense when abstractions must be added.
Btw, large teams may exist because tools don't solve the problem easily.
ps: It would be interesting to ask the CL community about Quicklisp social effects.
I don't think it's much different than the library situation with most other programming languages.
The one big difference is that Lisp libraries can add new controls structures to the language in ways that most language's libraries can't.
For example, in Python, I can fetch a website using the stdlib's http.client, or I can use the more convenient functions from the Requests library.
In Lisp, I can iterate and loop using the built-in "loop" construct, or I can use the more convenient control structures defined in the Iterate library.
The standard way to embed your own language is to use a Lisp to begin with.
Any Lisp language is effectively designed to provide standard tools and mechanisms for adding new language constructs so that you can write your own language without having to write either a full implementation from the tokenizer to the interpreter or use non-standard glue to hook an existing scripting language into a compatible C runtime.
The key idea is to grow layers of your own DSLs embedded in a Lisp. It is a bottom-up process of growing of a layers of abstractions, similar to building if a house (but, paradoxically, without blueorints) which some people called 'exploratory programming'.
PG in his "On Lisp" book explained this process in details. In some sense, those who never read it cannot claim to be a Lisp programmer.
So, you are not extending the language, you are growing a several new ones (layers) according to the domain at hands.
This is good one!) The insight were there for s long time - DSLs were popularized in SICP and AoMOP, and CL's looping expressions and SETF/GETF are the most famous, but PG is good at concise writing and he walked his talk with the Arc language.)
Don't get me wrong, I liked On Lisp a lot. I just think it's important to recognize that much of it is a distillation of pre-existing understandings and traditions.
This allows you to shrink the language syntactically, but not semantically. For instance, you can make it so setf isn't visible from your own package, but you can still call out to functions in other packages that will modify lists, so it's not suitable as a way to reduce the conceptual overhead "things you have to worry about when using lists".
In Haskell, there are language pragmas that enable extra features for a given module, but sometimes can also remove them (completely forbidding unsafe functions, disabling record syntax, and the like).
Haskell is really the kind of exploding language being discussed in the article though. While it has simple toggle switches for the end user, each of the underlying "extensions" is really a large modification to the compiler. There's still a monolithic parser, and authors of language extensions need to take into account all the other extensions to make sure they play nicely together. It's like they've heard of the open/closed principle but don't know how to apply it.
Specific extensions (like Safe/Trusted), or disabling record syntax etc are demonstrative of the problem - they're quite specific and implemented as part of the compiler because the language itself does not have the means of selectively enabling/disabling features via runtime code.
Kernel[1] has a much more interesting model based on first-class environments. One can selectively expose parts of the language to any execution context by creating a first-class environment containing only the required bindings, then execute some code in that environment. It provides a function ($remote-eval (code-to-exec) environment) for this purpose.
To give a very trivial example of this, lets say I want to expose only the numeric operators +, -, etc to some "safe calculator" context. I can simply bind these symbols to their kernel-standard-environment ones and be done.
Trying to do something like unsafe "read-file" in place of (+ (* 2 3) 1) will result in a runtime error because the binding read-file doesn't exist in that environment.
There's much more interesting stuff to be found in Kernel if you like the "small core" based approach to computing. Kernel is more schemy than scheme. Compiler hacks like "macros" are an example of something you wouldn't want lying around in your exploding language when you can implement them trivially with first class operatives. And why would you want quote implemented as a language primitive? Yuck!
> So for me, Common Lisp has actually shrunk without losing any functionality. This is not possible in languages without macros. Such languages are indeed doomed to either grow forever, or change in non-backwards-compatible ways (e.g. Python3). But with macros you can shrink a language as well as grow it.
Not true. If a language has adequate general constructs then you can replace language features with ordinary code written in the language. E.g. in Scala no-one uses "return" any more, because you can get the same functionality (in a better / more consistent way) using library types - just ordinary classes with ordinary methods, no macros needed.
With macros you've got a compiler support for anything you can imagine. And with macros you implement your DSLs as compilers, not interpreters, which is a huge advantage: you've got static verification, you've got any performance you like, and compilers are so much, much easier to implement than interpreters. So what's the point in doing the wrong, slow and error-prone thing instead of fast, easy and robust one?
If you're talking about this spire ( https://github.com/non/spire ) than it's using generics (i.e., poor man macros) and some real macros too, apparently.
> With macros you've got a compiler support for anything you can imagine.
Most of what you can imagine isn't useful or maintainable. Good tools should have more structure to guide you.
> And with macros you implement your DSLs as compilers, not interpreters
I'm not suggesting interpreters. If anything I'd say that macros - running arbitrary code at compile time - are more interpreterlike than what I'm describing.
> it's using generics (i.e., poor man macros)
Well if you're going to define every useful language feature as "macros" then of course you need macros to implement anything! But most of us consider generics to be different from macros.
> Good tools should have more structure to guide you.
It's not possible to have more solid and strict structure than with macro-based DSLs.
> I'm not suggesting interpreters.
You do. Either macros or interpreters. There is no other way.
> are more interpreterlike than what I'm describing.
You did not describe your approach to a problem yet. How would you implement an optimising BNF-based eDSL without macros? Spir and similar things are a totally different topic.
> But most of us consider generics to be different from macros.
Most people have absolutely no idea what metaprogramming is and how to implement DSLs properly. I would not refer to an opinion of a crowd.
> You did not describe your approach to a problem yet. How would you implement an optimising BNF-based eDSL without macros?
Let's get more concrete. What does the business requirement look like? Do you mean "a language that happens to be expressed in BNF", or are you asking for a DSL for expressing languages which itself looks like BNF?
> Let's imagine you want to embed parsers into your language. Choose any parsing algorithm you like, but parsers must be defined in a BNF-like syntax.
Doesn't sound hard to do in (macroless) scala. Just create objects and methods with appropriate names - and for the optimization part just ensure everything is lazy and preserves the structure so you have the AST available in the language and can do your optimizations at that level (which doesn't have to mean interpreting - we can use the type system to perform these computations at compile time[1]). The syntax will probably end up being slightly differently punctuated from actual BNF, which is a tradeoff for having syntax that follows the ordinary rules of the language and is accessible to e.g. an IDE.
I can agree that languages need to be able to perform complex transformations at compile time. But this doesn't have to be exactly the same kind as the compiler does itself, and as long as the language provides a sufficiently lightweight way of constructing an AST "in" the language, I think it's worthwhile making an explicit distinction between such ASTs and the AST of the language itself.
[1] I can imagine you objecting that this is just a macro system by another name, but it isn't (except in the trivial sense of turing equivalence). It has a different grain: it's more natural to create companion trees that mirror the structure of the AST exactly, and less natural to transform the AST by moving nodes around. And any such companions are explicitly distinct from the "original" tree, and the structure naturally lets you see both.
No, it's not easy, it's a barely usable hack. Coding anything on a type system level is like coding in Brainfuck or Unlambda, while with macros I can use whatever fancy DSLs I already have implemented for nice, declarative compiler construction.
Can you stop in the middle of your translation from BNF to low level code and dump a bunch of nice .dot files plus a tex documentation for the grammar? No. Your type system cannot do it, and you certainly do not want to do it in runtime, it's something to be done exclusively in compile time.
In a language with first-class functions, control constructs can be replaced by polymorphic methods - see smalltalk for the ultimate version of this. In scala the idiomatic way to do "short-circuit the rest of this in some cases" is usually for/yield with a type like scalaz's \/.
[UPDATE]: You can find my code here: https://github.com/rongarret/ergolib
Also, I forgot to mention another language-shrinker included in that library: REF. REF is a universal de-referencer that subsumes NTH, ELT, SLOT-VALUE, GETHASH and probably a few other things that I can't remember right now. It also lets you build abstract associative maps (a.k.a. dictionaries) with interchangeable implementations (see the DICTIONARY module), which lets you get rid of ASSOC and GETF.