Hello
just a quick summer update about some documentation cleanup and some checks on debugging HEΛP.
Have a look at the (small) changes and keep sending feedback.
Cheers
20230816
20230309
The LETV and LETV* Macros
Hello,
It has been a long time since I posted something here. I have been busy with my day job and bogged down in a major rewrite of something (more on this hopefully very soon now (tm)) that is full of rabbit's holes.
I was able to get out of one of these rabbit's holes with this little hack I cooked up that allows you to, possibly, write more concise code.
This little hack introduces two handy Common Lisp
macros, LETV and LETV* that allow you to mix
regular LET and MULTIPLE-VALUE-BIND forms in
a less verbose way. The amount of indentation needed is also reduced.
The syntax of LETV (or LETV*) is very
"loopy", with a nod to SML/OCaml/F#/Haskell/Julia. The current syntax
is the following:
letv ::= 'LETV' [vars] [IN [body]]
letvstar ::= 'LETV*' [vars] [IN [body]]
vars ::= var | var vars
var ::= ids '=' <form> [decls]
decls ::= decl | decl decls
decl ::= OF-TYPE idtypes
ids ::= <symbol> | '(' <symbol> + ')'
idtypes ::= <type designator> | '(' <type designator> + ')'
body ::= [<declarations>] <form> *
(I know: the grammar is not completely kosher, but I trust you will understand it.)
The two macros expand in forms that mimic the semantics of
let, LET* and
MULTIPLE-VALUE-BIND. All type declarations, if present,
are properly handled via locally forms.
LETV expands into a LET, with variable
initialized by PSETQs and
multiple-VALUE-SETQs.
LETV* expands into a form that is an interleaving of
LET* and MULTIPLE-VALUE-BIND.
The library exports only the two symbols LETV and
LETV*. The other "symbols" mentioned (=,
IN, OF-TYPE) are checked as in
LOOP; therefore you can use different styles to write
your code, as you would when writing LOOPs.
The library is available here.
Examples
- Simple case:
(letv x = 42 in (format t "The answer is ~D.~%" x)) - Same with declaraion:
(letv x = 42 of-type fixnum in (format t "The answer is ~D." x)) - Simple case with
MULTIPLE-VALUE-BIND:(letv (v found) = (gethash 'the-key *my-hash-table*) in (if found (format t "Found THE-KEY, doing stuff.~%") (error "THE-KEY not found."))) - Mixing things up:
(letv (v found) = (gethash 'the-key *my-hash-table*) of-type (fixnum boolean) x = 42 in (if found (format t "Found THE-KEY, adding to the answer ~D.~%" (+ v x)) (error "THE-KEY not found."))) - With
LETV*:(letv* (v found) = (gethash 'the-key *my-hash-table*) of-type (fixnum boolean) x = (when found (+ v 42)) in (if found (format t "Found THE-KEY, adding to the answer ~D.~%" x) (error "THE-KEY not found."))) - The other way around.
(letv x = 42 of-type integer (v found) = (gethash 'the-key *my-hash-table*) of-type (fixnum boolean) in (if found (format t "Found THE-KEY, adding to the answer ~D.~%" (+ v x)) (error "THE-KEY not found."))) - A more compelling example.
(letv* (p found) = (gethash 'the-key *points-table*) of-type (point) (x y) = (if found (unpack-point p) (values :missing :missing)) in (declare (type point p) (type real x y)) ; Adding declarations here also works. (do-stuff-with x y) (do-things-with p))
All the examples are meant to illustrate the use of
LETV and LETV*.
Notes
LETV and LETV* are obviously not the first
macros of this kind floating around: others are available and all have
their niceties. But I never claimed not to suffer from NIH syndrome.
LETV and LETV* do not do "destructuring" or
"pattern matching". That is a different can of worms; but you can
check cl-unification for a
library (always by yours truly) that provides facilities in that
sense.
'(Cheers)
20220121
CDR is Next...
Hi
as many may know, I have been nursing (almost to death!) the Common Lisp Document Repository (CDR). Pascal Costanza et al., started the project many years ago and then I sat on top of it for many more.
I finally found some time to work on it and the result is a revamped site (cdr.common-lisp.dev) with the addition of stashing documents in a Zenodo Community (CDR), which has the benefit of producing a DOI for each write-up.
Moreover, Pascal Bourguignon, Michał "phoe" Herda and Didier Verna have agreed to become CDR editors. Many thanks to them.
So, if anyone wants to submit a "specification" for something of interests to the CL community, she/he is most welcome to do so. Just remember that good specifications are not so easy to write.
20220105
My List of Common Lisp Libraries and SW 2022 (Plus a Couple of Other Things)
Hello
I have been bumping on a few "lists of Common Lisp libraries and tools" written by many people. I feel like rantin... pardon, blogging about this state of affairs.
First of all we have CLiki, which is a rather comprehensive list of CL libraries and whatnot, and then we have a few, unnamed, "state of the CL ecosystem", "preferred list of Common Lisp libraries", etc, etc.
I have nothing against people blogging or making lists of course, but I tend not to make generalized statements. Especially in order to avoid disrespecting some people's work, just by not knowing of its existence. This is a hint.
So, in order to proceed with my ran... blog post, here is the list of CL libraries I use. Turns out there is quite a bit of NIH syndrome here, but I never claimed not to suffer from it. Also remember that I have a Mac, a Windows and a Linux system at hand. I always try to have stuff that works on all of them.
Distribution Systems
I use, obviously, Quicklisp. It works. It can be improved, but hey! Let's just give Zach our love! 💓 (And money!)
Implementations
- I work mostly on Lispworks. I have the luck to be able to afford the Enterprise edition (or better, my funding does) and I am very happy with it. The folks at Lispworks know that I can be a pest (environments? rounding modes?) but they are just doing a fantastic job.
- I use, of course, SBCL. The implementation is rock solid and it has all the bells and whistles you need for a world class system. Recompiling and deploying for SBCL usually uncovers bugs and potential pitfalls in your code. I have a small rant about it though. Guys, do not write code that "works on SBCL" and just assume all is fine. It is not.
- I have also CMUCL installed and I use it as well. It still has a few good things to it.
- Next I use Armed Bear Common Lisp, an excellent testbed for checking portability.
- Of course, I have the free edition of Allegro CL: the other excellent commercial alternative.
- Last but not least (with memories going back decades) I have CCL installed and always ready to use.
- I am very intrigued by CLASP, but I admit, I have not had the strength to install it (too much of a production; remember that I try to get things working on three platforms); any help will be appreciated.
- Same for the latest incarnation of ECL. Sorry guys, I will get back to installing it soon.
- Finally, we always have CLisp: old but good.
I know I forgot many. Apologies for my senescence.
System Building
If you checked any of my libraries, you will have noticed that I still have .system files everywhere. I use ASDF of course, but I have been nursing (with Madhu) a version of MK:DEFSYSTEM; it is clunkier (and old) internally, but I feel it has a simpler and more straightforward interface, especially when it comes to its footprint on your system (i.e., somewhat simpler working of the registry).
Concurrency
I use Bordeaux Threads for portability. There are a few open issues about it that are of very difficult solution but it is Good Enough (it has a couple of warts: a bad .asd file for example - but this is a separate rant, pardon, blog post). Of course I also like very much Lispworks multiprocessing library, especially mailboxes, but I understand that not everybody has access to it.
Testing
I must say that none of the libraries I checked does ONE thing I really need: running something and stopping after a timeout (see the open issue I mentioned just before for Bordeaux Threads).
Having said that, I use FiveAM; I have to muck a bit around it to get the timeout I need, and the documentation, especially about macr... pardon, fixtures is lacking, but it does the job. I am experimenting with Parachute, but have not made the switch yet.
Portability
Most of my libraries try to be portable. One major library that helps you a lot with this thankless job ("let's have filename case insensitive anyone?") is of course UIOP. You have it pretty much by default, but it would be nice if it were eventually decoupled from ASDF.
Another little library I advocate (yes, Virginia, here they come!) is CLAD: it does not do much, but it gives you a bedrock upon which to build your work.
Editing
I use the Lispworks IDE or the Editor (as in "thou shalt not have other Editor than...") with SLIME. End of story.
HTML Generation
Here I start with the rantin..., thoughtful exposition of various libraries.
To generate (X)HTML (or HTML5) I use (X)HTMΛ. It is a pretty solid library with a couple of bells and whistles: mainly, it has an object model and it leverages the sorrily underleveraged Common Lisp pretty printer to produce readable HTML. The web page for (X)HTMΛ is produced using itself as a building block; see below.
Web Server and Client
In this case I stick with the tried and true Hunchentoot and its counterpart Drakma, although for simpler, one-shot things I do use Lispworks facilities when needed.
Documentation Generation
As I said, all my libraries suffer from a severe case of NIH, which is reflected in my use of the hacked up HEΛP documentation system. It is still not perfect, but, again, by using (X)HTMΛ and, again, leveraging the pretty printer, I find that the results are quite pleasant (for the neapolitans: ogni scarrafone...). Of course, if you agree to format your doc string in a Hyperspec way. How can it be improved? Four things.
- Incorporating Eclector as a "reader" (as of now the tool uses more than a kludge to get past the non standardized way of dealing with READER- and PACKAGE- errors).
- Finishing the HTML5 generation scheme.
- Adding Texinfo generation (as per Didier Verna's declt library).
- Fixing cross-referencing, especially with standard CL (cfr., Hyperspec).
Mathematics
Why did I start HEΛP? Many years ago (or many Kgs ago) a post on comp.lang.lisp stated that you could shadow +, -, *, and / in a package and make them generic functions (as an aside, I recently found out that the idea of generics dates back to PL/I - but this is a different rabbit's hole).
The result was what is now a forever embryonic Common Math, which led to my forays in "let's do what R can do" -- Ρ (greek Rho) -- and to the stuff I am talking about in "Why you cannot write an Interval Arithmetic Library...". All of this another rabbit hole: bottom line, this is what I am working on and off right now, apart from my day job.
Other Libraries
Laziness as a Way of Life
Of course, rabbits' holes are never linear. They branch in several directions at once. E.g., once you look at other "modern" languages (I am partial to Haskell) you want to have some of these features in Common Lisp as well); hence the CLAZY library (laziness as a way of life).
Code Handling
Needless to say, CLAZY eventually needed some proper code walking, or better it needed some proper Abstract Syntax Tree (AST) handling, which led to the CLAST library: this last library allows you to code walk and to properly inspect a piece of Common Lisp code, striving very hard to portably manage the environments that were present in CLtL2 but did not make it in the ANSI spec.
Programming Common Lisp
As we all know, Lisp is like a ball of mud. Over the years I threw a lot of mud at it.
Some of the mud I liked the most is embedded in the following libraries
- definer is a small hack to have Pythonesque def available in Common Lisp, of course, in an extensible way (documentation being updated soon).
- cl-unification is a full blown unification machine for Common Lisp objects and "templates"; not as fast as a simpler pattern matcher (like several listed in CLiki Pattern Matching), but very general (it also relies on CL-PPCRE).
- cl-enumeration was the first full blown Java-like enumeration/iterator library for Common Lisp; it works.
- defenum is another Java-ism ported to Common Lisp; it essentially creates enums that work like Java, i.e., with quite a bit of underlying machinery. Of course you do not have Haskell data definitions available, but it was fun to write.
- with-contexts was a bout of Python-envy, but then again, I think I was able to show that you really can program Common Lisp in a fun way.
Literary Programming in Common Lisp
One final library I am quite fond of expresses my love of (certain) books. And to deal with books, you need a Library and a Librarian. Hence the OOK compiler in Common Lisp. Don't say Monkey!
Not Done Yet...
Although I am dabbling with Julia (a Common Lisp in a drag) and Rust (I have a day job which also involves programming languages and not only Cancer Research) I still have a lot of "not-quite-there" stuff I wrote in Common Lisp.
All I can say about it is... stay tuned, I do put out stuff and I do pester people about Common Lisp. You may find some of the stuff interesting.
(cheers)
20210320
HEΛPing ASDF
... more fixing and, ça va sans dire, more creeping features.
I got prodded to integrate HEΛP with other tools; mostly, of course, ASDF. A simple solution was to define a document-op for a system. After jumping through a few hoops, the solution was to use the :properties of a system to pile up arguments for the main HEΛP document function (well, only one for the time being). Bottom line, suppose you have:
(asdf:defsystem "foosys"
:pathname #P"D:/Common Lisp/Systems/foosys/")
now you just issue
(asdf:operate 'hlp:document-op "foosys")
and the documentation for the system "foosys" will appear in the "docs/html/" subfolder.
If you want to pass a title to the document function, you set up your system as:
(asdf:defsystem "foosys"
:properties (:documentation-title "The FOO Omnipotent Tool")
:pathname #P"D:/Common Lisp/Systems/foosys/")
and the parameter will be used (instead of the bare system name).
It works! 😁
Some more fixing and more extensions may be needed (hlp:document takes a lot of parameters) but it is already usable.
All the necessary bits and pieces are in the HEΛP repository, and they should get into Quicklisp in the next release.
Enjoy
(cheers)
20210316
Need more HEΛP?
Just a quick note for people following these... parentheses.
I have carved out some time to do some more Lisp hacking and this lead me to look at the very nice usocket library (I want to do some network programming). The usocket library documentation page has a bit of an "old" and "handcrafted" look and feel to it, so I tried to produce a version of the documentation with help from my HEΛP library.
Well, it turns out that usocket has some more than legitimate code within it that my HEΛP library was not handling; even worse, it unearthed a bug in the Lambda List parsing routines.
As an example, usocket uses the following idiom to set some of the documentation strings.
(setf (documentation 'fun 'function) "Ain't this fun?")
This is perfectly fine, but it needed some extra twist to get HEΛP do what is, IMHO, the right thing: in this case it meant ensuring that the lambda list of the function was properly rendered in the final documentation.
Apart from that, a few not so nice buglets were exposed in the code parsing lambda lists. The result is that now the logic of that piece of code is simpler and somewhat cleaner.
So, if you want to get HEΛP to document your Common Lisp code, give it a spin.
(cheers)
20201212
With what are we contextualizing?
Common Lisp
programmers may write
many with-something overt their careers; the
language specification itself is ripe with such constructs: witness with-open-file. Many other libraries also
introduce a slew of with- macros dealing with this or that case.
So, if this is the case, what prevents Common
Lisp programmers from coming up with a
generalized with
macro?
It appears that the question has been answered, rather satisfactorily, in Python and Julia (at least). Python offers the with statement, alongside a library of "contexts" (Python introduced the with statement in 2005 with PEP 343) and Julia offers its do blocks.
In the following I will present WITH-CONTEXTS, a Common Lisp answer to the question. The library is patterned after the ideas embodied in the Python solution, but with several (common) "lispy" twists.
Here is the standard - underwhelming - example:
(with f = (open "foo.bar") do
(do-something-with f))
That's it as far as syntax is concerned
(the 'var =' being optional, obviously
not in this example; the syntax was chosen to
be loop-like, instead of using
Python's as keyword). Things become more
interesting when you look under the hood.
Traditional Common Lisp with- macros expand in variations of unwind-protect or handle-case (and friends). The example above, if written with with-open-file would probably expand into something like the following:
(let ((f nil))
(unwind-protect
(progn
(setq f (open "foo.bar"))
(do-something-with f))
(when f (close f))))
Python generalizes this scheme by introducing a enter/exit protocol that is invoked by the with statement. Please refer to the Python documentation on contexts and their __enter__ and __exit__ methods.
The "WITH" Macro in Common Lisp: Contexts and Protocol
In order to introduce a with macro in Common Lisp that mimicked what Python programmers expect and what Common Lisp programmers are used to some twists are necessary. To achieve this goal, a protocol of three generic functions is provided alongside a library of contexts.
The ENTER/HANDLE/EXIT Context Protocol
The WITH-CONTEXTS library provides three generic functions that are called at different times within the code resulting from the expansion of the onvocation of the with macro.
- enter: this generic function is invoked when the with macro "enters" the context; its main argument is the result of the expression that is the argument of the with macro.
- handle: this generic function is called to take care of exceptional situations that may arise during the call to enter or during the execution of the body of the with macro.
- exit: this generic function is called to "clean up"
before exiting the context entered by means of
the
withmacro.
Given the protocol (from now on referred to as the "EHE-C protocol"), the (undewhelming) "open file" example expands in the following:
(let ((f nil)) (unwind-protect (progn (setq f (enter (open "contexts.lisp"))) (handler-case (open-stream-p f) (error (#:ctcx-err-e-41883) (handle f #:ctcx-err-e-41883)))) (exit f)))
Apart from the gensymmed variable the expansion is
pretty straightforward. The function enter is
called on the newly opened stream (and is essentially an identity
function) and sets the variable. If some error happens while the
body of the macro is executing then control is passed to
the handle function (which, in its most basic form
just re-signals the condition). Finally, the unwind-protect has a
chance to clean up by calling exit (which, when
passed an open stream, just closes it).
One unexpected behavior for Common Lisp programmers is that the variable (f in the case above) escapes the with constructs. This is in line with what Python does, and it may have its uses. The file opening example thus has the following behavior:
CL-prompt > (with f = (open "contexts.lisp") do
(open-stream-p f))
T
CL-prompt > (open-stream-p f)
NIL
To ensure that this behavior is reflected in the implementation, the actual macroexpansion of the with call becomes the following.
(let ((#:ctxt-esc-val-41882 nil))
(multiple-value-prog1
(let ((f nil))
(unwind-protect
(progn
(setq f (enter (open "contexts.lisp")))
(handler-case
(open-stream-p f)
(error (#:ctcx-err-e-41883)
(handle f #:ctcx-err-e-41883))))
(multiple-value-prog1
(exit f)
(setf #:ctxt-esc-val-41882 f))))
(setf f #:ctxt-esc-val-41882)))
This "feature" will help in - possibly - porting some Python code to Common Lisp.
"Contexts"
Python attaches to the with statement the notion of contexts. In Common Lisp, far as the with macro is concerned, anything that is passed as the expression to it, must respect the enter/handle/exit. protocol. The three generic functions enter, handle, exit, have simple defaults that essentially let everything "pass through", but specialized context classes have been defined that parallel the Python context library classes.
First of all, the current library defines the EHE-C protocol for streams. This is the strightforward way to obtain the desired behavior for opening and closing files as with with-open-file.
Next, the library defines the following "contexts" (as Python does).
- null-context:
this is a full "pass through" context, just encapsulating the expression passed to it. - managed-resource-context:
this is a first cut implementation of a "managed resource", which implements also an acquire/release protocol; of course, this would become more useful in presence of mutltiprocessing (see notes in Limitations). - redirect-context:
this is a context that redirects output to a different stream. - suppress-context:
this is a context that selectively handles some conditions, while ignoring other ones. - exit-stack-context:
this is a context that essentially allows a programmer to manipulate the "state of the computation" within it body and combine other "contexts"; to achieve its design goal, it leverages the functions of a protocol comprising the enter-context, push-context, callback, pop-all and unwind (this is equivalent to the Python close() context method).
This should be a good enough base to start working with contexts in Common Lisp. It is unclear whether the Python decorator interface would provide some extra functionality in this Common Lisp implementation of contexts and the with macro.
Limitations
The current implementation has a semantics that is obviously not the same as the corresponding Python one, but it is hoped that it still provided useful functionality. There are some obvious limitations that should be taken into account.
The current implementation of the library does not take into consideration threading issues. It could, by providing a locking-context based on a portable multiprocessing API (e.g., bordeaux-threads).
The Python implementation of contexts relies heavily on the yield statement. Again, the current implementation does not provide similar functionality, although it could possibly be implemented using a delimited continuation library (e.g., cl-cont).
Disclaimer
The code associated to these documents is not completely tested and it is bound to contain errors and omissions. This documentation may contain errors and omissions as well. Moreover, some design choices are recognized as sub-optimal and may change in the future.
License
The file COPYING that accompanies the library contains a Berkeley-style license. You are advised to use the code at your own risk. No warranty whatsoever is provided, the author will not be held responsible for any effect generated by your use of the library, and you can put here the scariest extra disclaimer you can think of.
Repository and Downloads
The with-contexts library is available on Quicklisp (not yet).
The with-contexts library. is hosted at common-lisp.net.
The git repository can be gotten from the common-lisp.net Gitlab instance in the with-macro project page.
(cheers)
Subscribe to:
Posts (Atom)