20240303

A Rant about R and Python (and anything with a-lists, tuples and "dictionaries").

R and Python come from Lisp (R for sure, Python will deny it). Early Lisp. Even before the first edition of "AI Programming" by Charniak, Riesbek and McDermott.

At that time, there were a-lists and p-lists. Then Charniak, Riesbeck and McDermott taught us (not only them of course) to create records in Lisp. You know... those things like:

    DECLARE
      1 STUDENT,
        2 NAME     CHAR (30),
        2 SURNAME  CHAR (50),
        2 ID FIXED DECIMAL (8),
        2 ADDRESS,
          3 STREET  CHAR (80),
          3 NUMBER  FIXED DECIMAL (5),
          3 CITY    CHAR (80),
          3 PRST    CHAR (20),
          3 ZIP     CHAR (10),
          3 COUNTRY CHAR (80);

Using a-lists the above may become:

    (defvar student '((name . "John")
                      (surname . "Blutarski")
                      (id . 42)
                      (address . ((street . "United State Senate")
                                  (number . 0)
                                  (city . "Washington")
                                  (prst . "DC")
                                  (zip . "20510")
                                  (country . "U.S.A.")))
                      ))

In Python you can do the following.

    student = {}    # a 'Dict'; i.e., a hash table.
    student['name'] = "John"
    student['surname'] = "Blutarski"
    student['id'] = 42
    student['address'] = {}
    student['address']['street'] = "United State Senate"
    student['address']['number'] = 0
    student['address']['city'] = "Washington"
    student['address']['prst'] = "DC"
    student['address']['zip'] = "20510"
    student['address']['country'] = "U.S.A."

Not that you must, but surely you can; and this, as in the case of R below, is the root of my rant.

In R you use lists; a misnomer for something that is essentially a dictionary like in Python, patterned, ça va sans dire, after a-lists.

    student = list()
    student$name = "John"
    student$surname = "Blutarsky"
    student$id = 42
    student$address = list()
    student$address$street = "United State Senate"
    student$address$number = 0
    student$address$city = "Washington"
    student$address$prst = "DC"
    student$address$zip = "20150"
    student$address$country = "U.S.A."

This of course gives you a lot of flexibility; e.g., if - in the middle of your code - you need to deal with the student's nickname, you just write the following.

In (Common) Lisp:

    (defun tracking-student ()
        ...
        ...
        (setf student (acons 'nickname "Bluto" student))
        ...
        )

In Python:

    def tracking_student():
        ...
        ...
        student['nickname'] = "Bluto"
        ...

In R:

    tracking_student <- function() {
        ...
        ...
        student$nickname = "Bluto"
        student <<- student     # Yes, R scoping rules are ... interesting.
        ...
    }

The example is relatively simple and innocuous, but it has some consequences when you have to actually read some code.

The problem I have (it may be just me) is that this programming style does not give me an overview of what a data structure (a record) actually is. The flexibility that this style allows for is usually not accompanied by the necessary documentation effort telling the reader what is going into an "object". The reader is therefore left wondering, while taking copious notes about what is what and where.

Bottom line: don't do that. Use defstruct and defclass in CL, classes in Python, struct in Julia, etc. etc. etc. In R, please document your stuff.

You may feel that your code is less malleable, but, in the end it becomes easier to read. At least for me. Sorry.


(cheers)

20240104

A Parenthetical Year and New Open Parentheses

It has been a few years that I have spent the New Year Holiday to update my Common Lisp libraries and to think about all the open parentheses that are left unclosed. This year is no different, only I have had even less time to work on CL. Therefore, my updates to my libraries on common-lisp.net and Sourceforge all have had some minor repairs and copyright updates. As usual, I direct you to HEΛP and CLAST; most of my other libraries are support for these two.

One effort, I would like to finalize in the coming year (that is; closing this parenthesis) is the CL-LIA layer/library. Help is wanted for this! Drop me a line if you want to chip in. The main issues are API designs around the floating point environment condition handling.

Apart from that, I have been looking more and more to the future, that is, Julia (I should have listened more to some people in the CL community years ago), and... in the past.

Julia and Cancer Research

I have advised a student to work on Julia and the result is this nice (very rough) simulator for solid tumors with input from images and tracking of cells lineages: SOPHYSM. SOPHYSM builds on J-Space which is the simulator core; you can find the publication at BMC Bioinformatics.

I am currently pushing Julia to my research group, especially for new AI and ML applications to Data Analisys in Cancer Research (cfr., Data and Computational Biology Laboratory at Università degli Studi di Milano-Bicocca). This is because new Julia ML and AI frameworks have become much more solid in recent months; in particular Flux and SciML.ai.

Programming Archeology

If you have followed my older posts, you know I fell into a rabbit hole (with several side tunnels). I have been dabbling with very old technology, which turned out to be ... fun (don't tell my Department Head!).

I will just say that now I can understand the vagaries of PL/I (where do you think that loop and generic functions come from?) and that I can write decent Fortran IV (66) (downgrading from Fortran 77). And yes! I can compile my program on Hercules (of course using Emacs to submit jobs to MVS TK5; some work required).

Happy Hacking and Science Year!

That's it. These are my report for 2023, and my wishes for you all in 2024.

Just one more thing: you reader know that I am becoming and old geezer. I somehow rationalize my dabbling with parentheses and even older stuff as "taking care" of things that should not be lost; even if now we have Julia.

Let's all try to take care of each other and the world, of the old things and of the new ones that will inherit it.


'(peace)

20230816

Documenting/debugging HEΛP

Hello

just a quick summer update about some documentation cleanup and some checks on debugging HEΛP.

Have a look at the (small) changes and keep sending feedback.



Cheers

20230309

The LETV and LETV* Macros

Hello,

It has been a long time since I posted something here.  I have been busy with my day job and bogged down in a major rewrite of something (more on this hopefully very soon now (tm)) that is full of rabbit's holes.

I was able to get out of one of these rabbit's holes with this little hack I cooked up that allows you to, possibly, write more concise code.

This little hack introduces two handy Common Lisp macros, LETV and LETV* that allow you to mix regular LET and MULTIPLE-VALUE-BIND forms in a less verbose way. The amount of indentation needed is also reduced.

The syntax of LETV (or LETV*) is very "loopy", with a nod to SML/OCaml/F#/Haskell/Julia. The current syntax is the following:

    letv     ::= 'LETV' [vars] [IN [body]]
    letvstar ::= 'LETV*' [vars] [IN [body]]

    vars     ::= var | var vars

    var      ::= ids '=' <form> [decls]

    decls    ::= decl | decl decls
    decl     ::= OF-TYPE idtypes

    ids      ::= <symbol> | '(' <symbol> + ')'
    idtypes  ::= <type designator> | '(' <type designator> + ')'

    body     ::= [<declarations>] <form> * 

(I know: the grammar is not completely kosher, but I trust you will understand it.)

The two macros expand in forms that mimic the semantics of let, LET* and MULTIPLE-VALUE-BIND. All type declarations, if present, are properly handled via locally forms. LETV expands into a LET, with variable initialized by PSETQs and multiple-VALUE-SETQs. LETV* expands into a form that is an interleaving of LET* and MULTIPLE-VALUE-BIND.

The library exports only the two symbols LETV and LETV*. The other "symbols" mentioned (=, IN, OF-TYPE) are checked as in LOOP; therefore you can use different styles to write your code, as you would when writing LOOPs.

The library is available here.

Examples

  1. Simple case:
      (letv x = 42 in (format t "The answer is ~D.~%" x))
      
  2. Same with declaraion:
      (letv x = 42 of-type fixnum
    	in (format t "The answer is ~D." x))
      
  3. Simple case with MULTIPLE-VALUE-BIND:
      (letv (v found) = (gethash 'the-key *my-hash-table*)
            in (if found
                   (format t "Found THE-KEY, doing stuff.~%")
                   (error "THE-KEY not found.")))
      
  4. Mixing things up:
      (letv (v found) = (gethash 'the-key *my-hash-table*)
                      of-type (fixnum boolean)
             x = 42
             in (if found
                     (format t "Found THE-KEY, adding to the answer ~D.~%"
                             (+ v x))
                     (error "THE-KEY not found.")))
      
  5. With LETV*:
      (letv* (v found) = (gethash 'the-key *my-hash-table*)
                       of-type (fixnum boolean)
             x = (when found (+ v 42))
             in (if found
                    (format t "Found THE-KEY, adding to the answer ~D.~%"
    			 	        x)
                    (error "THE-KEY not found.")))
      
  6. The other way around.
      (letv x = 42 of-type integer
            (v found) = (gethash 'the-key *my-hash-table*)
                      of-type (fixnum boolean)
            in (if found
                   (format t "Found THE-KEY, adding to the answer ~D.~%"
                           (+ v x))
                   (error "THE-KEY not found.")))
      
  7. A more compelling example.
      (letv* (p found) = (gethash 'the-key *points-table*) of-type (point)
             (x y) = (if found
                         (unpack-point p)
                         (values :missing :missing))
             in
             (declare (type point p)
                      (type real x y))  ; Adding declarations here also works.
    
             (do-stuff-with x y)
             (do-things-with p))
      

All the examples are meant to illustrate the use of LETV and LETV*.

Notes

LETV and LETV* are obviously not the first macros of this kind floating around: others are available and all have their niceties. But I never claimed not to suffer from NIH syndrome.

LETV and LETV* do not do "destructuring" or "pattern matching". That is a different can of worms; but you can check cl-unification for a library (always by yours truly) that provides facilities in that sense.


'(Cheers)

20220121

CDR is Next...

 Hi

as many may know, I have been nursing (almost to death!) the Common Lisp Document Repository (CDR).  Pascal Costanza et al., started the project many years ago and then I sat on top of it for many more.

I finally found some time to work on it and the result is a revamped site (cdr.common-lisp.dev) with the addition of stashing documents in a Zenodo Community (CDR), which has the benefit of producing a DOI for each write-up.

Moreover, Pascal Bourguignon, Michał "phoe" Herda and Didier Verna have agreed to become CDR editors.  Many thanks to them.

So, if anyone wants to submit a "specification" for something of interests to the CL community, she/he is most welcome to do so.  Just remember that good specifications are not so easy to write.

20220105

My List of Common Lisp Libraries and SW 2022 (Plus a Couple of Other Things)

Hello

I have been bumping on a few "lists of Common Lisp libraries and tools" written by many people.  I feel like rantin... pardon, blogging about this state of affairs.

First of all we have CLiki, which is a rather comprehensive list of CL libraries and whatnot, and then we have a few, unnamed, "state of the CL ecosystem", "preferred list of Common Lisp libraries", etc, etc.

I have nothing against people blogging or making lists of course, but I tend not to make generalized statements.  Especially in order to avoid disrespecting some people's work, just by not knowing of its existence.  This is a hint.

So, in order to proceed with my ran... blog post, here is the list of CL libraries I use.  Turns out there is quite a bit of NIH syndrome here, but I never claimed not to suffer from it.  Also remember that I have a Mac, a Windows and a Linux system at hand.  I always try to have stuff that works on all of them.

Distribution Systems

I use, obviously, Quicklisp.  It works.  It can be improved, but hey!  Let's just give Zach our love! 💓 (And money!)

Implementations

  • I work mostly on Lispworks.  I have the luck to be able to afford the Enterprise edition (or better, my funding does) and I am very happy with it.  The folks at Lispworks know that I can be a pest (environments? rounding modes?) but they are just doing a fantastic job.
  • I use, of course, SBCL.  The implementation is rock solid and it has all the bells and whistles you need for a world class system.  Recompiling and deploying for SBCL usually uncovers bugs and potential pitfalls in your code.  I have a small rant about it though.  Guys, do not write code that "works on SBCL" and just assume all is fine. It is not.
  • I have also CMUCL installed and I use it as well.  It still has a few good things to it.
  • Next I use Armed Bear Common Lisp, an excellent testbed for checking portability.
  • Of course, I have the free edition of Allegro CL: the other excellent commercial alternative.
  • Last but not least (with memories going back decades) I have CCL installed and always ready to use.
  • I am very intrigued by CLASP, but I admit, I have not had the strength to install it (too much of a production; remember that I try to get things working on three platforms); any help will be appreciated.
  • Same for the latest incarnation of ECL.  Sorry guys, I will get back to installing it soon.
  • Finally, we always have CLisp: old but good.
I know I forgot many.  Apologies for my senescence.

System Building

If you checked any of my libraries, you will have noticed that I still have .system files everywhere.  I use ASDF of course, but I have been nursing (with Madhu) a version of MK:DEFSYSTEM; it is clunkier (and old) internally, but I feel it has a simpler and more straightforward interface, especially when it comes to its footprint on your system (i.e., somewhat simpler working of the registry).

Concurrency

I use Bordeaux Threads for portability.  There are a few open issues about it that are of very difficult solution but it is Good Enough (it has a couple of warts: a bad .asd file for example - but this is a separate rant, pardon, blog post).  Of course I also like very much Lispworks multiprocessing library, especially mailboxes, but I understand that not everybody has access to it.

Testing

I must say that none of the libraries I checked does ONE thing I really need: running something and stopping after a timeout (see the open issue I mentioned just before for Bordeaux Threads).
Having said that, I use FiveAM; I have to muck a bit around it to get the timeout I need, and the documentation, especially about macr... pardon, fixtures is lacking, but it does the job.  I am experimenting with Parachute, but have not made the switch yet.

Portability

Most of my libraries try to be portable.  One major library that helps you a lot with this thankless job ("let's have filename case insensitive anyone?") is of course UIOP.  You have it pretty much by default, but it would be nice if it were eventually decoupled from ASDF.
Another little library I advocate (yes, Virginia, here they come!) is CLAD: it does not do much, but it gives you a bedrock upon which to build your work.

Editing

I use the Lispworks IDE or the Editor (as in "thou shalt not have other Editor than...") with SLIME.  End of story.

HTML Generation

Here I start with the rantin..., thoughtful exposition of various libraries.
To generate (X)HTML (or HTML5) I use (X)HTMΛ.  It is a pretty solid library with a couple of bells and whistles: mainly, it has an object model and it leverages the sorrily underleveraged Common Lisp pretty printer to produce readable HTML.  The web page for (X)HTMΛ is produced using itself as a building block; see below.

Web Server and Client

In this case I stick with the tried and true Hunchentoot and its counterpart Drakma, although for simpler, one-shot things I do use Lispworks facilities when needed.

Documentation Generation

As I said, all my libraries suffer from a severe case of NIH, which is reflected in my use of the hacked up HEΛP documentation system.  It is still not perfect, but, again, by using (X)HTMΛ and, again, leveraging the pretty printer, I find that the results are quite pleasant (for the neapolitans: ogni scarrafone...).  Of course, if you agree to format your doc string in a Hyperspec way.  How can it be improved?  Four things.
  1. Incorporating Eclector as a "reader" (as of now the tool uses more than a kludge to get past the non standardized way of dealing with READER- and PACKAGE- errors).
  2. Finishing the HTML5 generation scheme.
  3. Adding Texinfo generation (as per Didier Verna's declt library).
  4. Fixing cross-referencing, especially with standard CL (cfr., Hyperspec).

Mathematics

Why did I start HEΛP?  Many years ago (or many Kgs ago) a post on comp.lang.lisp stated that you could shadow +, -, *, and / in a package and make them generic functions (as an aside, I recently found out that the idea of generics dates back to PL/I - but this is a different rabbit's hole).
The result was what is now a forever embryonic Common Math, which led to my forays in "let's do what R can do" -- Ρ (greek Rho) -- and to the stuff I am talking about in "Why you cannot write an Interval Arithmetic Library...".  All of this another rabbit hole: bottom line, this is what I am working on and off right now, apart from my day job.

Other Libraries

Laziness as a Way of Life

Of course, rabbits' holes are never linear.  They branch in several directions at once.  E.g., once you look at other "modern" languages (I am partial to Haskell) you want to have some of these features in Common Lisp as well); hence the CLAZY library (laziness as a way of life).

Code Handling

Needless to say, CLAZY eventually needed some proper code walking, or better it needed some proper Abstract Syntax Tree (AST) handling, which led to the CLAST library: this last library allows you to code walk and to properly inspect a piece of Common Lisp code, striving very hard to portably manage the environments that were present in CLtL2 but did not make it in the ANSI spec.

Programming Common Lisp

As we all know, Lisp is like a ball of mud.  Over the years I threw a lot of mud at it.
Some of the mud I liked the most is embedded in the following libraries
  • definer is a small hack to have Pythonesque def available in Common Lisp, of course, in an extensible way (documentation being updated soon).
  • cl-unification is a full blown unification machine for Common Lisp objects and "templates"; not as fast as a simpler pattern matcher (like several listed in CLiki Pattern Matching), but very general (it also relies on CL-PPCRE).
  • cl-enumeration was the first full blown Java-like enumeration/iterator library for Common Lisp; it works.
  • defenum is another Java-ism ported to Common Lisp; it essentially creates enums that work like Java, i.e., with quite a bit of underlying machinery.  Of course you do not have Haskell data definitions available, but it was fun to write.
  • with-contexts was a bout of Python-envy, but then again, I think I was able to show that you really can program Common Lisp in a fun way.

Literary Programming in Common Lisp

One final library I am quite fond of expresses my love of (certain) books.  And to deal with books, you need a Library and a Librarian.  Hence the OOK compiler in Common Lisp.  Don't say Monkey!

Not Done Yet...

Although I am dabbling with Julia (a Common Lisp in a drag) and Rust (I have a day job which also involves programming languages and not only Cancer Research) I still have a lot of "not-quite-there" stuff I wrote in Common Lisp.
All I can say about it is... stay tuned,  I do put out stuff and I do pester people about Common Lisp.  You may find some of the stuff interesting.



(cheers)





20210320

HEΛPing ASDF

 ... more fixing and, ça va sans dire, more creeping features.

I got prodded to integrate HEΛP with other tools; mostly, of course, ASDF.  A simple solution was to define a document-op for a system.  After jumping through a few hoops, the solution was to use the :properties of a system to pile up arguments for the main HEΛP document function (well, only one for the time being).  Bottom line, suppose you have:

  (asdf:defsystem "foosys"
     :pathname #P"D:/Common Lisp/Systems/foosys/")

now you just issue

  (asdf:operate 'hlp:document-op "foosys")

and the documentation for the system "foosys" will appear in the "docs/html/" subfolder.

If you want to pass a title to the document function, you set up your system as:

  (asdf:defsystem "foosys"
     :properties (:documentation-title "The FOO Omnipotent Tool")
     :pathname #P"D:/Common Lisp/Systems/foosys/")

and the parameter will be used (instead of the bare system name).

It works! 😁

Some more fixing and more extensions may be needed (hlp:document takes a lot of parameters) but it is already usable.

All the necessary bits and pieces are in the HEΛP repository, and they should get into Quicklisp in the next release.

Enjoy


(cheers)