The LETV and LETV* Macros

Hello,

It has been a long time since I posted something here.  I have been busy with my day job and bogged down in a major rewrite of something (more on this hopefully very soon now (tm)) that is full of rabbit's holes.

I was able to get out of one of these rabbit's holes with this little hack I cooked up that allows you to, possibly, write more concise code.

This little hack introduces two handy Common Lisp macros, LETV and LETV* that allow you to mix regular LET and MULTIPLE-VALUE-BIND forms in a less verbose way. The amount of indentation needed is also reduced.

The syntax of LETV (or LETV*) is very "loopy", with a nod to SML/OCaml/F#/Haskell/Julia. The current syntax is the following:

    letv     ::= 'LETV' [vars] [IN [body]]
    letvstar ::= 'LETV*' [vars] [IN [body]]

    vars     ::= var | var vars

    var      ::= ids '=' <form> [decls]

    decls    ::= decl | decl decls
    decl     ::= OF-TYPE idtypes

    ids      ::= <symbol> | '(' <symbol> + ')'
    idtypes  ::= <type designator> | '(' <type designator> + ')'

    body     ::= [<declarations>] <form> * 

(I know: the grammar is not completely kosher, but I trust you will understand it.)

The two macros expand in forms that mimic the semantics of let, LET* and MULTIPLE-VALUE-BIND. All type declarations, if present, are properly handled via locally forms. LETV expands into a LET, with variable initialized by PSETQs and multiple-VALUE-SETQs. LETV* expands into a form that is an interleaving of LET* and MULTIPLE-VALUE-BIND.

The library exports only the two symbols LETV and LETV*. The other "symbols" mentioned (=, IN, OF-TYPE) are checked as in LOOP; therefore you can use different styles to write your code, as you would when writing LOOPs.

The library is available here.

Examples

1. Simple case:

     (letv x = 42 in (format t "The answer is ~D.~%" x))
     

2. Same with declaraion:

     (letv x = 42 of-type fixnum
   	in (format t "The answer is ~D." x))
     

3. Simple case with MULTIPLE-VALUE-BIND:

     (letv (v found) = (gethash 'the-key *my-hash-table*)
           in (if found
                  (format t "Found THE-KEY, doing stuff.~%")
                  (error "THE-KEY not found.")))
     

4. Mixing things up:

     (letv (v found) = (gethash 'the-key *my-hash-table*)
                     of-type (fixnum boolean)
            x = 42
            in (if found
                    (format t "Found THE-KEY, adding to the answer ~D.~%"
                            (+ v x))
                    (error "THE-KEY not found.")))
     

5. With LETV*:

     (letv* (v found) = (gethash 'the-key *my-hash-table*)
                      of-type (fixnum boolean)
            x = (when found (+ v 42))
            in (if found
                   (format t "Found THE-KEY, adding to the answer ~D.~%"
   			 	        x)
                   (error "THE-KEY not found.")))
     

6. The other way around.

     (letv x = 42 of-type integer
           (v found) = (gethash 'the-key *my-hash-table*)
                     of-type (fixnum boolean)
           in (if found
                  (format t "Found THE-KEY, adding to the answer ~D.~%"
                          (+ v x))
                  (error "THE-KEY not found.")))
     

7. A more compelling example.

     (letv* (p found) = (gethash 'the-key *points-table*) of-type (point)
            (x y) = (if found
                        (unpack-point p)
                        (values :missing :missing))
            in
            (declare (type point p)
                     (type real x y))  ; Adding declarations here also works.
   
            (do-stuff-with x y)
            (do-things-with p))
     

All the examples are meant to illustrate the use of LETV and LETV*.

Notes

LETV and LETV* are obviously not the first macros of this kind floating around: others are available and all have their niceties. But I never claimed not to suffer from NIH syndrome.

LETV and LETV* do not do "destructuring" or "pattern matching". That is a different can of worms; but you can check cl-unification for a library (always by yours truly) that provides facilities in that sense.

---

'(Cheers)

CDR is Next...

 Hi

as many may know, I have been nursing (almost to death!) the Common Lisp Document Repository (CDR).  Pascal Costanza et al., started the project many years ago and then I sat on top of it for many more.

I finally found some time to work on it and the result is a revamped site (cdr.common-lisp.dev) with the addition of stashing documents in a Zenodo Community (CDR), which has the benefit of producing a DOI for each write-up.

Moreover, Pascal Bourguignon, Michał "phoe" Herda and Didier Verna have agreed to become CDR editors.  Many thanks to them.

So, if anyone wants to submit a "specification" for something of interests to the CL community, she/he is most welcome to do so.  Just remember that good specifications are not so easy to write.

My List of Common Lisp Libraries and SW 2022 (Plus a Couple of Other Things)

Hello

I have been bumping on a few "lists of Common Lisp libraries and tools" written by many people.  I feel like rantin... pardon, blogging about this state of affairs.

First of all we have CLiki, which is a rather comprehensive list of CL libraries and whatnot, and then we have a few, unnamed, "state of the CL ecosystem", "preferred list of Common Lisp libraries", etc, etc.

I have nothing against people blogging or making lists of course, but I tend not to make generalized statements.  Especially in order to avoid disrespecting some people's work, just by not knowing of its existence.  This is a hint.

So, in order to proceed with my ran... blog post, here is the list of CL libraries I use.  Turns out there is quite a bit of NIH syndrome here, but I never claimed not to suffer from it.  Also remember that I have a Mac, a Windows and a Linux system at hand.  I always try to have stuff that works on all of them.

Distribution Systems

I use, obviously, Quicklisp.  It works.  It can be improved, but hey!  Let's just give Zach our love! 💓 (And money!)

Implementations

• I work mostly on Lispworks.  I have the luck to be able to afford the Enterprise edition (or better, my funding does) and I am very happy with it.  The folks at Lispworks know that I can be a pest (environments? rounding modes?) but they are just doing a fantastic job.
• I use, of course, SBCL.  The implementation is rock solid and it has all the bells and whistles you need for a world class system.  Recompiling and deploying for SBCL usually uncovers bugs and potential pitfalls in your code.  I have a small rant about it though.  Guys, do not write code that "works on SBCL" and just assume all is fine. It is not.
• I have also CMUCL installed and I use it as well.  It still has a few good things to it.
• Next I use Armed Bear Common Lisp, an excellent testbed for checking portability.
• Of course, I have the free edition of Allegro CL: the other excellent commercial alternative.
• Last but not least (with memories going back decades) I have CCL installed and always ready to use.
• I am very intrigued by CLASP, but I admit, I have not had the strength to install it (too much of a production; remember that I try to get things working on three platforms); any help will be appreciated.
• Same for the latest incarnation of ECL.  Sorry guys, I will get back to installing it soon.
• Finally, we always have CLisp: old but good.

I know I forgot many.  Apologies for my senescence.

System Building

If you checked any of my libraries, you will have noticed that I still have .system files everywhere.  I use ASDF of course, but I have been nursing (with Madhu) a version of MK:DEFSYSTEM; it is clunkier (and old) internally, but I feel it has a simpler and more straightforward interface, especially when it comes to its footprint on your system (i.e., somewhat simpler working of the registry).

Concurrency

I use Bordeaux Threads for portability.  There are a few open issues about it that are of very difficult solution but it is Good Enough (it has a couple of warts: a bad .asd file for example - but this is a separate rant, pardon, blog post).  Of course I also like very much Lispworks multiprocessing library, especially mailboxes, but I understand that not everybody has access to it.

Testing

I must say that none of the libraries I checked does ONE thing I really need: running something and stopping after a timeout (see the open issue I mentioned just before for Bordeaux Threads).

Having said that, I use FiveAM; I have to muck a bit around it to get the timeout I need, and the documentation, especially about macr... pardon, fixtures is lacking, but it does the job.  I am experimenting with Parachute, but have not made the switch yet.

Portability

Most of my libraries try to be portable.  One major library that helps you a lot with this thankless job ("let's have filename case insensitive anyone?") is of course UIOP.  You have it pretty much by default, but it would be nice if it were eventually decoupled from ASDF.

Another little library I advocate (yes, Virginia, here they come!) is CLAD: it does not do much, but it gives you a bedrock upon which to build your work.

Editing

I use the Lispworks IDE or the Editor (as in "thou shalt not have other Editor than...") with SLIME.  End of story.

HTML Generation

Here I start with the rantin..., thoughtful exposition of various libraries.

To generate (X)HTML (or HTML5) I use (X)HTMΛ.  It is a pretty solid library with a couple of bells and whistles: mainly, it has an object model and it leverages the sorrily underleveraged Common Lisp pretty printer to produce readable HTML.  The web page for (X)HTMΛ is produced using itself as a building block; see below.

Web Server and Client

In this case I stick with the tried and true Hunchentoot and its counterpart Drakma, although for simpler, one-shot things I do use Lispworks facilities when needed.

Documentation Generation

As I said, all my libraries suffer from a severe case of NIH, which is reflected in my use of the hacked up HEΛP documentation system.  It is still not perfect, but, again, by using (X)HTMΛ and, again, leveraging the pretty printer, I find that the results are quite pleasant (for the neapolitans: ogni scarrafone...).  Of course, if you agree to format your doc string in a Hyperspec way.  How can it be improved?  Four things.

1. Incorporating Eclector as a "reader" (as of now the tool uses more than a kludge to get past the non standardized way of dealing with READER- and PACKAGE- errors).
2. Finishing the HTML5 generation scheme.
3. Adding Texinfo generation (as per Didier Verna's declt library).
4. Fixing cross-referencing, especially with standard CL (cfr., Hyperspec).

Mathematics

Why did I start HEΛP?  Many years ago (or many Kgs ago) a post on comp.lang.lisp stated that you could shadow +, -, *, and / in a package and make them generic functions (as an aside, I recently found out that the idea of generics dates back to PL/I - but this is a different rabbit's hole).

The result was what is now a forever embryonic Common Math, which led to my forays in "let's do what R can do" -- Ρ (greek Rho) -- and to the stuff I am talking about in "Why you cannot write an Interval Arithmetic Library...".  All of this another rabbit hole: bottom line, this is what I am working on and off right now, apart from my day job.

Other Libraries

Laziness as a Way of Life

Of course, rabbits' holes are never linear.  They branch in several directions at once.  E.g., once you look at other "modern" languages (I am partial to Haskell) you want to have some of these features in Common Lisp as well); hence the CLAZY library (laziness as a way of life).

Code Handling

Needless to say, CLAZY eventually needed some proper code walking, or better it needed some proper Abstract Syntax Tree (AST) handling, which led to the CLAST library: this last library allows you to code walk and to properly inspect a piece of Common Lisp code, striving very hard to portably manage the environments that were present in CLtL2 but did not make it in the ANSI spec.

Programming Common Lisp

As we all know, Lisp is like a ball of mud.  Over the years I threw a lot of mud at it.

Some of the mud I liked the most is embedded in the following libraries

• definer is a small hack to have Pythonesque def available in Common Lisp, of course, in an extensible way (documentation being updated soon).
  
• cl-unification is a full blown unification machine for Common Lisp objects and "templates"; not as fast as a simpler pattern matcher (like several listed in CLiki Pattern Matching), but very general (it also relies on CL-PPCRE).
  
• cl-enumeration was the first full blown Java-like enumeration/iterator library for Common Lisp; it works.
  
• defenum is another Java-ism ported to Common Lisp; it essentially creates enums that work like Java, i.e., with quite a bit of underlying machinery.  Of course you do not have Haskell data definitions available, but it was fun to write.
  
• with-contexts was a bout of Python-envy, but then again, I think I was able to show that you really can program Common Lisp in a fun way.

Literary Programming in Common Lisp

One final library I am quite fond of expresses my love of (certain) books.  And to deal with books, you need a Library and a Librarian.  Hence the OOK compiler in Common Lisp.  Don't say Monkey!

Not Done Yet...

Although I am dabbling with Julia (a Common Lisp in a drag) and Rust (I have a day job which also involves programming languages and not only Cancer Research) I still have a lot of "not-quite-there" stuff I wrote in Common Lisp.

All I can say about it is... stay tuned,  I do put out stuff and I do pester people about Common Lisp.  You may find some of the stuff interesting.

---

(cheers)

HEΛPing ASDF

 ... more fixing and, ça va sans dire, more creeping features.

I got prodded to integrate HEΛP with other tools; mostly, of course, ASDF.  A simple solution was to define a document-op for a system.  After jumping through a few hoops, the solution was to use the :properties of a system to pile up arguments for the main HEΛP document function (well, only one for the time being).  Bottom line, suppose you have:

  (asdf:defsystem "foosys"
     :pathname #P"D:/Common Lisp/Systems/foosys/")

now you just issue

  (asdf:operate 'hlp:document-op "foosys")

and the documentation for the system "foosys" will appear in the "docs/html/" subfolder.

If you want to pass a title to the document function, you set up your system as:

  (asdf:defsystem "foosys"
     :properties (:documentation-title "The FOO Omnipotent Tool")
     :pathname #P"D:/Common Lisp/Systems/foosys/")

and the parameter will be used (instead of the bare system name).

It works! 😁

Some more fixing and more extensions may be needed (hlp:document takes a lot of parameters) but it is already usable.

All the necessary bits and pieces are in the HEΛP repository, and they should get into Quicklisp in the next release.

Enjoy

---

(cheers)

Need more HEΛP?

Just a quick note for people following these... parentheses.

I have carved out some time to do some more Lisp hacking and this lead me to look at the very nice usocket library (I want to do some network programming).  The usocket library documentation page has a bit of an "old" and "handcrafted" look and feel to it, so I tried to produce a version of the documentation with help from my HEΛP library.

Well, it turns out that usocket has some more than legitimate code within it that my HEΛP library was not handling; even worse, it unearthed a bug in the Lambda List parsing routines.

As an example, usocket uses the following idiom to set some of the documentation strings.

    (setf (documentation 'fun 'function) "Ain't this fun?") 

This is perfectly fine, but it needed some extra twist to get HEΛP do what is, IMHO, the right thing: in this case it meant ensuring that the lambda list of the function was properly rendered in the final documentation.

Apart from that, a few not so nice buglets were exposed in the code parsing lambda lists.  The result is that now the logic of that piece of code is simpler and somewhat cleaner.

So, if you want to get HEΛP to document your Common Lisp code, give it a spin.

---

(cheers)

With what are we contextualizing?

Common Lisp programmers may write many with-something overt their careers; the language specification itself is ripe with such constructs: witness with-open-file. Many other libraries also introduce a slew of with- macros dealing with this or that case.

So, if this is the case, what prevents Common Lisp programmers from coming up with a generalized with macro?

It appears that the question has been answered, rather satisfactorily, in Python and Julia (at least). Python offers the with statement, alongside a library of "contexts" (Python introduced the with statement in 2005 with PEP 343) and Julia offers its do blocks.

In the following I will present WITH-CONTEXTS, a Common Lisp answer to the question. The library is patterned after the ideas embodied in the Python solution, but with several (common) "lispy" twists.

Here is the standard - underwhelming - example:

      (with f = (open "foo.bar") do
         (do-something-with f))
    

That's it as far as syntax is concerned (the 'var =' being optional, obviously not in this example; the syntax was chosen to be loop-like, instead of using Python's as keyword). Things become more interesting when you look under the hood.

Traditional Common Lisp with- macros expand in variations of unwind-protect or handle-case (and friends). The example above, if written with with-open-file would probably expand into something like the following:

      (let ((f nil))
         (unwind-protect
              (progn
                 (setq f (open "foo.bar"))
                 (do-something-with f))
            (when f (close f))))
    

Python generalizes this scheme by introducing a enter/exit protocol that is invoked by the with statement. Please refer to the Python documentation on contexts and their __enter__ and __exit__ methods.

The "WITH" Macro in Common Lisp: Contexts and Protocol

In order to introduce a with macro in Common Lisp that mimicked what Python programmers expect and what Common Lisp programmers are used to some twists are necessary. To achieve this goal, a protocol of three generic functions is provided alongside a library of contexts.

The ENTER/HANDLE/EXIT Context Protocol

The WITH-CONTEXTS library provides three generic functions that are called at different times within the code resulting from the expansion of the onvocation of the with macro.

• enter: this generic function is invoked when the with macro "enters" the context; its main argument is the result of the expression that is the argument of the with macro.
• handle: this generic function is called to take care of exceptional situations that may arise during the call to enter or during the execution of the body of the with macro.
• exit: this generic function is called to "clean up" before exiting the context entered by means of the with macro.

Given the protocol (from now on referred to as the "EHE-C protocol"), the (undewhelming) "open file" example expands in the following:

      (let ((f nil))
	(unwind-protect
	    (progn
	      (setq f (enter (open "contexts.lisp")))
	      (handler-case (open-stream-p f)
		(error (#:ctcx-err-e-41883)
		  (handle f #:ctcx-err-e-41883))))
	  (exit f)))
    

Apart from the gensymmed variable the expansion is pretty straightforward. The function enter is called on the newly opened stream (and is essentially an identity function) and sets the variable. If some error happens while the body of the macro is executing then control is passed to the handle function (which, in its most basic form just re-signals the condition). Finally, the unwind-protect has a chance to clean up by calling exit (which, when passed an open stream, just closes it).

One unexpected behavior for Common Lisp programmers is that the variable (f in the case above) escapes the with constructs. This is in line with what Python does, and it may have its uses. The file opening example thus has the following behavior:

    CL-prompt > (with f = (open "contexts.lisp") do
                    (open-stream-p f))
    T

    CL-prompt > (open-stream-p f)
    NIL
  

To ensure that this behavior is reflected in the implementation, the actual macroexpansion of the with call becomes the following.

      (let ((#:ctxt-esc-val-41882 nil))
        (multiple-value-prog1
	    (let ((f nil))
	      (unwind-protect
		  (progn
		    (setq f (enter (open "contexts.lisp")))
		    (handler-case
			(open-stream-p f)
		      (error (#:ctcx-err-e-41883)
			(handle f #:ctcx-err-e-41883))))
		(multiple-value-prog1
		    (exit f)
		  (setf #:ctxt-esc-val-41882 f))))
	  (setf f #:ctxt-esc-val-41882)))
    

This "feature" will help in - possibly - porting some Python code to Common Lisp.

"Contexts"

Python attaches to the with statement the notion of contexts. In Common Lisp, far as the with macro is concerned, anything that is passed as the expression to it, must respect the enter/handle/exit. protocol. The three generic functions enter, handle, exit, have simple defaults that essentially let everything "pass through", but specialized context classes have been defined that parallel the Python context library classes.

First of all, the current library defines the EHE-C protocol for streams. This is the strightforward way to obtain the desired behavior for opening and closing files as with with-open-file.

Next, the library defines the following "contexts" (as Python does).

• null-context:
  this is a full "pass through" context, just encapsulating the expression passed to it.
• managed-resource-context:
  this is a first cut implementation of a "managed resource", which implements also an acquire/release protocol; of course, this would become more useful in presence of mutltiprocessing (see notes in Limitations).
• redirect-context:
  this is a context that redirects output to a different stream.
• suppress-context:
  this is a context that selectively handles some conditions, while ignoring other ones.
• exit-stack-context:
  this is a context that essentially allows a programmer to manipulate the "state of the computation" within it body and combine other "contexts"; to achieve its design goal, it leverages the functions of a protocol comprising the enter-context, push-context, callback, pop-all and unwind (this is equivalent to the Python close() context method).

This should be a good enough base to start working with contexts in Common Lisp. It is unclear whether the Python decorator interface would provide some extra functionality in this Common Lisp implementation of contexts and the with macro.

Limitations

The current implementation has a semantics that is obviously not the same as the corresponding Python one, but it is hoped that it still provided useful functionality. There are some obvious limitations that should be taken into account.

The current implementation of the library does not take into consideration threading issues. It could, by providing a locking-context based on a portable multiprocessing API (e.g., bordeaux-threads).

The Python implementation of contexts relies heavily on the yield statement. Again, the current implementation does not provide similar functionality, although it could possibly be implemented using a delimited continuation library (e.g., cl-cont).

Disclaimer

The code associated to these documents is not completely tested and it is bound to contain errors and omissions. This documentation may contain errors and omissions as well. Moreover, some design choices are recognized as sub-optimal and may change in the future.

License

The file COPYING that accompanies the library contains a Berkeley-style license. You are advised to use the code at your own risk. No warranty whatsoever is provided, the author will not be held responsible for any effect generated by your use of the library, and you can put here the scariest extra disclaimer you can think of.

Repository and Downloads

The with-contexts library is available on Quicklisp (not yet).

The with-contexts library. is hosted at common-lisp.net.

The git repository can be gotten from the common-lisp.net Gitlab instance in the with-macro project page.

---

(cheers)

Iron handling (with Emacs Lisp)

At the beginning of the pandemic I stumbled upon an article regarding the problems that the State of New Jersey was having in issuing relief checks and funding due to the lack of ... COBOL programmers.  At the time I followed a couple of links, landing on this "Hello World on z/OS" blog post.  I was curious and obviously looking for something other than my usual day job, plus, I swear, I had never written some COBOL code.

What follows is a report of the things I learned and how I solved them.  If you are easily bored, just jump to the end of this (long) post to check out the IRON MAIN Emacs Lisp package.

A Foray in the Big Iron Internet

Well, to make a long story short, I eventually installed the Hercules emulator (and other ones - more on this maybe later) in its SDL/Hyperion incarnation and installed MVS on it; the versions I installed are TK4- and a "Jay Moseley" build (special thanks to Jay, who is one of the most gracious and patient people I interacted with over the Internet).  I also installed other "big iron" OSes, e.g., MTS, on the various emulators and experimented a bit (again, maybe I will report on this later).

It has been a lot of fun, and I discovered a very lively (if grizzled) community of enthusiasts, who mostly gathers around a few groups.io groups, e.g., H390-MVS.  The community is very helpful and, at this point, very similar, IMHO, to the "Lisp" communities out there, if you get my drift.

Anyway, Back to hacking

One way to interact with "the mainframe" (i.e., MVS running on Hercules) is to write your JCL in your host system (Linux, Windows, Mac OS) and then to submit it to a simulated card reader listening over a socket (port 3505, which is meaningful to the IBM mainframe crowd).  JCL code is interesting, as is the overall forma mentis that is required to interact with the mainframe, especially for somebody who was initially taught UNIX, saw some VMS and a few hours of Univac Exec 8. In any case, you can write your JCL, where you can embed whole Assembler, COBOL, Fortran, PL/I etc code, using some editor on Windows, Linux or Mac OS etc.

Of course, Lisp guys know that there is one Editor, with its church. So, what one does is to list-all-packages and install jcl-mo...  Wait...

To the best of my knowledge, as of December 2020, there is no jcl-mode to edit JCL code in Emacs.

It immediately became a categorical imperative to build one, which I did, while learning a bit of Emacs Lisp, that is, all the intricacies of writing modes and eventually post them on MELPA.

Writing the IRON MAIN Emacs Lisp Package

Writing a major mode for Emacs in 2020 is simple in principle, but tricky in practice, especially, if, like me, you start with only a basic knowledge of the system as a user.

One starts with define-derived-mode and, in theory, things should be relatively easy from there on.  The first thing you want to do is to get your font-lock-mode specifications right.  Next you want to add some other nice visual tools to your mode.  Finally you want to package your code to play nice with the Emacs ecosystem.

Font Lock

Font Lock mode (a minor mode) does have some quirks that make it a bit difficult to understand without in depth reading of the manual and of the (sparse) examples one finds over the Internet.  Of course, one never does enough RTFM, but I believe a few key points should be reported here.

Font Lock mode does two "fontification" operations/passes.  At least this seem the way to interpret them.

1. A search based one: where "keywords" are "searched" and "highlighted" (read: they are rendered according to the face declared for them).
2. A syntax table one: where fontification is performed based on properties set for a given character in a syntax table.

To interact with Font Lock, a mode must eventually set the variable font-lock-defaults.  The specification of the object contained in this variable is complicated.  This variable is eventually a list with at least one element (the "keywords"); the optional second one controls whether the syntax table pass (2) is performed or not. I found that the interaction between the first two elements must be carefully planned.  Essentially you must decide whether you want only the search based ("keyword") fontification or the syntax table based (2) fontification too.

If you do not want the syntax table based (2) fontification then you want to have the second element of font-lock-defaults set to non-NIL.

The first element of font-lock-defaults is where most of the action is.  Eventually it becomes the value of the variable font-lock-keywords that Font Lock uses to perform search based fontification (1).  The full range of values that font-lock-keywords may assume is quite rich; eventually its structure is just a list of "fontificators". There are two things to note however, which I found very useful.

First, Font Lock applies each element of font-lock-keywords (i.e., (first font-lock-defaults)) in order.  This means that a certain chunk of text may be fontified more than once.  Which brings us to the second bit of useful information.

Each element that eventually ends up in font-lock-keywords may have the form

(matcher . subexp-highlighter)
where subexp-highligther = (subexp facespec [override [laxmatch]])

(see the full documentation for more details).

Fontification is not applied to chunks of text that have already been fontified, unless override is set to non-NIL.  In this case the current fontification is applied.  This is very important for things like strings and comments, which may interact in unexpected ways, unless you are careful with the order of font-lock-keywords.

I suggest you download and use the wonderful library font-lock-studio by Anders Lindgren to debug your Font Lock specifications.

Ruler mode

When you write lines, pardon, cards for MVS or z/OS it is nice to have a ruler to count on that tells you at what column you are (and remember that once you hit column 72 you'd better... continue).  Emacs has a built in nice little utility that does just that: a minor mode named ruler-mode, which shows a ruler in the top row of your buffer.

There is a snag.

Emacs counts columns from 0.  MVS, z/OS and friends count columns from 1.  Popping up the ruler of ruler-mode in a buffer containing JCL (or COBOL, or Fortran) shows that you are "one off": not nice.

Almost luckily, in Emacs 27.x (which is what I am using) you can control this behavior using the variable column-number-indicator-zero-based, which is available when you turn on the minor mode column-number-mode. Its default is t, but if you set it to nil, the columns in the buffer will start at 1, which is "mainframe friendly".  Alas, this change does not percolate (yet - it needs to be fixed in Emacs) to ruler-mode, which insists on counting from 0.

End of story: some - very minor - hacking was needed to fix the rather long "ruler function" to convince it to count columns from 1.

Packaging

Is there a good way to do this?

It appears that most Emacs "packages" are one-file affairs.  The package I wrote needs to be split up in a few files, but it is unclear (remember that I never do enough RTFM) how to keep thinks together for distribution, e.g., on MELPA or, more simply in your Emacs load-path.

What I would like to achieve is to just do a load (or a load-library) of a single file that caused the loading of the other bits and pieces.  It appears that Emacs Lisp does not have an ASDF or a MK:DEFSYSTEM as you have in Common Lisp (I will be glad to be proven wrong), so, as my package is rather small after all, I resorted to writing a main file that is named after the library and which can be thus referenced in the -pkg.el file that Emacs packaging requires.  I could have used use-package, but its intent appear to be dealing with packages that are already "installed" in your Emacs environment.

MELPA comes with it recipes format to register your package; it is a description of your folder structure and it is useful, but it is something you need to submit separately to the main site, let me add, in a rather cumbersome way. Quicklisp is far friendlier.

One other rant I have with the Emacs package distribution sites (e.g., MELPA and El-Get) is that eventually they assume you are on UN*X (Linux) and require you to have installed bits and pieces of the traditional UN*X toolchain (read: make) or worse.  I am running on W10 these days and there must be a better way.

Bottom line: I created a top file (iron-main.el) which just sets up a few things and requires and/or loads the other files that are part of or needed by the package.  One of the files contains the definition of a minor mode called iron-main-mode (in an eponymous .el file).

I am wondering whether this is the best way of doing things in Emacs Lisp.  Please tell me in the comments section.

The IRON MAIN Emacs Lisp Package

At the end of the story, here is the link to the GitHub repository for the IRON MAIN Emacs package to interact with the mainframe.

As you see the package is rather simple.

It is essentially three files plus the "main" one and a few ancillary ones.

• iron-main.el: the main "loader" file.
  
• iron-main-mode.el: the minor mode invoked by the other major modes defined below.
  
• jcl-mode.el: a major mode to handle JCL files (pardon, datasets).
  
• asmibm-mode.el: a major mode to handle IBM Assemblers.
  

One of the nice things I was able to include in jcl-mode is the ability to submit the buffer content (or another .jcl file, pardon, dataset) to the mainframe card reader listening on port 3505 (by default, assuming such a card reader has been configured).

This turns out to be useful, because it allows you to avoid using netcat, nc.exe or nc64.exe, which, at least on W10, always trigger Windows Defender.  Plus everything remains integrated with Emacs.  Remember: there's an Emacs command for that!

To conclude here are two screenshots (one "light", one "dark") of a test JCL included in the release. Submitting it from Emacs to  TK4- and to a "Jay Moseley's build" seems to work pretty well.  Just select the Submit menu under JCL OS or invoke the submit function via M-x.

What's next?  A few things apart from cleaning up, like exploring polymode; after all, embedding code in JCL is not unheard of.

That's it.  It has been fun and I literally learned a lot of new things.  Possibly useful.

If you are a mainframe person, do jump on the Emacs bandwagon.  Hey, you may want to write a ISPF editor emulator for it  😏😄