Iron handling (with Emacs Lisp)

At the beginning of the pandemic I stumbled upon an article regarding the problems that the State of New Jersey was having in issuing relief checks and funding due to the lack of ... COBOL programmers.  At the time I followed a couple of links, landing on this "Hello World on z/OS" blog post.  I was curious and obviously looking for something other than my usual day job, plus, I swear, I had never written some COBOL code.

What follows is a report of the things I learned and how I solved them.  If you are easily bored, just jump to the end of this (long) post to check out the IRON MAIN Emacs Lisp package.

A Foray in the Big Iron Internet

Well, to make a long story short, I eventually installed the Hercules emulator (and other ones - more on this maybe later) in its SDL/Hyperion incarnation and installed MVS on it; the versions I installed are TK4- and a "Jay Moseley" build (special thanks to Jay, who is one of the most gracious and patient people I interacted with over the Internet).  I also installed other "big iron" OSes, e.g., MTS, on the various emulators and experimented a bit (again, maybe I will report on this later).

It has been a lot of fun, and I discovered a very lively (if grizzled) community of enthusiasts, who mostly gathers around a few groups.io groups, e.g., H390-MVS.  The community is very helpful and, at this point, very similar, IMHO, to the "Lisp" communities out there, if you get my drift.

Anyway, Back to hacking

One way to interact with "the mainframe" (i.e., MVS running on Hercules) is to write your JCL in your host system (Linux, Windows, Mac OS) and then to submit it to a simulated card reader listening over a socket (port 3505, which is meaningful to the IBM mainframe crowd).  JCL code is interesting, as is the overall forma mentis that is required to interact with the mainframe, especially for somebody who was initially taught UNIX, saw some VMS and a few hours of Univac Exec 8. In any case, you can write your JCL, where you can embed whole Assembler, COBOL, Fortran, PL/I etc code, using some editor on Windows, Linux or Mac OS etc.

Of course, Lisp guys know that there is one Editor, with its church. So, what one does is to list-all-packages and install jcl-mo...  Wait...

To the best of my knowledge, as of December 2020, there is no jcl-mode to edit JCL code in Emacs.

It immediately became a categorical imperative to build one, which I did, while learning a bit of Emacs Lisp, that is, all the intricacies of writing modes and eventually post them on MELPA.

Writing the IRON MAIN Emacs Lisp Package

Writing a major mode for Emacs in 2020 is simple in principle, but tricky in practice, especially, if, like me, you start with only a basic knowledge of the system as a user.

One starts with define-derived-mode and, in theory, things should be relatively easy from there on.  The first thing you want to do is to get your font-lock-mode specifications right.  Next you want to add some other nice visual tools to your mode.  Finally you want to package your code to play nice with the Emacs ecosystem.

Font Lock

Font Lock mode (a minor mode) does have some quirks that make it a bit difficult to understand without in depth reading of the manual and of the (sparse) examples one finds over the Internet.  Of course, one never does enough RTFM, but I believe a few key points should be reported here.

Font Lock mode does two "fontification" operations/passes.  At least this seem the way to interpret them.

1. A search based one: where "keywords" are "searched" and "highlighted" (read: they are rendered according to the face declared for them).
2. A syntax table one: where fontification is performed based on properties set for a given character in a syntax table.

To interact with Font Lock, a mode must eventually set the variable font-lock-defaults.  The specification of the object contained in this variable is complicated.  This variable is eventually a list with at least one element (the "keywords"); the optional second one controls whether the syntax table pass (2) is performed or not. I found that the interaction between the first two elements must be carefully planned.  Essentially you must decide whether you want only the search based ("keyword") fontification or the syntax table based (2) fontification too.

If you do not want the syntax table based (2) fontification then you want to have the second element of font-lock-defaults set to non-NIL.

The first element of font-lock-defaults is where most of the action is.  Eventually it becomes the value of the variable font-lock-keywords that Font Lock uses to perform search based fontification (1).  The full range of values that font-lock-keywords may assume is quite rich; eventually its structure is just a list of "fontificators". There are two things to note however, which I found very useful.

First, Font Lock applies each element of font-lock-keywords (i.e., (first font-lock-defaults)) in order.  This means that a certain chunk of text may be fontified more than once.  Which brings us to the second bit of useful information.

Each element that eventually ends up in font-lock-keywords may have the form

(matcher . subexp-highlighter)
where subexp-highligther = (subexp facespec [override [laxmatch]])

(see the full documentation for more details).

Fontification is not applied to chunks of text that have already been fontified, unless override is set to non-NIL.  In this case the current fontification is applied.  This is very important for things like strings and comments, which may interact in unexpected ways, unless you are careful with the order of font-lock-keywords.

I suggest you download and use the wonderful library font-lock-studio by Anders Lindgren to debug your Font Lock specifications.

Ruler mode

When you write lines, pardon, cards for MVS or z/OS it is nice to have a ruler to count on that tells you at what column you are (and remember that once you hit column 72 you'd better... continue).  Emacs has a built in nice little utility that does just that: a minor mode named ruler-mode, which shows a ruler in the top row of your buffer.

There is a snag.

Emacs counts columns from 0.  MVS, z/OS and friends count columns from 1.  Popping up the ruler of ruler-mode in a buffer containing JCL (or COBOL, or Fortran) shows that you are "one off": not nice.

Almost luckily, in Emacs 27.x (which is what I am using) you can control this behavior using the variable column-number-indicator-zero-based, which is available when you turn on the minor mode column-number-mode. Its default is t, but if you set it to nil, the columns in the buffer will start at 1, which is "mainframe friendly".  Alas, this change does not percolate (yet - it needs to be fixed in Emacs) to ruler-mode, which insists on counting from 0.

End of story: some - very minor - hacking was needed to fix the rather long "ruler function" to convince it to count columns from 1.

Packaging

Is there a good way to do this?

It appears that most Emacs "packages" are one-file affairs.  The package I wrote needs to be split up in a few files, but it is unclear (remember that I never do enough RTFM) how to keep thinks together for distribution, e.g., on MELPA or, more simply in your Emacs load-path.

What I would like to achieve is to just do a load (or a load-library) of a single file that caused the loading of the other bits and pieces.  It appears that Emacs Lisp does not have an ASDF or a MK:DEFSYSTEM as you have in Common Lisp (I will be glad to be proven wrong), so, as my package is rather small after all, I resorted to writing a main file that is named after the library and which can be thus referenced in the -pkg.el file that Emacs packaging requires.  I could have used use-package, but its intent appear to be dealing with packages that are already "installed" in your Emacs environment.

MELPA comes with it recipes format to register your package; it is a description of your folder structure and it is useful, but it is something you need to submit separately to the main site, let me add, in a rather cumbersome way. Quicklisp is far friendlier.

One other rant I have with the Emacs package distribution sites (e.g., MELPA and El-Get) is that eventually they assume you are on UN*X (Linux) and require you to have installed bits and pieces of the traditional UN*X toolchain (read: make) or worse.  I am running on W10 these days and there must be a better way.

Bottom line: I created a top file (iron-main.el) which just sets up a few things and requires and/or loads the other files that are part of or needed by the package.  One of the files contains the definition of a minor mode called iron-main-mode (in an eponymous .el file).

I am wondering whether this is the best way of doing things in Emacs Lisp.  Please tell me in the comments section.

The IRON MAIN Emacs Lisp Package

At the end of the story, here is the link to the GitHub repository for the IRON MAIN Emacs package to interact with the mainframe.

As you see the package is rather simple.

It is essentially three files plus the "main" one and a few ancillary ones.

• iron-main.el: the main "loader" file.
  
• iron-main-mode.el: the minor mode invoked by the other major modes defined below.
  
• jcl-mode.el: a major mode to handle JCL files (pardon, datasets).
  
• asmibm-mode.el: a major mode to handle IBM Assemblers.
  

One of the nice things I was able to include in jcl-mode is the ability to submit the buffer content (or another .jcl file, pardon, dataset) to the mainframe card reader listening on port 3505 (by default, assuming such a card reader has been configured).

This turns out to be useful, because it allows you to avoid using netcat, nc.exe or nc64.exe, which, at least on W10, always trigger Windows Defender.  Plus everything remains integrated with Emacs.  Remember: there's an Emacs command for that!

To conclude here are two screenshots (one "light", one "dark") of a test JCL included in the release. Submitting it from Emacs to  TK4- and to a "Jay Moseley's build" seems to work pretty well.  Just select the Submit menu under JCL OS or invoke the submit function via M-x.

What's next?  A few things apart from cleaning up, like exploring polymode; after all, embedding code in JCL is not unheard of.

That's it.  It has been fun and I literally learned a lot of new things.  Possibly useful.

If you are a mainframe person, do jump on the Emacs bandwagon.  Hey, you may want to write a ISPF editor emulator for it  😏😄

 

(cheers)

MA