Bootstrapping a Forth in 40 lines of Lua code (2008)

An interesting curiosity about Forth is the fact that the final stage of FreeBSD kernel bootstrapping, the loader program, contains a Forth interpreter. If you have a look in the /boot directory on a FreeBSD system, you will see quite a few files named with the extension "4th" there, and in the loader(8) man page we find among other things the following:

loader [...] provides a scripting language that can be used to automate tasks, do pre-configuration or assist in recovery procedures. This scripting language is roughly divided in two main components. [...] The bigger component is an ANS Forth compatible Forth interpreter based on FICL, by John Sadler.

[...]

BUILTINS AND FORTH

All builtin words are state-smart, immediate words. If interpreted, they behave exactly as described previously. If they are compiled, though, they extract their arguments from the stack instead of the command line.

If compiled, the builtin words expect to find, at execution time, the following parameters on the stack:

addrN lenN ... addr2 len2 addr1 len1 N

where addrX lenX are strings which will compose the command line that will be parsed into the builtin's arguments. Internally, these strings are concatenated in from 1 to N, with a space put between each one.

If no arguments are passed, a 0 must be passed, even if the builtin accepts no arguments.

While this behavior has benefits, it has its trade-offs. If the execution token of a builtin is acquired (through ' or [']), and then passed to catch or execute, the builtin behavior will depend on the system state at the time catch or execute is processed! This is particularly annoying for programs that want or need to handle exceptions. In this case, the use of a proxy is recommended. For example:

: (boot) boot;

FICL

FICL is a Forth interpreter written in C, in the form of a forth virtual machine library that can be called by C functions and vice versa.

In loader, each line read interactively is then fed to FICL, which may call loader back to execute the builtin words. The builtin include will also feed FICL, one line at a time.

The words available to FICL can be classified into four groups. The ANS Forth standard words, extra FICL words, extra FreeBSD words, and the builtin commands; the latter were already described. The ANS Forth standard words are listed in the STANDARDS section. The words falling in the two other groups are described in the following subsections.

https://www.freebsd.org/cgi/man.cgi?query=loader&apropos=0&sektion=8&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html ↗

I’ve been in a “retro computing” deep dive lately and it’s been fascinating to discover Forth. I was surprised that an entirely novel paradigm was hidden despite much digging over many years. All of the sudden I am seeing it everywhere now too - confirmation bias from growing up skill wise in the age of dot com boom vs right before?

I like the feeling of concatenative languages. I especially seem to like the postfix notation over prefix and largely this creates the paradigm of lisp vs forth at first step - in my opinion.

I’ve recently dug in to Factor and it’s quite an amazing piece of art that is highly usable and up to date despite such a small community. It almost feels like a secret weapon and too good to be true. I’m having a ton of fun hacking on it. It’s like Smalltalk and forth had a baby.

I dream of making an OS with Forth/Factor similar to Plan9 and Oberon.

I get a kick out of Forth being made of words and Genesis from the Bible.

At first was the Word, and the Word was with Forth, and the Word was Forth.

https://factorcode.org/ ↗

https://en.wikipedia.org/wiki/Oberon_ ↗(operating_system)

Tidbit if anyone cares: I’ve been obsessing over returning to first principles of computing. I’m bored of the internet and browsers. I want to have fun with software AND hardware and Forth seems like such a perfect language to do exactly that with. Factor is a nice grown up example of that but it’s definitely a few steps removed from portability and self bootstrapping behavior like in CollapseOS.

Retro Forth is extremely impressive: http://forth.works ↗

If you want to understand Forth, I recommend this article:

A sometimes minimal FORTH compiler and tutorial for Linux / i386 systems by Richard W.M. Jones

"LISP is the ultimate high-level language, and features from LISP are being added every decade to the more common languages. But FORTH is in some ways the ultimate in low level programming. Out of the box it lacks features like dynamic memory management and even strings. In fact, at its primitive level it lacks even basic concepts like IF-statements and loops.

Why then would you want to learn FORTH? There are several very good reasons. First and foremost, FORTH is minimal. You really can write a complete FORTH in, say, 2000 lines of code. I don't just mean a FORTH program, I mean a complete FORTH operating system, environment and language. You could boot such a FORTH on a bare PC and it would come up with a prompt where you could start doing useful work. (...)

Secondly FORTH has a peculiar bootstrapping property. By that I mean that after writing a little bit of assembly to talk to the hardware and implement a few primitives, all the rest of the language and compiler is written in FORTH itself. Remember I said before that FORTH lacked IF-statements and loops? Well of course it doesn't really because such a lanuage would be useless, but my point was rather that IF-statements and loops are written in FORTH itself."

https://github.com/nornagon/jonesforth/blob/master/jonesforth.S ↗

Something I like about this article is how it helps reveal what Forth's essence is:

Forth is an extensible virtual machine

Most interpreted languages that are implemented with a virtual machine (sometimes known as a bytecode interpreter) have a fixed set of instructions. Modulo some implementation details, writing new Forth words is akin to extending the virtual machine with new instructions.

However, the article's Forth is a more extensible form of this. The entire evaluation of the virtual machine is split into an extensible set of states (starting with only interpret, compile, head, forth, lit), new types of instructions can be introduced by adding to a list of heads (starting with only DOCOL and EXIT), and then words are, specifically, lists of addresses that the forth state interprets, whose evaluation is triggered by a DOCOL seen in the head state.

This opens up possibilities to define other sorts of interpreters for specific purposes. The article gives the example of polynomial evaluation, but you could also do something like have a variant of the forth state for token threaded code in memory-constrained environments. Sort of like how ARM has an additional Thumb instruction set.

More detail for this last idea: you'd have an array of, say, 256 pointers, and in the bforth state you'd read the next byte of the thread, look up the address in the table, push it on the RS, then go into the head state. This gives you perfect interoperability. You'd just need a special bcompiler state that would look up the bytecode for each word.

Yea! An informed, intelligent discussion about my now-favourite language. I've been near fanatical about it for the last 3-4 years mainly using Mecrisp Stellaris on the STM32F4xx. I find the elegant simplicity and raw power so refreshing after all the bloatware and nanny-knows-best typed code. Even the stack juggling is absorbing puzzle-solving fun. I've successfully built OLED and MIDI drivers from scratch and am now tackling DSP SIMD assembly FFT routines - stuff I would never normally attempt without this experience. It harks back to the simpler days of the 1980s yet it is really is only limited by one's approach - given some thought, you can adapt it to just about any programming paradigm - should you actually need it, that is. I'm probably not currently considered a full Forthwright yet not having built myself a Forth from scratch (yet). I know Forth is not perfect, but it's the closest I've found yet and deserves a renaissance, if it withered on the vine that would be a travesty - it's too good for that.