in these days of coding agents and what-not, i often think of gerald sussman's comment that one no longer constructs systems from known parts, that instead one does basic science on the functionality of foreign libraries; he was right then and i hate it as much as i did 16 years ago

I started drafting a reply, but it quickly began to spiral into a full-blown essay. It turns out that I have a lot of related thoughts that I've been meaning to get down for a while.

From the linked blog post:

... Costanza asked Sussman why MIT had switched away from Scheme for their introductory programming course, 6.001. This was a gem. He said that the reason that happened was because engineering in 1980 was not what it was in the mid-90s or in 2000. In 1980, good programmers spent a lot of time thinking, and then produced spare code that they thought should work. Code ran close to the metal, even Scheme — it was understandable all the way down. Like a resistor, where you could read the bands and know the power rating and the tolerance and the resistance and V=IR and that's all there was to know. 6.001 had been conceived to teach engineers how to take small parts that they understood entirely and use simple techniques to compose them into larger things that do what you want.

But programming now isn't so much like that, said Sussman. Nowadays you muck around with incomprehensible or nonexistent man pages for software you don't know who wrote. You have to do basic science on your libraries to see how they work, trying out different inputs and seeing how the code reacts. This is a fundamentally different job, and it needed a different course.

(For a primary source, here's a video of Sussman answering the same question at a different event, starting at 59:35.)

I have really weird feelings about this assertion. It is obviously true that the software stack that most of the world runs on is a towering mess of leaky abstractions, a sprawling mass of haphazardly-interconnected components that nobody could claim to fully understand. It's also often the case that documentation is wildly inadequate, and experiment can sometimes be the simplest route to clearer understanding.

But the implication is almost that we can't understand the components we're building on top of, that this way of working died in the 90s and now the job of programming is overwhelmingly just poking at stuff until it seems to work, and that's just the complete opposite of my personal experience.

I was born in 1982, and started programming pretty much as soon as I could read. For many years starting out I really didn't understand how anything worked, and things got more complicated much faster than I could make sense of them. I started out programming BASIC on 8-bit computers like the VIC-20 and Apple II; programming meant sequencing commands that made Things Happen, and if BASIC didn't have a command for it, I couldn't do it. I knew assembly language existed, but that was a different universe; when I would occasionally find myself dropped into the Apple II machine code monitor, the only thing I could figure out to do with it was frantically mash Ctrl+Reset to attempt to get back to a BASIC prompt again.

Later I got a 286, and I found myself straining against the limitations of QBASIC. It had more commands, which meant I could do more things, but it wasn't enough for me. I managed to get my hands on a copy of Microsoft QuickC, assuming that my experience with QBASIC would transfer over to this more powerful language that surely had far more commands that would let me do more things, but was dismayed to discover that the help files somehow didn't reveal any graphics commands at all!

Eventually I got access to BBSes, and found some programming tutorials that explained how VGA worked, and even got my hands on a shareware BASIC compiler that let me try some of these ideas out. I still didn't feel like I really understand what was going on, but I had names for concepts at least, and I was able to get pixels on the screen. I even wrote a terrible paint program to create those pixels with, because parsing real graphics formats was beyond me, but I could manage “width, height, pixels”.

Then I got a 486 and the internet and suddenly if I wanted to use any newer tools, none of the pieces I had struggled so hard to understand worked anymore. The old stuff was still there, but it was buried under a whole new layer of complexity. I spent so much time totally lost, my accumulated knowledge repeatedly crumbling to dust, the Next Important Thing I was supposed to learn hopelessly out of reach. I would get books on assembly language or win32 and they would fill me with despair. I would read people posting in newsgroups about DPMI, and how you couldn't make BIOS calls without switching between real mode and protected mode, and wonder how the fuck anyone was supposed to get a pixel on the screen.

This was the mid-90s; about the timeframe when Sussman would have stopped teaching 6.001. The paraphrasing of the blog post is a bit less clear, but in the video Sussman is explicit that this is exactly the timeframe where he saw engineering changing in fundamental ways. This was not easy to deal with as a teenager who was trying to learn this stuff!

All I wanted in the world was to mess around with stuff until it sort of worked. Understanding was for chumps. I didn't want to think about the problem space, or the messy realities of my platform of choice. I wanted the computer to Do Thing. I wanted libraries and languages with a simple face, that would solve problems for me without me having to think them through, because the amount of shit that didn't make sense to me was so overwhelming. I just wanted it to be easy.

I would like to share with you a personal accomplishment from this era. I was proud of it then. I am not proud now, but remembering it helps me remember what things were really like for me, when I didn't have the skills and mindset I have now.

DJGPP and Allegro had gotten me back into the comfortable world of using Commands that Did Things to make games. I was using Scream Tracker to write music, and I wanted to use the popular Mikmod library to put that music into my games. There was a companion library called “Mikalleg” which took care of connecting the sound output routines to Allegro's audio engine, and did some preprocessor trickery to work around the fact that both Mikmod and Allegro defined a struct called SAMPLE.

I found it all kind of complicated to use. Mikmod / Mikalleg tended to provide flexible APIs that had a lot of confusing parameters. I didn't want to have to understand what all that stuff was; I couldn't understand why they didn't just give you a function that started playing the music you told it to play. So I figured out some values that worked, and wrote wrapper functions that removed all that flexibility and just did exactly the thing that I assumed everyone would want.

This wasn't my only innovation. Mikalleg didn't interoperate with Allegro's facility for compressed data files. I wanted it to. (Real Games didn't just leave music files lying around that you could easily listen to or modify!) Mikmod had a generic I/O interface, so that you could override the file loading routine by providing “read” and “write” calls. It looked complicated, so I ignored it. Instead I wrote a function that would take the data in the Allegro file – which would have been decompressed and loaded into RAM at this point – write it to a file on whatever disk you happened to be using, tell Mikmod to load the file, and then delete it. I knew this probably wasn't an ideal solution, but gosh, it worked great! And it was so easy to do! Why would I ever want to spend so much effort to do things “right”?

I proudly packaged this all up into a library I called “Easymik”. I made a webpage for it and everything. The Mikalleg author even prominently linked to it from the Mikalleg home page! People were desperate for a solution to this datafile problem, and I built one!

I wanted, so badly, to be a Real Programmer. I knew what tools Real Programmers used; they used C and assembly. I knew what Real Programs looked like; there was an .exe file, and DOS4GW for some reason, and special data files in special formats, never .S3M files that I could listen to, never .PCX files that I could open in my paint programs, never text files that I could look at and edit. I didn't know why things were this way, just that they were, so the closer I could get to that, the more Real my programs became, the more likely it was that I would be Good At It, that I could make the computer do anything and everything I could imagine.

If you had given me a magic box that I could ask to write programs for me, that generated code that I didn't understand, that sort of worked but might have weird problems, that I could pester with questions about esoteric technical subjects until it gave me reasonable-sounding-but-maybe-wrong answers that were on my level, I would have been delirious with joy. I would have shaken the devil's hand, weeping with gratitude, and leapt face-first into vibe coding with a ferocity you could scarcely imagine. Sure, it's a bit shit, but all of the resources I had access to were shit.

I could have gotten stuck that way. I could have flailed around, not understanding, not wanting to understand, for many more years than I did. But I think there were a few events sent me on a better trajectory.

The first event was that I improbably landed a programming job working with QNX while I was still in high school. By that time I'd learned enough C to be dangerous; I'd mostly figured out how to use pointers, and could use malloc and free even though I couldn't have told you the difference between the stack and the heap. (I have a vivid sense memory of the panic and terror I felt when a coworker casually dropped one of those terms while talking to me about something.)

QNX was everything MS-DOS and Windows were not; clear, elegant, understandable, reliable. I read Rob Krten's book and said to myself, “of course this is how an operating system should work.” I never managed to write a win32 program, but I could wrap my head around Photon. It proved to me that things didn't need to be so awful; that you could make a great many things wildly easier to understand simply by finding a good design.

The only problem was I didn't know how to find a good design. I wanted to know, so badly, but the advice I was reading didn't make any sense to me; lots of people just asserting that you should build things a certain way, follow some arbitrary rules, often contradicting each other, usually in vague terms I didn't really understand. I got very interested in alternative programming languages and tools – DSLs, Lisp, Forth – stuff that had a tangible form, but was spoken about as though it was magic. Surely they'd cracked the problem.

At my next job, I felt like I was ready to try out some of the concepts I was reading about. I had just finished the unenviable task of patching dozens of bespoke tools to support a new peripheral. These tools had all been haphazardly built by cloning the last one and swizzling its code to support the new product; at its core there were really only two use cases. I proposed a single, unified codebase to replace them all. I spent months building a generic tool-building toolkit, infinitely configurable via XML. Solving 80% of the problem was reasonably straightforward, but the last 20% had me doing absurd contortions like defining a bidirectional algebra for specifying the structure of a serial number string. I was undaunted; I powered through and ended up with something that worked pretty well for the first use case, even if writing the XML was a little bit tricky sometimes.

When I extended the system to handle the second use case, though, it became apparent that I had begun to run afoul of Greenspun's tenth rule. I had defined a dynamic Value class that could contain Anything, so that I could dynamically define the data schemas for binary log files in XML, rather than C++. This ended up taking a few hundred kilobytes of logs, and running my beefy development PC completely out of RAM just trying to parse them. It was a fiasco.

I eventually managed to optimize it enough that it could ship; IIRC, I ended up giving up and hardcoding the schema. The way I'd constructed everything, this meant that everyone touching the one unified generic codebase would have to add product-specific code anyway, exactly the thing I worked for months to avoid. It was obviously inferior to its replacement. I'd worked so hard to do the elegant thing, to build a reusable system out of simple, beautiful pieces, and I had produced a huge blob of pointless complexity which had only made life harder for myself and whoever would have to work with it next.

Everything still felt so hard, but I was more convinced than ever that the secret to making everything easier was out there, that better tools were the answer. Propelled by a conviction that domain-specific languages were going to be the answer to everything, I moved thousands of kilometers away, temporarily immigrated to the United States, and landed an even more unlikely job where a billionaire paid me to fail to fix a tree-merging algorithm for five years.

It turns out that if you are not thinking about things the right way, it is possible to spin your wheels struggling with a single problem for a very, very, very long time.

It was at that job that, for the first time, I finally confronted the fear that had been silently driving me throughout my career – the fear of truly digging into the problem. The fear of admitting when I don't really understand something. The fear that it will all be too complicated and overwhelming to deal with. I realized, finally, that what I had secretly been yearning for all along, the dream I had spent years of my life trying to realize, that I uprooted my life to pursue, was some magical tool, some impossible technique, that would free me from the need to learn things that felt too hard and scary.

But not learning? Trying to build something without understanding it? It turns out that's so much harder.

You must understand a problem before you can solve it. That lesson is core to everything I have done since. If a component isn't working the way you expect, you have to dig in and figure out how it actually works, what it's actually doing, or you have just given yourself a problem that you refuse to understand. If the problem is of any importance, then that is a terrible mistake. I don't want to waste years of my life spinning my wheels like that ever again.

Everything gets easier once you commit to understanding how things work. The more you do it, the easier it gets. The more you learn, the more you understand, the more you can accomplish. This is the magical tool I spent half my career searching for.

I printed out the checklist from Polya's How To Solve It and started referring to it whenever I sat down to work. I never read the rest of the book. It's just a handful of questions to ask yourself, some suggestions to help make sure you have a handle on things, but it's powerful stuff. I had been slowly making progress with the tree-merging algorithm, finding bugs with an ad-hoc randomized testing system (basically QuickCheck without a shrinking pass, so every failed test case was a monster that took days to untangle). I finally got things to a point where I wasn't hitting data loss or crash bugs, but the output was still pretty far from ideal. I finally confronted some core assumptions about the tree-merge algorithm that the codebase had started with, and which I had never felt comfortable enough to question, and realized that they had been pointlessly making my job wildly more difficult. I tore the whole thing apart, the fragile thing I had spent years patching, the thing I had finally managed to get sort-of working, and rebuilt it from scratch.

It worked flawlessly.

The main thing I now bring to the table, as a software professional, is the ability to understand what's really going on. Sometimes, yes, this involves doing basic science to answer a question, but I usually find it hard to be satisfied with an answer until I have gone down to the source code of the confusing component to really understand it. This is an unreasonably effective skill that has only become more relevant over the years, not less.

Sussman correctly identifies the 90s as a time when complexity exploded in engineering, both electrical and software. But I'm not willing to grant that we have been entirely unsuccessful at taming the complexity in the intervening decades. You will never convince me that the state of affairs now, from a programmer's perspective, is worse than writing Windows 95 apps in C. The huge thing that separates the complexity of the 90s from the complexity we have now was that, in the 90s, the inner workings of everything was secret. You didn't get the source code to Windows 95, you programmed to the docs, and when it had bugs, or didn't behave the way the docs said it did, or you just didn't understand how it was supposed to work, there was virtually nothing else you could do but poke at your code until it stopped triggering the problem. Breaking out the kernel debugger and reverse engineering your operating system is not a viable approach for most people!

But this is not the same situation software developers are in today. Yes, we have huge libraries, more now than ever! But most are open source; when they don't behave as we expect, it is orders of magnitude easier to figure out why than it used to be. Most are more reliable than they were then, either by being written in languages that make writing reliable software easier, or through simply having been around for decades, with most major bugs slowly killed off through sheer force of attrition. Components worth using in 2025 usually have a coherent way in which they are designed to work, they will usually work in that way, and it is usually possible to learn what that is. There is still plenty of garbage out there, of course – you should choose your dependencies carefully! – but you are absolutely spoiled for choice in comparison to the 90s.

I did some Android UI programming about a decade ago now. I would write custom View subclasses from time to time, and every so often I would struggle with bugs where the UI layout would need to be recomputed, but it just wouldn't happen. The documentation was useless. I would try to sprinkle around even more calls to requestLayout or forceLayout, but nothing seemed to help.

Finally I got fed up and dug into the source. Turns out the way it works under the hood is that requestLayout and forceLayout just set a few flags on the view object; they don't “schedule” anything, contrary to the docs. requestLayout recursively calls itself on the parent, which is what gives the signal to Android the next time it goes to draw the screen that some stuff needs laying out again. forceLayout sets the same flags, but only on itself.

I don't know what actual use forceLayout is meant to have, because in practice, what it actually means is “if one of my descendants calls requestLayout, stop propagating that flag upwards once you reach me”. It's a method whose sole purpose seems to be to create layout bugs that are impossible to track down. I also remember there being a data race, where if you called one of those functions while layout was happening, the flags of the various views in the tree could get in funky inconsistent states, and everything would get screwed up.

After I read the code, I finally understood how Android's layout algorithm actually worked – exactly when measurement passes ran, what was triggering them, what I needed to avoid doing in them. I could rework my custom views so they wouldn't trip this problem anymore. I understood weird behaviour I had experienced in other situations. Running experiments, poking things, doing science – that got me brittle, buggy code that took forever to get working and that I was afraid to mess with. Opening up the black box gave me confidence.

It is still possible to build a complex piece of software that works out of simpler pieces of software that work. You can choose dependencies that are understandable. Programming in 2025 does not have to be about fumbling in the dark. Everything gets easier if you're willing to turn on the light.

It didn't seem like too much of a hardship. In a previous Forth I'd built, I'd implemented do / loop using the return stack, but it was... ugly. The code to implement it was ugly, the code it generated was ugly (and large!), and I didn't find a lot of places where it was actually much nicer to use than explicit begin-based loops. I was able to implement an 8086 assembler and a Minesweeper game without bothering to build do / loop. I didn't really miss it, but I had a design percolating in the back of my mind that I wanted to try.

At some point I came across some writing that suggested that Forth had a “loop control stack”. Wouldn't it be nice if I could implement some kind of loop control stack that worked for all kinds of iteration?

The thing I built has blown me away with how flexible, composable, and useful it's turned out to be. It's way more powerful than I was expecting. And the code that leverages it is inevitably much simpler and easier to read.

The Stacks

I added two loop control stacks – what I call the i-stack, and the next-stack. The i-stack contains the current value(s) being iterated over, and is read from with the i and j words like normal. The next-stack is where the magic happens.

When iterating, the top value of the next-stack is a pointer to a small structure called an iterator. It's a very simple structure, only two cells. The first cell contains the execution token of a word that will either update the current values on the i-stack and return true, or remove its state from both stacks and return false. The second cell points to a cancellation function, that cleans up whatever state the iterator has kept on the two stacks without iterating further, and returns nothing.

Iterators

I built some simple helpers for creating iterators. It took a few tries to nail down this design, but I'm happy with it now. defiter creates a “blank” iterator, which, when called, pushes itself to the next-stack. :iter does the same thing but allows you to write some code that accepts parameters and prepares the loop stacks first. :next defines a new anonymous “next-word” and assigns it to the most-recently defined iterator. :cancel does the same thing, but for cancellation.

On its own, this is already quite nice. I've got a page or so of basically-trivial iterators. Here's one:

:iter times ( n -- ) >i ;
:next <i dup 1- >i finish? ;
:cancel idrop nextdrop ;

times keeps its state in the i-stack – it initializes itself by pushing the number of times to repeat onto it. When fetching the next value, it pops the current value off the i-stack, decrements it, and pushes it back, leaving the old value on the data stack. finish? is a simple helper word that peeks at the top of the stack and runs the current cancellation function if it's false, or in this case, if we've already hit 0. Since cleaning up after an iterator is often the same job whether you're exiting early or not, this word is very handy. Explicitly defining cancellation for this iterator isn't actually necessary in my current implementation, because idrop nextdrop is common enough that I use it as the default.

each / next

I can use these iteration words (within a compiled definition) like this:

5 times each i . next
( outputs: 4 3 2 1 0 )

All the common loop types are easy to build in this system, as well as some uncommon ones:

5 10 for each i . next ( outputs: 5 6 7 8 9 )
0 10 2 for+ each i . next ( outputs: 0 2 4 6 8 )
( pchars yields pointers to each byte in a zero-terminated string )
s" hello" pchars each i b@ emit next ( outputs: hello )

Generic cancellation, of course, allows us to trivially implement break; just cancel the iteration at the top of the stack, and then jump to the each loop exit point, after next. continue is even simpler, just jump back to the top of the loop.

5 times each i 3 < if break then i . next ( outputs: 4 3 )
5 times each i 2 % if continue then i . next ( outputs 4 2 0 )

Under the hood, each just calls the “next-word” of the iterator and jumps to the end of the loop if it returns 0 – conceptually identical to begin iterate while, with next meaning the same thing as repeat. This allows for iterators that return no values.

0 times each i . next ( outputs: )

Generators

That's nice, but it's not exactly setting the world on fire; it's a fair amount of work just to end up with a few different ways of writing “for” loops in practice, that Forth systems have had forever anyway. Is it really worth the cost of this abstraction?

Turns out, absolutely, yes, it is, because you can also build generators on it, and that blows things wide open.

First, a simple example:

: 5-2-8 (( 5 yield 2 yield 8 yield )) ;
5-2-8 each i . next ( outputs: 5 2 8 )

(( defines the start of the generator, and )) defines the end (and pushes it onto the next-stack). Any valid Forth code goes in between. yield takes the top of the stack, pushes it onto the i-stack, and then suspends the generator until the next iteration. How does this work? Essentially, yield takes the top of the return stack and pushes it onto the next-stack, then pushes an iterator that pops it off the next-stack and pushes it back onto the return stack. The details get a little messier in order to support some more advanced use cases, but that's the simple idea at the core of it.

OK, neat trick, we've built ourselves a nice little coroutine-like system. But wait! It gets better! When yield resumes, it immediately removes all of its state from the iteration stacks. This means that generators can safely interact with any iterator that might be “underneath” it. They can iterate over things and yield in the middle! They can yield different things based on those values! We've accidentally built an extremely powerful, totally generic map/filter capability!

: doubled (( each i i + map next )) ;
5 times doubled each i . next ( outputs: 8 6 4 2 0 )
: odd (( each i 2 % filter next )) ;
5 times odd each i . next ( outputs: 3 1 )

map and filter are more yield-like words – it turns out that there's a number of these that you might want to implement, with different logic for suspending, resuming, and cancelling. map saves the top of the i-stack onto the next-stack and replaces it with the input, restoring the original value after resuming (necessary since the iterator underneath might be using that value as its state). filter conditionally suspends based on the top of the data stack but otherwise doesn't touch the i-stack, leaving whatever iterator is running underneath to provide the value. Both of these words push iterators with special cancel logic that knows that there is another iterator underneath, and can cancel again recursively once they've cleaned themselves up.

Generator state

This design can almost be made to work for generators that have extra state, but it's awkward and incomplete. You must ensure the data stack is clean whenever you yield, so you're forced to manually shuffle data to and from the next stack. Consider a filter that only returns values that are divisible by a certain number:

: divisible-by ( n -- ) >next 
  (( <next each i over % 0 = swap >next filter <next next drop )) ;
5 divisible-by 21 times each i . next ( ouputs: 20 15 10 5 0 )

This works, but there's so much stack noise! And it breaks down if you need to cancel, because filter has no idea that there's extra stuff on the next-stack that it needs to clear. Ideally there would be some automatic way of keeping the state of the generator on the data stack while it's running, and push it safely away when we suspend. Could there be some way to write divisible-by like this?

: divisible-by ( n -- ) >arg (( each i over % 0 = filter next drop )) ;

In fact, this code works in my implementation. The scheme to make this happen is a little bit subtle, but it can be done efficiently with a minimum of bookkeeping noise in most cases. I define a variable, gen-arg-count, that starts at zero. >arg is an immediate word that compiles a call to >next and increments that variable. Then, any time I compile a yielding word, I append the value of gen-arg-count to the instruction stream – much like lit. When suspending, the yielding word reads that value out of the instruction stream and transfers that many values from the data stack to the next-stack. Then it moves the pointer to the instruction stream from the return stack to the next-stack, and finally pushes the yielding iterator. That iterator then pulls the instruction pointer back off the next-stack to determine how many values to move from the next-stack back onto the data stack, as well as where to resume the instruction stream. Cancellation similarly can read the arg-count byte to know how many extra values to drop from the next-stack.

Generators need to ensure the data stack is empty before exiting at )). At one point I considered having )) compile the appropriate number of drop calls automatically, but in the end I decided that it's reasonable and idiomatic to expect a generator to exit with a clean stack, like any other Forth word would.

With this extension, it's trivial to write all kinds of new iterators – we could even do away with the base iterator system entirely and just express everything as generators. There are lots nice one-line definitions of times:

( 1 ) : times ( n -- ) >arg (( begin dup while 1- dup yield repeat drop )) ;
( 2 ) : times ( n -- ) >next (( <next begin dup while 1- yield> repeat drop )) ;
( 3 ) : times ( n -- ) >arg (( -arg begin dup while 1- yield> repeat drop )) ;
( 4 ) ( suspend ) ' noop ( resume ) ' noop ( cancel ) ' idrop :yield iyield
: times ( n -- ) >i (( begin i while <i 1- >i iyield repeat idrop )) ;

Definition 1 doesn't use anything I haven't already explained. The state of the iterator is managed on the data stack, and automatically shuffled back and forth from the next-stack by yield.

Definition 2 adds a new word. yield> is a yielder that moves the yielded value from the i-stack back onto the data stack when it resumes, instead of dropping it. The state of the iterator starts on the next-stack but is moved to the i-stack once the iteration loop actually starts.

Definition 3 is virtually the same as 2, but demonstrates the ability to handle changes in the amount of state. -arg is an immediate word that generates no code, but decrements gen-arg-count so that you can express that you've consumed the argument and the next yield should preserve one less value on the data stack. (+arg is also defined, performing an increment, in case you generate more values on the stack than you started with.)

Definition 4 is built to keep all state on the i-stack from the beginning. Here we use :yield to define a new yielding word. I realized I hadn't built a yielder that left the i-stack alone when resuming, but would drop the value when cancelling, so I added one.

All of these options will correctly be cancelled if the code iterating over it calls break, with no special effort!

Final thoughts

With this scheme, generators always take up at least two spaces on the next-stack – one for the yielder's iterator, and one for the resume point. But if all iterators were defined as generators, and all yielding words had to be defined with :yield to ensure a uniform structure, we could just push the resume point. iterate and cancel could easily find the appropriate function pointer by looking next to the resume point for the address of the yielder and digging inside. I think this could be built in such a way that it would be basically as efficient as the existing scheme, at the cost of making the whole thing more complex to explain. It might be worth pursuing, because generators are so pleasant to read and write, and raw iterators are... less so. I basically never want to write a raw iterator besides the very basic ones that are built-in.

All the source for my Forth system is available online; the iteration system is defined in iter.jrt. There are some interesting examples of generators in embed.jrt, dialer.jrt and rick.jrt – some highlights:

• rle-decode – takes a pointer to some run-length encoded packed data, yields a stream of values. Uses the times iterator internally to count off the repeated values.
• menu-options – Provides a dynamic list of items to display in a menu. Yields 2 values at a time – the text to display, and the function to execute when the user selects it.
• xmit-iter – Writes text to the screen with a small delay between each character, to simulate a slow serial connection. An extremely simple loop that can be driven by complex generation logic – including streaming RLE-encoded data with embedded colour information.

#forth #code #essays

Because... functions should be re-entrant by default! They should clean up after themselves! Global variables are bad and must be avoided at all costs! Functions should be “pure” and take all of their inputs as parameters, avoiding hidden dependencies!

All of these ideas of what “good code” looks like are wrong in Forth.

It is actually extremely common for Forth words to rely on implicit context, which is globally accessible through other Forth words. This is often how you build DSLs!

Perhaps you are familiar with the JavaScript canvas API. It's based on PostScript, as are most vector drawing APIs, and PostScript, as you may know, is a Forth-like postfix language for printed graphics. The canvas API has a bunch of implicit state. When you draw a rectangle, for example, you pass in just the position and size. If you want to specify properties like the fill colour, stroke colour, stroke width, line cap style, and on and on and on, you call setter methods before calling the draw function. If you want to preserve the previous canvas state and return to it when you're done, you can explicitly push it onto a stack.

This is one secret sauce to writing small Forth words – you build little vocabularies that all work with some kernel of shared state.

Let's implement Bresenham's line algorithm

I had the idea to implement an algorithm where juggling all of the state on the stack would be a nightmare, to show an example of what this looks like in practice. I've always found Bresenham's line-drawing algorithm kind of awkward – most implementations in C switch between several nearly-identical code blocks depending on how steep the line is. But the core idea is actually very simple, and the awkward near-duplication of the standard C implementation does not have to be reproduced in Forth.

First we will define a simple textual canvas vocabulary:

80 CONSTANT SCREEN-W 
24 CONSTANT SCREEN-H
CREATE SCREEN SCREEN-W SCREEN-H * ALLOT
CREATE SCREEN-BRUSH KEY + C,

: SET-BRUSH ( -- ) KEY SCREEN-BRUSH C! ;
: FILL-SCREEN ( -- ) SCREEN-W SCREEN-H * SCREEN + SCREEN DO I SCREEN-BRUSH C@ SWAP C! LOOP ;
: SCREEN-XY ( x y -- ptr ) SCREEN-W * + SCREEN + ;
: PLOT-XY ( x y -- ) SCREEN-XY SCREEN-BRUSH C@ SWAP C! ;
: PRINT-ROW ( y -- ) 0 SWAP SCREEN-XY SCREEN-W TYPE ;
: PRINT-SCREEN SCREEN-H 0 DO I PRINT-ROW CR LOOP ;

This is ANS Forth – my personal Forths have all been lowercase, I don't usually like all the shouting.

This creates a buffer called SCREEN that is 80 columns wide by 24 rows tall. It also defines the concept of a brush, which is just an ASCII character that is put into this buffer by PLOT-XY. Our line-drawing routine will use PLOT-XY to put “pixels” on the “screen” without caring about what they look like. Kind of a canvassy idea.

Now let's clear the screen:

SET-BRUSH +
FILL-SCREEN 
SET-BRUSH $

I use the + character for “off” and the $ character for “on” because they were about the same width in the variable-width font that my browser picked when plugging this code into jsForth. The trick where SET-BRUSH reads the next character in the code directly is cute but brittle; it only works interactively and will break weirdly in a : definition. WAForth can't handle it at all, it pops up a dialog box asking for you to type a character. Feel free to use 43 SCREEN-BRUSH C! to draw with + and 36 SCREEN-BRUSH C! to draw with $ if you want to follow along in WAForth. Define little helper words for them even, like BRUSH-+ and BRUSH-$. It's not a big problem, don't overthink it, but do make yourself comfortable.

An aside: How to draw a line

So let's talk for a minute about how Bresenham's line-drawing algorithm works. The Wikipedia article has a bunch of math and symbols but at its core it's really very simple. Start with a specific kind of line, that slopes upwards and to the right, but not steeper than 45 degrees.

1. Start at the bottom-left side of the line. Draw that pixel.
2. Move your X coordinate one to the right. Now you need to decide if the Y coordinate needs to move up one or stay where it is.
3. To do that, you keep track of a subpixel fraction; ie. you start in the middle of a pixel (0.5), and increment it by the amount that the line has risen over the last pixel: (y2-y1)/(x2-x1) or dy/dx.
4. If the fraction is >1, move Y up one pixel and subtract 1 from the fraction; the fraction value is now somewhere within the bottom half of the next highest pixel.
5. Now draw the next pixel and go back to step 2 until you end up at the top-right end of the line.

This is very simple! We then layer on just a few simple tricks:

• Instead of always moving along the X axis, for lines that are taller than they are long, we need to move along the Y axis. To do this we simply always move in the direction of the longer side, and run the decision logic along the shorter axis. This way the slope is never steeper than 45 degrees.
• If, for example, the line slopes down instead of up, when we decide whether to move along the Y axis, we need to move down one pixel instead of up. We can handle this by simply incrementing instead of decrementing along the appropriate axis.
• In the olden days, floating point numbers were very slow and integers were fast. Since the “error” value (really a fractional pixel location, but everyone calls it “error”) always has the same denominator, and we don't do anything more complicated than adding more fractions with the same denominator to it, we can just keep the denominator implicit and store the numerator in an integer. We choose 2 * dx (when x is the long axis) as the denominator so that we can easily start exactly on a half pixel (ie. our starting value is dx/2dx, and we increment by 2 * dy every step). It doesn't actually make a huge amount of difference what you use for a starting value though, as long as it's smaller than your implicit denominator then you'll end up with a line that starts and ends where you expect.

That's it! That's the whole thing.

Now back to writing Forth

So, first off, let's define the state that we'll need. Starting and ending X and Y coordinates, the current X and Y coordinates, and the fractional “error” value. Definitely need to remember all that.

VARIABLE LINE-X1 VARIABLE LINE-Y1 
VARIABLE LINE-X2 VARIABLE LINE-Y2
VARIABLE LINE-X  VARIABLE LINE-Y  VARIABLE LINE-ERR

Now we can start defining helper words. Let's write a couple of words to figure out the length of the line along each axis:

: LINE-DX ( -- dx ) LINE-X2 @ LINE-X1 @ - ;
: LINE-DY ( -- dy ) LINE-Y2 @ LINE-Y1 @ - ;

No sweat; just take x2 - x1 or y2 - y1. How about some words to decide which axis is longer, and what direction each axis is moving in?

: X-LONGER? ( -- f ) LINE-DX ABS LINE-DY ABS > ;
: LINE-LEFT? ( -- f ) LINE-DX 0 < ;
: LINE-UP? ( -- f ) LINE-DY 0 < ;

Even if you're not well-practiced reading postfix, I hope it's pretty clear what these are doing.

Now let's define some words for incrementing or decrementing, depending on which direction the line is going:

: LINE-XINC ( x -- x ) LINE-LEFT? IF 1- ELSE 1+ THEN ;
: LINE-YINC ( y -- y ) LINE-UP? IF 1- ELSE 1+ THEN ;
: LINE-INC ( x|y x? -- x|y ) IF LINE-XINC ELSE LINE-YINC THEN ;

LINE-INC is our first and only word to take two values on the stack – the top is a boolean that determines if we're talking about the X or Y axis. We will soon use it in conjunction with X-LONGER? to abstract away incrementing the “long” vs. “short” axis.

: LINE-LONG ( -- p ) X-LONGER? IF LINE-X ELSE LINE-Y THEN ;
: LINE-SHORT ( -- p ) X-LONGER? 0= IF LINE-X ELSE LINE-Y THEN ;
: LINE-LONG-INC! ( -- ) LINE-LONG @ X-LONGER? LINE-INC LINE-LONG ! ;
: LINE-SHORT-INC! ( -- ) LINE-SHORT @ X-LONGER? 0= LINE-INC LINE-SHORT ! ;

LINE-LONG-INC! is a little tricky, so let's walk through it:

• LINE-LONG returns a pointer to either the LINE-X or LINE-Y variable.
• @ fetches the current coordinate along the long axis.
• X-LONGER? pushes “true” onto the stack if X is the long axis (and thus the X coordinate is what's on the stack)
• LINE-INC calls LINE-XINC if X is long, or LINE-YINC if Y is long. This increments or decrements the value, depending on the direction of the line. The new coordinate is the one value left on the stack.
• LINE-LONG ! fetches the appropriate pointer again and stores the new value.

LINE-SHORT-INC! is basically the same, except with an 0= in there as a “logical not” for X-LONGER?. (It didn't quite seem worthwhile to define Y-LONGER? on its own.)

Now let's define some useful words for the error / fractional pixel calculation:

: LINE-LONG-LEN ( -- l ) X-LONGER? IF LINE-DX ELSE LINE-DY THEN ABS ;
: LINE-SHORT-LEN ( -- l ) X-LONGER? IF LINE-DY ELSE LINE-DX THEN ABS ;
: LINE-LONG-ERR ( -- err ) LINE-LONG-LEN 2 * ;
: LINE-SHORT-ERR ( -- err ) LINE-SHORT-LEN 2 * ;
: LINE-INIT-ERR! ( -- ) LINE-LONG-LEN LINE-ERR ! ;
: LINE-ERR-ACC ( -- err ) LINE-ERR @ LINE-SHORT-ERR + ;

LINE-INIT-ERR! defines the initial error value as half a pixel (with LINE-LONG-ERR being the implicit denominator). LINE-ERR-ACC fetches the current error and adds the appropriate fraction along the short axis, leaving the new value on the stack.

: LINE-ERR-INC! ( err -- err ) DUP LINE-LONG-ERR >= IF LINE-LONG-ERR - LINE-SHORT-INC! THEN ;
: LINE-ERR-ACC! ( -- ) LINE-ERR-ACC LINE-ERR-INC! LINE-ERR ! ;
: LINE-STEP ( -- ) LINE-LONG-INC! LINE-ERR-ACC! ;

LINE-ERR-INC! takes the incremented error value, determines if we've overflown the fraction into the next pixel, and if so, decrements the error value and increments the coordinate along the short axis. The updated error value is left on the stack. This is the only place in the algorithm where I chose to use a stack-manipulation word. I could have gotten by without it by just calling LINE-ERR-ACC a couple of times, but it would have made the definition longer and arguably harder to follow.

LINE-ERR-ACC! handles accumulating the error, incrementing the short axis if necessary, and storing the new error. Finally, LINE-STEP puts all the core logic together – increment along the long axis, then decide whether we need to increment along the short axis.

All that's left is to run it in a loop:

: PLOT-LINE-STEP ( -- ) LINE-X @ LINE-Y @ PLOT-XY ;
: DO-LINE ( -- ) LINE-INIT-ERR! LINE-LONG-LEN 0 DO PLOT-LINE-STEP LINE-STEP LOOP PLOT-LINE-STEP ;

: LINE ( x1 y1 x2 y2 -- ) 
  LINE-Y2 ! LINE-X2 ! DUP LINE-Y ! LINE-Y1 ! DUP LINE-X ! LINE-X1 ! DO-LINE ;

The final definition of LINE takes four values on the stack and immediately puts them into variables that are used by all the other words.

IMO, this is what Forth enthusiasts mean when they say things like “write lots of small definitions”, or “the stack shouldn't need to be very deep”, or “you don't need local variables”. There are 24 one line function definitions up there. No individual definition is particularly complicated or hard to read. We do virtually no stack manipulation.

Let's see it in action!

0 0 0 15 LINE
0 0 15 15 LINE
30 15 0 0 LINE
60 15 0 0 LINE
79 7 0 0 LINE
79 7 60 15 LINE
0 15 60 15 LINE

PRINT-SCREEN

$$$$$$++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$$$$$$$$$$$$$$$$$+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$+$+$$+$$$$++++++$$$$$$$$$$$$+++++++++++++++++++++++++++++++++++++++++++++++++++
$++$++$$+++$$$$++++++++++++++$$$$$$$$$$$++++++++++++++++++++++++++++++++++++++++
$+++$+++$$+++++$$$$+++++++++++++++++++++$$$$$$$$$$$+++++++++++++++++++++++++++++
$++++$++++$$+++++++$$$$++++++++++++++++++++++++++++$$$$$$$$$$$$+++++++++++++++++
$+++++$+++++$$+++++++++$$$$++++++++++++++++++++++++++++++++++++$$$$$$$$$$$++++++
$++++++$++++++$$+++++++++++$$$$+++++++++++++++++++++++++++++++++++++++++++$$$$$$
$+++++++$+++++++$$+++++++++++++$$$$+++++++++++++++++++++++++++++++++++++++++$$++
$++++++++$++++++++$$+++++++++++++++$$$$+++++++++++++++++++++++++++++++++++$$++++
$+++++++++$+++++++++$$+++++++++++++++++$$$$++++++++++++++++++++++++++++$$$++++++
$++++++++++$++++++++++$$+++++++++++++++++++$$$$++++++++++++++++++++++$$+++++++++
$+++++++++++$+++++++++++$$+++++++++++++++++++++$$$$+++++++++++++++$$$+++++++++++
$++++++++++++$++++++++++++$$+++++++++++++++++++++++$$$$+++++++++$$++++++++++++++
$+++++++++++++$+++++++++++++$$+++++++++++++++++++++++++$$$$+++$$++++++++++++++++
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lovely!

Now, there is plenty to criticize about this code. It does all kinds of redundant recalculation that in any sane C implementation would have been stashed away into a local, for example. But that's fixable with a little more effort; I might do another blog post where I apply some of Forth's fun metaprogramming tricks to that problem.

#forth #essays #code

HERE BE SPOILERS. You've been warned.

So, the ending. What does it mean?

I am not interested in the question of “what 'really' happened” – that's a boring question, and the game takes great pains to make clear that the answer is not important. What I am interested in is, “what is the game trying to say?”

Return to Monkey Island touches on a lot of themes and asks a lot of questions, but I'm going to focus on two:

• What is The Secret of Monkey Island?
• What makes Guybrush any better than LeChuck?

The game's answer to the first question is clear: it is not important. It could be anything. Feel free to imagine whatever makes you happiest. This is what every character in the game who speculates about The Secret does, and this is what the game invites the player to do at the end.

The second question – the game doesn't call explicit attention to it very often, but it runs from the beginning of the game to the very end. The first real dialog tree of the game is Guybrush going through all the ways that he believes LeChuck has wronged him, while the lookout doesn't seem particularly convinced. You're both pirates, after all. He's after exactly the same thing as you. What makes you the hero and him the villain?

He's an evil zombie ghost pirate, terror of the seas and sometimes the land! A nefarious, conniving, murdering scallywag! A depraved, ruthless villain! And a loud talker! And I'm... not those things!

But. You take a spot on LeChuck's crew anyway. You have casual, low-stakes conversations with him. You fill out LeChuck's paperwork. He's... just a pirate, who wants exactly the same thing as you, for similarly vague reasons.

On Monkey Island, you are forced to get even closer to LeChuck in order to break the magical voodoo lock on the map to The Secret – learning his favourite food, reading his diary, humming his theme song. In the end you're both double-crossed by Captain Madison and the new pirate leaders.

As you search for the golden keys in Part IV the game starts to make it really obvious that Guybrush's single-minded pursuit of The Secret is starting to have negative consequences on the people around him. Scenes that were played for laughs get re-contextualized as Elaine learns more about Guybrush has been up to. It's revealed that Guybrush even directly fucked up the project Elaine has been working tirelessly at for... however long it takes to turn an entire island into a grove of lime trees. A hell of a lot longer than Guybrush has been seeking out The Secret.

There is also apparently a subtle thing where the pamphlet boasting of LeChuck's accomplishment fills in as Guybrush does the same nasty stuff. The bingo card doesn't quite fill up – Guybrush can't murder anyone who gets in his way – but the game wants the player to pause and realize that Guybrush really can be quite the scoundrel. After all... he's a pirate.

Which leads us, finally, to the ending. After chasing LeChuck through an underground puzzle gauntlet back on Monkey Island once more, Guybrush walks through a door and ends up... on Amusement Park Melee Island. Stan appears and tells Guybrush it's closing time; Elaine asks if he's ready to go. An animatronic LeChuck and an animatronic Captain Lila are torturing an animatronic Locke Smith, in an endless loop, demanding the key that opens the chest that contains The Secret. There's also animatronic Otis, an animatronic Old Pirate Leader, an animatronic Widey Bones – everyone you'd expect to meet on High Street is back, but as a cardboard cutout version of themselves.

From here you can pretty much pick your ending; they all end up essentially the same, with unique little 5-second videos that play after the (unskippable!) credits roll. There are 11 of these in total. YouTube is there and it doesn't take long, but you aren't really meant to watch them all – you're meant to pick the option that's most meaningful to you and that becomes the conclusion to your Monkey Island story.

In some ways this is a nice way to acknowledge that Monkey Island is many things to many people. But... if that's all it was, there are many more narratively satisfying ways to write it. You could have had Boybrush interrupt Guybrush right before he opens the chest and let the player pick what's inside as it's revealed, for example. Instead it's a t-shirt, and it's only after Boybrush complains that the player can make something else up. Something else is happening here.

Ron Gilbert and Dave Grossman had free reign to do whatever they wanted. If they thought Guybrush should have a final confrontation with LeChuck before the rug was pulled, there would have been one. Stopping the story before it happens, making the player see a crummy secret, ending up in a cheap fun-house mirror version of the world... it all must add up to something. I couldn't see what it could be, at first.

My current reading is this: The deliberate anticlimax you, as a player, feel, as you see behind the curtain, as you easily pop open that chest – Guybrush felt it too. How could he not? Elaine warns him, after rattling off the complaints of everyone he's screwed over along the way:

It's just that I'm worried that The Secret can't possibly measure up to the effort and anticipation.

Was it worth it? No. No treasure in the world could possibly have been worth it. Nothing in that chest could have made him happy.

Confronted with that? Suddenly his quest must feel... fake. Hollow. Pointless and cheap. A trip to the amusement park that's over. Nothing left to do but to turn out the lights and go home.

Guybrush has a distinctly different character arc in different Monkey Island games. In Secret of Monkey Island, he's – how shall I put this – a little shit. Sure, he's a sweet little doe-eyed boy who thinks pirates are swell and just wants to go on an adventure, but he's also the kind of kid who will walk into someone's house and just insult the owner until he's physically thrown out. Still, he doesn't really leave anyone worse off than he found them, except for maybe accidentally sinking the Sea Monkey with a giant boulder. But that's an honest mistake; it's possible to make Guybrush avoid doing it.

In Monkey Island 2, Guybrush is, quite frankly, abusive and selfish. Everyone he meets, he throws directly under the bus. You have to steal poor Wally's monocle, without which he's blind. Get the cook fired to steal his job, which you immediately bail on. Saw off the Man of Low Moral Fiber's pegleg. Nail Stan into a coffin and leave him for dead. Get Kate Capsize arrested.

Even Elaine isn't spared. First he crashes her party and steals her grandfather's map piece. Then when he's caught, Elaine reveals that she'd dumped him. Here Guybrush can, at the players option, can endlessly neg her with quips like “I came all this way to see you – at least get me a beer” or “Gosh you're cute when you're pretending to be mad.” Eventually the player will catch on that the right strategy is to express how madly in love Guybrush still is with her; how miserable he is without her... and as soon as the walls start to come down and Elaine shows the slightest warmth towards him, he says, “Does that mean you're going to let me have the map?”

In a way, Guybrush's arc in Return to Monkey Island seems similar – his obsession with finding The Secret at all costs mirroring his obsession with finding Big Whoop at all costs. Two differences stand out.

The first is his reason for pursuing it. With Big Whoop, he wanted a new story that would command respect from his peers. He's driven by the fear that his biggest triumph is already behind him; that he has to top his last adventure in order to convince everyone that he truly is a mighty pirate, and that defeating LeChuck wasn't just a fluke. With The Secret, Guybrush is driven largely by his rivalry with LeChuck – he wants it because under no circumstances can LeChuck be allowed to find it first.

The second is his relationships with the people around him. In MI2, Guybrush cares about nobody and nobody cares about him. He screws them over without giving them a second thought, confident that soon he'll find Big Whoop and be a big shot among pirates. In RtMI, on the other hand, he greets everyone on Melee like an old friend. He's cheerful and friendly with everyone new that he encounters. He's married to Elaine and clearly still deeply in love. The destruction he leaves behind is... different. A thoughtlessness without callousness. Guybrush takes Elaine's flyer because he needs it, and it hasn't occurred to him that this might cause a huge headache for his wife – not because he doesn't care if it does or not.

But of course, intent isn't magic. Well-intentioned or not, his list of misdeeds rivals LeChuck's. “I just hope it turns out to be worth all the effort,” says Elaine (being, quite frankly, unreasonably supportive of her dipshit husband).

So. Was a stupid t-shirt worth it? The environmental devastation caused by whittling a mop handle, of all things? The earthquake that burned down the Scumm Bar? The closing of the pirate museum, where so much of his history was being reverently kept? The implosion of an island's government? The torpedoing of his own wife's tireless efforts to cure a pirate plague?

Of course not. It couldn't be.

Guybrush gets to the very end of his quest and discovers that the contents of the chest is not what was important.

What, in this story, might Guybrush decide was more important? The adventure? The story? Or... his friends? His wife, who is there waiting for him, as he turns out the lights in Amusement Park Melee Island, ready to take him home and face whatever comes next after the disappointment?

Why is there no showdown with LeChuck? Because LeChuck was never the problem. The showdown is between Guybrush and The Secret, and I think Guybrush loses, in that he doesn't get what he thought he wanted, and also, he wins, in that, maybe he's finally able to see a childish fantasy for what it is, and start to focus on what is important in his life instead.

The last line of the game is Elaine saying to Guybrush, “I found the lost map to the treasure of Mire Island. It's going to be a fun adventure.” Clearly, they will embark on it together.

#games #essays

still can't believe the shithead behind Pupkin managed to get his webcomic turned into a movie with Jennifer Lopez

Nobody responded to this in any way, because, quite frankly, nobody who wasn't an absolute fanatical follower of webcomics discourse 20 years ago has any idea what the fuck this means. Hell, I barely understand my feelings about this. But to do them justice, we are going to have to unpack 25 years of webcomics history.

The process of researching and writing this has rearranged my view of the formative years of webcomics. I've done my best to cite my sources, but primary sources are not always readily available anymore. I ended up using the Wayback Machine so much that it locked me out for a few hours. Support the Internet Archive.

Keenspot: Origins

Let's go chronologically. In January 1997, Darren Bluel started publishing a webcomic called “Nukees”. It's more or less the prototypical “me and my wacky college friends” strip, in that it stars thinly veiled versions of the author and his wacky college friends. 25 years later, it's still being regularly updated. Nukees is published with a Perl script called AutoKeen. We'll talk about AutoKeen in more depth in a bit.

In March 1999, Chris Crosby started publishing a comic called Superosity. The main characters include a lovable idiot named Chris, his hyperintelligent surfboard-shaped robot friend named Boardy, and his evil asshole brother, Bobby. Bobby is drawn with all-black eyes and if he had a catchphrase it would probably be “Shut the hell up, you stupid idiot!“

Remember Bobby. He's important.

Superosity is also regularly updated – very regularly. Literally every single day since its inception. It updated today, the day you are reading this. No vacations, no filler days, ever, for almost 23 years. There was a contest called “The Daily Grind Iron Man Challenge” where you had to update your comic on time, Monday to Friday, every day, no exceptions, and when Crosby was late with an update people were legitimately worried he had died. (Turns out a snowstorm had knocked out his power.)

Aside: I discovered while writing this that the Daily Grind Iron Man Challenge ran for fifteen years before a single victor finally claimed the prize in 2020.

By 2000 webcomics were exploding, and the place to find new comics to read was Big Panda – a giant list of comics, ranked by a peculiar algorithm. Many of the comics listed on it were also hosted there. Here's a bunch of interviews about Big Panda with webcomic artists who were there, conducted in 2006.

The algorithm was meant to reward comics for prominently linking to Big Panda (generally using a dropdown list of Big Panda comics, injected into your page with JavaScript) and encouraging their readers to check out other comics linked to by Big Panda. Yes, even in 1999, The Algorithm ranking content by engagement was there, although it was at least explicit and openly explained. Of course that didn't stop it from being mercilessly gamed and exploited.

Looking at the Wayback Machine from November 1999, I pretty much read all of these. Big Panda was where webcomics lived.

In early 2000, the creator of Big Panda lost interest in maintaining it, and the site started to go down regularly. So Darren Bluel, the creator of Nukees, Chris Crosby, the creator of Superosity, Teri Crosby, Chris' mother and the colourist of Superosity, and some dude named Nate Stone who ended up CTO but I couldn't discover much else about, launched a new webcomics portal called Keenspot, in March of 2000.

Keenspot provided a similar service to Big Panda – a one-stop portal where you could discover other webcomics, cross-promotional tools so everyone on Keenspot could boost the profile of everyone else on Keenspot, and hosting. Unlike Big Panda, however, they wouldn't just add anyone to the list – Keenspot had to sign you. And Keenspot started by signing virtually all of the most popular comics on Big Panda, which means Keenspot started with an exceptionally strong lineup.

A few months after Keenspot launched, they launched Keenspace. This was Keenspot's weird little brother; anyone could create a Keenspace account and start publishing comics. You wouldn't get the full promotional might of Keenspot, but you'd get the same hosting and publishing tools.

If your Keenspace comic was good, and updated regularly, the promise was that you might get “promoted” to Keenspot. For a time, this meant that there was prestige attached to being a Keenspot comic. Having your comic on Keenspot meant that it had Made It. People who were into webcomics paid attention. It was News.

Keenspace was the Geocities of webcomics and therefore, unsurprisingly, I loved it; a perfect example of the y2k-era web where a guy with a server and a janky perl script could provide thousands of people with a home for their creative work. It's still around, though in 2004 the name changed to Comic Genesis, I guess so it wouldn't be confused with Keenspot.

As might be obvious from the name, Keenspot / Keenspace was built on Darren Bluel's AutoKeen. Prior to AutoKeen, artists would generally roll their own HTML by hand; AutoKeen was a big improvement in tooling for its time. The way it works: artists upload specially named image files and HTML templates via FTP. AutoKeen runs nightly, generating the day's new archive pages and front page. You could upload comics in advance and they would be automatically published on the date given in the filename. The archives were accessible via little HTML calendars, built from <table>s.

Bluel even released a “lite” version of Autokeen as public domain open source.

Keenspot also wasted no time in organizing significant cross-promotional events, like, uh, Bikeeni Summer 2000, where a bunch of Keenspot cartoonists drew pinup art of their characters in swimsuits. Weird horniness of early 2000s webcomics aside, what I want to convey is that when Keenspot launched, its members were excited to join and support each other. It was a group of peers trying to make something exciting and unprecedented happen. The web was providing a new opportunity to folks who had been shut out of the old way of doing things; rejected by newspaper syndicates, producing work that could never be published in a mainstream way.

That's not to say that everyone thought everything about Keenspot was entirely keen – see, for example, keenparody, a comic (hosted on Keenspace, and which Chris Crosby linked from his Superosity news page) that portrays, among other nastiness, Darren Bluel cutting open the corpse of a panda with a chainsaw. It's, uh, not subtle. (Also CW for nudity in some of the other strips. I wasn't kidding about the weird horniness of the era.)

Enter Bobby Crosby

Now, remember Bobby? Turns out Chris Crosby has a brother in real life named Bobby. In 2002, Bobby Crosby began publishing Pupkin. Pupkin is a round orange dog, who can't seem to find his forever home.

I won't say that nobody liked Pupkin – it managed to attract some fans – but it is generally not well-remembered. Pupkin isn't particularly well-drawn or funny. The punchline of the first strip is a baby saying “Pupkin” while looking at a round orange dog. The punchline of the second strip is Pupkin worried that he might have AIDS.

It turns out that Bobby Crosby does not take criticism well. When someone writes anything negative about his comics, he shows up in the comments to yell at them. I haven't found anything from 2002 about Pupkin specifically, but here he is going absolutely nuclear on someone who rated one of his comics 3/5 stars:

Nothing that you said in your review made any sense on any level and at least half of everything you just said is a lie...

I could go on and on and on forever refuting all of your nonsense points, but I’m trying to stop wasting time doing such things and yours would be the longest of all. I mean, you’re yet another person who actually somehow thinks that ALL OF THE ZOMBIES USED TO BE VAMPIRES, as if the world was made up of seven billion vampires before this all started instead of seven billion humans. You CAN’T READ.

I don’t give a shit about word choice and tone and remaining civil. I hate everyone and everything, especially morons like you idiots who can’t read.

I recall Bobby Crosby showing up absolutely everywhere his comic was mentioned to respond to everything. If I remember right, Bobby would sign off all of his responses with “Thank you for loving Pupkin”, no matter how much the original commenter had hated Pupkin. If you google this phrase now, you'll find that this became a Something Awful meme.

By 2009 (I will link the article later), Gary of the webcomics blog Fleen is saying things like:

Bobby Crosby tossed in his two cents (at some point in the future, there may well be a “Bobby’s Law”, the point after which no useful discussion on a webcomics topic can take place).

You know you're a level-headed individual when someone decides that they need to name a variant of Godwin's Law after you.

As far as I am able to discern, Pupkin was never officially a Keenspot comic. But it had a Keenspot banner at the top, was clearly using AutoKeen, and as far as I can tell, there was never an associated Keenspace account. Chris Crosby linked to it with a prominent banner on Superosity's sidebar.

Trouble brewing

2002 is also when we really started to see some other groups outside of Keenspot try to take on webcomics. Joey Manley started Modern Tales, which took the approach of selling paid subscriptions to access comic archives. Several high-profile cartoonists quit their popular Keenspot comics to go independent – Jeff Rowland ended “When I Grow Up” and started “Wigu”, John Allison ended “Bobbins” and started “Scary Go Round”. Both were part of a loose collective called “Dumbrella”, which had a portal to some comics, but was largely a message board community site where cartoonists hung out with each other and their readerships.

By 2005, the Dumbrella wiki had this to say about Keenspot:

Keenspot is now a festering assland of a turdhole powered by Chris Crosby's gravitational pull and haunted by the tortured souls of captured Internet cartoonists.

I would wager that this was probably written by a member of the message boards, rather than an artist, but still. It's not totally obvious to an outsider exactly what was so egregious about Keenspot that caused talented artists to flee with that level of acrimony. But clearly something went very wrong.

In 2004, Keenspot moved its corporate headquarters in Cresbard, South Dakota, population 121. The people of Cresbard had decided, due to its dwindling population, to close its high school. Keenspot bought the high school on the cheap, and as far as I can tell, Chris Crosby and his mom Teri moved to rural South Dakota to run Keenspot there. There is a Superosity storyline about how terrible an idea moving to South Dakota was, posted shortly after the blizzard that knocked Crosby out of the Daily Grind.

That's weird, right? Moving your internet media company to an abandoned high school in rural South Dakota? That's kind of an unhinged thing to do?

Blatant Comics and the push for More Bobby

In 2006, Bobby Crosby launched two new comics, both now fully backed by Keenspot: +EV, a comic strip about online poker, and Last Blood, a zombie-themed graphic novel serialized online on Keenspot, published in print by Blatant Comics, and which Bobby very explicitly wants, from day one, to turn into a movie. Bobby is the writer for both of these comics, but this time is collaborating with artists. (Even Pupkin's most ardent fans would have to admit that its art is pretty amateurish.)

Wait. What's Blatant Comics?

In 2007, Blatant Comics had a handful of different books by a couple of different artists that they were advertising. They're... embarassing shlock. There's a superhero parody of The Office, there's Bobby's zombie book, there's “Impeach Bush! A Funny Li'l Graphical Novel About The Worstest Pres'dent In The History Of Forevar”, and there's “Dead Sonja”, which is a zombie parody of the Marvel hero “Red Sonja”, I guess? The company appears to be called “Blatant Comics” so that they can put “A Blatant Parody” on the cover. Their webpage displays their address in the footer... Cresbard, SD.

Blatant Comics has a Wikipedia page, which I am gonna quote from:

Blatant Comics is an independent American comic book publisher founded in 1997 by Chris Crosby... Blatant is known for publishing parody comic books such as Sloth Park, XXXena: Warrior Pornstar, and Dead Sonja: She-Zombie with a Sword...

So prior to Keenspot, Chris Crosby wasn't just making webcomics – he was publishing print comics. Awful, pulpy, dumb parody print comics.

In 1997 Blatant Comics published something called “The EboniX-Files” with taglines like “The truth be out there, g!” and “Yo, bust a cap in da future's ass!” and thought this joke was clever enough that they ALSO, published in the SAME BOOK, included “The Ungrammatical EboniX-Men”. Chris Crosby is credited as the writer.

I vaguely remember some shitty stereotypes of Black people in Superosity, and honestly it's possible they may have been as bad as this, I don't remember. But, yiiiiiiikes! Fuck this!

In 2006, it would appear that Chris was using his old print comics company, which hadn't published much of anything since Keenspot launched, to try to kickstart the career of his brother Bobby. He certainly wasn't using it to promote anyone else's Keenspot comic. Today, Blatant Comics' website is nothing but Bobby Crosby, and in fact looks startlingly similar to bobbycrosby.com.

Chris also launched a new Keenspot comic towards the end of 2006: “WICKEDPOWERED”, a paid advertisement for “a laser pointer with a range of 141 miles that can melt a garbage bag.” WICKEDPOWERED ran for a year and a half before the sponsor pulled the plug.

It seems clear that by 2006, we are looking at a Keenspot that is not driven by a group of peers trying to create a new medium and support each other – it is being driven, more and more, by the Crosbys' desire to cash in as hard as they possibly can.

Keenspot becomes The Crosby Show

In March 2008, Chris and Teri Crosby buy Darren Bluel and Nate Stone's stake in Keenspot, leaving the Crosbys as the sole owners. Chris was interviewed by comixtalk.com about it. I find this quote incredibly revealing:

Eight years later, we failed to reach most of our goals. I'm hoping to turn that around, so that five or ten years later I can look back and be a little prouder of what Keenspot is. We'll see.

To me, Keenspot was absolutely at its most successful, its most magical, right at the start; all those talented artists, breaking free from the gatekeepers of newspaper comics syndicates, using new tools to find new audiences, producing work that couldn't be published any other way. What are the goals for Keenspot that they failed to reach?

Here's Chris posting to the Bad Webcomics Wiki forums in June 2009:

I enjoy working on SORE THUMBS and WICKEDPOWERED, but they were written primarily for the money, not because they're the kind of thing I love to write. They both are purposefully created to be as dumb and pandering as possible. Heck, WICKEDPOWERED was a paid advertisement for a handheld laser manufacturer. And CROW SCARE is intended to be a SCI FI Channel original movie illustrated on cheap newsprint.

Yes, I haven't aimed very high thus far. Bobby's goal with everything but PUPKIN and +EV has been to write fantastic blockbuster movies in graphic novel form. Maybe I should try that…

Someone replies:

So you're basically admitting to being a hack that panders to the lowest common denominator for money?

And Chris responds, simply:

I like money.

Keenspot begins to implode

At this point in time, Keenspot has slowly gone from The Place For Quality Webcomics to an aging portal whose design has not significantly changed in 8 years, and whose creators are still on it largely out of inertia. AutoKeen has not been updated in any significant way – it hasn't been upgraded to do anything as basic and obvious as generating RSS feeds, for example. By 2009 Google Reader would have been in its heydey, and RSS feeds a perfect fit for daily comics. (AutoKeen would eventually be rewritten out of frustration by an administrator of Keenspace around 2010, but didn't necessarily add much in the way of new features – just made it easier to diagnose when it failed.) The Comicpress Wordpress theme has been out for years, making self-hosting comics with archives more accessible than ever. Comics are starting to take off on Tumblr and Twitter. As far as I can tell, Keenspot artists are still carefully naming their files, uploading them to FTP sites, and hand-tweaking bespoke HTML templates. Keenspot as a technology, in terms of the service it provides for creators, is completely stagnant and neglected.

July 2009 – Jodie Troutman, a fairly prominent cartoonist who had “graduated” from Keenspace to Keenspot, is fired from Keenspot for no particular reason that anyone wants to disclose. (CW: deadnaming in linked article) Nothing like this has happened before in Keenspot's history. Troutman's response:

Though I believe my membership was terminated unjustly and through no fault of my own, I suspect I’ll be much better off without Keenspot, whose management I never really saw eye-to-eye with. All my friends have had great success as indie webcomics, so I can only hope to follow in their footsteps.

I see this sort of sentiment a lot – that Keenspot isn't really helping creators, and that publishing independently is a much better value proposition.

December 2009 – Kel McDonald is fired from Keenspot and publicly airs some dirty laundry about how unprofessional and basically useless Keenspot leadership has been towards its artists not named Crosby, and about how nobody is getting paid on time.

Teri Crosby responds in the comments with a polite point-by-point rebuttal, signing off with “Smiles”.

Bobby Crosby responds in the comments accusing everyone of being liars, ranting about how tiny and insignificant Keenspot is, how anything it does for its creators is a gift and above and beyond what they deserve and that they should be grateful, and posits that Keenspot should just be shut down.

“To have a booth presence mismanaged year after year, as Keenspot's frequently is, is unacceptable this far down the road.”

Why??? Who cares??? What does it matter to you? Keenspot is a tiny company that shouldn't even have a booth at SDCC in the first place but does so anyway mostly just as a little bonus to its members, a little gift. Are you the same type of person who turns down a free gift because it's not nice enough for someone of your stature??? Who gives a fuck???

“We all know you guys have Keenspot on autopilot and are basically using it to fund Chris and Bobby's side projects.”

It's the OPPOSITE. Our projects keep Keenspot alive so even more time and money can be wasted on it because its owners for some reason love a bunch of people who mostly hate them because of your lies.

No one in my family lives in a school.

There's lots going on in the comments but I would like to quote this section of Scott Kurtz's reply to Bobby:

Blatant and Keenspot are these very disparate entities when it suits you and sister companies when it suits you. You own both companies. You took the Keenspot booth this year and gave half to Blatant.

The Keen dream is dead

Within days after Kel McDonald's firing, Keenspot posts new mandatory contracts that all of its artists will need to sign in six months, or else leave Keenspot. The terms of this contract are... unfavourable towards creators, compared to what came before. These contracts are not sent to creators directly but instead quietly dumped in a private forum post, leaving many in the dark. Webcomics blog Fleen published a detailed exposé.

As one creator puts it:

Every Keenspot member I’ve spoken to agrees that this is the Crosbys’ way of firing everyone without having to fire anyone, since trying to ditch Kel [McDonald] blew up in their faces.

The new contract is ridiculous, completely unreasonable, and they know that. It doesn’t just mandate a revenue split, but requires cartoonists to give up their domains, and the contracts are slated to last three to five years.

Chris Crosby doesn't really disagree:

As well as not inviting or accepting any new members, we may also politely decline existing members who decide to sign the new contract. We’ll be having long discussions with each interested creator (assuming there are any) in order to work out what’s mutually beneficial and what’s not. If Keenspot cannot bring something substantial to the table for the creator in question, we will stop working with them.

I had hoped Keenspot the webcomics collective and Keenspot the independent publishing concern could co-exist happily. But after two years [following a 2008 reorganizaton and the buyout of former partners] the resounding answer is no. Those two sides of Keenspot resent each other, and neither side is happy.

[G]oing forward our focus will be directed solely at properties we have a long-term investment in, which is primarily Crosby-produced comics and related projects. That’s what makes the most business sense for us as a company, and we make no apologies for it.

Aftermath

Have a look at Chris Crosby's print comic credits, paying attention to the stuff with the “Keenspot” banner on top. “Fartnite.” “Yang Gang.” “Barry Steakfries: From the Files of Jetpack Joyride.” Chris Crosby started his career selling trash and it appears that's how he's determined to end it.

There is a big list of comics on the Keenspot frontpage; only the titles in bold are “currently updating”. 7 comics are marked in bold. One of them is Marry Me, which completed its run in 2008, but for obvious reasons has ads in every news box and site banner. Another is Head Trip, which hasn't updated since July 2017. A third is No Pink Ponies, which last updated March 2018.

In 2022, Keenspot is 4 comics.

Marry Me

In February 2007, Bobby Crosby and artist Remi “Eisu” Mokhtar launch “Marry Me”, at the URL marrymemovie.com. The URL and the commentary for the first page makes it clear: this is a graphic novel for which Bobby will be adapting into a screenplay and trying to sell to Hollywood. The webcomic exists as an elaborate movie pitch.

You might be aware that “Marry Me” is now, in 2022, a big-budget Hollywood romcom, starring Jennifer Lopez and Owen Wilson. It took 15 years and the total implosion of Keenspot, a breathtaking, agonizing squandering of an enormous amount of actual talent in service of one unbearable asshole, but the motherfucker actually did it.

And yet. None of the marketing around this movie mentions the comic it was based on. Nobody going to see it knows Bobby Crosby's name, or Blatant Comics, or Keenspot. The Crosbys spent 16 years pushing Bobby's work, and in the end of all that is this disposable rom-com; every creative decision made by everyone involved for the sole reason that they thought it would sell.

I wonder: is Chris Crosby able to look back and be proud of that?

#webcomics #essays

I go down rabbit holes. One of the great pleasures of this dumb future we live in is that you can dig through the milk crates of our culture forever, following whatever interests you, and there is no bottom. My latest rabbit hole looked something like this:

• I have been spending a few minutes each day seeking out backgrounds for use on Google Meet calls at work that match my t-shirts.
• I was wearing a t-shirt featuring some early computer comic art from a Beagle Bros catalogue, drawn by Robert Cavey
• Started poking around at old Beagle Bros catalogues, settled on this background
• Started poking around Apple II magazines for more Bob Cavey comics
• Discovered the following advertisement:

Wait, hold on a minute. This kind of sounds exactly like the weird idealized programming environment I keep in my head. Obviously with serious restrictions and caveats, but, as a learning tool? They implemented this on an Apple II in 1984? How far exactly did they go?

I mean, look at this:

• “Everything a Spellcaster program does leaves marks on the screen. You watch all its inner workings in motion.” Programming is frustratingly opaque – it can be incredibly difficult, especially for a beginner, to answer the question, “why is the computer doing this?” But for the purposes of a teaching language, why not put all of the program's state on the screen?
• “Debugging a Spellcaster program is easy, because you can stop it, make it back up to the mistake (while you watch), change it, and let it run forward again.” I have been fighting to build this literal exact workflow for YEARS. Time travel debugging crossed with edit and continue??
• “Imagine an editor and interpreter so wed that every keystroke, as it is typed, is syntactically checked and executed, so you instantly see its effects. If you backspace, the program reconstructs its previous state — even in the middle of conditions and loops.” A special livecoding editor built for the tightest possible feedback loop, with no invalid states?

So of course I went looking for disk images.

There were no disk images. There were a couple of scanned magazines with the same ad. That was all I could turn up.

I started asking around on the internet and a couple of people tracked down a few more references; a couple more short reviews in some magazines. The jackpot, though, was this reimplementation of Spellcaster written in Processing. The readme gave me names, and the names gave me email addresses.

Long story short, I reached out to Scott Gilkeson, the original programmer of the C64 version, and it turned out that he had conveniently made disk images a few years ago. He also put me in touch with John Fairfield, the original designer and programmer of the Apple II version, who gave his blessing to share them. (The Apple II version is lost, as far as anyone knows.) There was also a PC version, written in C; nobody seems to know where that is, either. Within a day of reaching out, I had a copy.

You can use Spellcaster now, right from your browser.

A manual is also available. (Scott is seeing about creating a higher-quality scan, but what's there is much better than nothing.)

John Fairfield would go on to cofound the company that produced the Rosetta Stone language learning software. Spellcaster is, in a deep way, also language learning software. There are some truly fascinating design decisions made, not only in the Spellcaster programming environment, as advertised, but also as a language.

• It is the only programming language I am aware of that is designed to be spoken. The way the editor works, you generally press keys corresponding to syllables, rather than letters. There are occasionally some mnemonic properties to those syllables, but they're pretty arbitrary; you end up writing words like NUBOBORIBOBOLIAKA. Presumably, in a group learning setting like a classroom, this means programs can be unambiguously talked about, down to the last instruction – an often overlooked but useful property, as anyone who has tried to read code off a slide can attest.
• There is a base 4 numbering system that's used in a few different places; 4 of the 5 vowels are used as digits, and paired with a starting consonant to mean different things that are done with that number (MABO means “repeat BO once”, KA means “set the pen to colour #1”). The fifth vowel, U, is used to choose a random number between 0 and 3. Because there are four cardinal directions, and 16 colours, this means something like MURI (turn clockwise a random number of times) is a nice concise way to write “point my pen in a random direction”, and KUKU is a nice concise way to write “choose a random colour”. Again, designed to be spoken, but also designed to be concise.
• You can have spaces in identifiers, but not the letter Z. Z is short for the ZIM syllable, which ends free text entry. Honestly? I kind of really love that tradeoff. A beginner is much more likely to run into the question of “why can't I write two words here” long before they ask “why can't I write a Z”.
• You can also give a spell an empty name. I did this by accident but the manual says “Well, that's the name that has no letters in it, and it's a perfectly good name.” And indeed, it just works, to call it you just write TUZIM, a little like if there was a special function you could call in Lisp by writing ().
• Reading just a little further into the manual, it turns out the spell with the empty name has an actual, specific purpose! VUZIM means “see if the user pressed a key, and if they did, run a spell corresponding to that key”. So if the user hits the “A” key, it will run the A spell. So of course since you can't have a Z spell, if the user hits the “Z” key, the empty spell is called. Given that you do have to hit the “Z” key to create the spell, there is a certain strange elegance to it; I don't entirely know if I think it's a good design but it's sure not a choice I would have thought to make.

#preservation #retrocomputing

Step 1: Write a 6502 assembler

The Honeylisp assembler is written in Fennel, which is a Lisp dialect that compiles to Lua. My goal in writing it was that the input to the assembler would be simple lists, making Fennel into a super-powerful macro assembler basically for free. 6502 assembly is treated as simple data, a kind of embedded DSL that can trivially be generated by simple Fennel functions.

Step 2: Integrate into a text editor

The Honeylisp environment runs on top of lite, a very small and extensible programmer's text editor written in Lua. It's trivial to add new commands that trigger whatever kind of interactive build process I want. It's also straightforward to extend to create custom UI; I put together the bones of a simple tile editor in an evening. I ended up porting lite to the love2d runtime to allow me more flexibility in how I could build these custom editors.

It is also a fairly straightforward matter to add hot code reloading, though it requires a bit of forethought to take full advantage of it. When I put together the tile editor, I could make changes to it, hit Alt-R, and those changes were immediately live. It's cool to be able to use your editor to live-edit your editor.

Step 3: Integrate into an emulator

Back in July, I came across this video of Dagen Brock talking about an Apple IIgs emulator project he was working on that had an interesting property. Instead of integrating a debugger into the emulator, he decided he would implement a socket-based debugging interface that anyone could write a front-end for. Any external program could use this to gain full control of the emulated computer – read and write memory, set registers, set breakpoints, anything.

It turns out this idea kind of languished, and he never quite got to polishing up and releasing his debugger. But this emulator, GSPlus, exists, and it gave me the core idea at the heart of Honeylisp – if I can arbitrarily read and write memory, and I have the ability to augment my assembler, then I can do anything. I can take snapshots and jump back and forth between them, recovering from hard crashes. I can push code updates while the game is running; I just need to be able to specify an appropriate “merge point”. I can map variables from one version of the program to another, so even if data moves around in memory, I can compensate. I can put breakpoints on memory I know should be read-only, to catch wild pointer accesses. I can implement a REPL entirely on the PC side; assemble tiny snippets of code and trigger them. I can integrate with my tools, so making changes to a graphical tile in my editor instantly updates the screen of my emulated Apple II. The possibilities are truly vast.

Step 4: Integrate with hardware

One of the pretty unique features of the Apple II is that it comes with a built-in machine language monitor, complete with a mini assembler and disassembler. It turns out that this can trivially be controlled over a serial port to do things like... write arbitrary memory. So I can plug my laptop directly into a stock Apple II, type IN #1, and then run a single command to bootstrap my environment. In fact writing arbitrary memory in response to serial port commands is not very complicated, and many of the magical abilities I imagine for my environment can be done directly on real hardware.

Step 5: Write a game

There is no point in investing all of this effort into software development tools if you don't plan to develop any software. I intend to port my MS-DOS game Neut Tower to the Apple II using this system. Neut Tower was written using a hand-rolled Forth system; I've built a simple Forth-like stack VM to allow for non-performance-critical code to be written compactly. Because my assembler is so easily extensible, I can also easily use simple Fennel code to generate VM bytecode, which means I can do without a huge amount of the overhead of a full Forth as all of the compiler-y bits live on my laptop. This VM can also be the basis of my interactive REPL, which will be much nicer to write than little assembly snippets.

Step 6: Tell people about it

It's fairly early stages for all of these steps – I have a good start in all of them, but there is still lots of work to be done. I'm really excited about this project, though, and I want to talk about it! Hopefully this blog will be a useful place to do that. I'm looking forward to continuing to share my progress.

#lisp #honeylisp #retrocomputing #apple2 #neuttower

I wrote a game on an MS-DOS 286 PC, using only tools I built myself or tools that were available during the era where they were still selling 286 PCs. It's called Neut Tower and you can play it on your MS-DOS PC, in DOSBox, or in your browser. As part of this project, I implemented a Forth system, and built most of my game and its tools using it.

My motivation at the start of the project was this: I was enjoying using my 286. I liked the single-tasking workflow; there were no distractions. I was downloading games and apps and it was fun! So I figured I'd take the next step and write a little game or something.

When I was a teenager, I had a 286, and I tried to learn low-level programming on it because my options were “low-level programming” and “BASIC”, and I had hit my limit with BASIC. Assembly might as well have been Martian to me, but I got a book about C, and I got a book about game programming, and I sort of got some stuff working. But mostly the stuff I tried to do myself from scratch, or port from other sources, didn't work, and I didn't know why. Eventually I also got access to a 486, and then a Pentium, and the internet, and djgpp and Allegro, and suddenly I had an embarrassment of nice graphics and sound libraries and tooling, segment:offset addressing didn't matter, and I never had to worry about trying to understand how Mode X worked ever again.

Twentyish years later, I wanted to learn all the stuff that never quite clicked for me. I wanted to dig into how everything worked, to make sense of the tutorials that once baffled me. I wanted to really understand it all. So I started writing little prototypes, and pretty soon, yeah, I had a cool EGA graphics engine, with two way scrolling of a tilemap and 16x16 sprites drawn on top, running at a decent speed on actual hardware. Everything fell into place one tiny experiment at a time.

With the hardware programming side of things, I learned that my teenage understanding hadn't really been all that far off the mark – my problems weren't so much that I didn't understand the tutorials and resources that were available to me, it was more that I was simply bad at debugging my beginner code, and didn't have the tools or the patience to fix it. With 20 years of professional programming experience under my belt, and a wealth of resources on the internet that explained how things worked in depth, this was no longer an issue.

Then I started to write a game loop in C, and didn't really like it. I knew in the back of my head that, for what I wanted to do, I really wanted some kind of scripting language. And I remembered Forth existed.

In my 20s, obsessed with both the world of programming languages and the world of embedded systems, it was inevitable that I would learn about Forth – it's a particularly uncommon blend of small and powerful, that could run directly on hardware, that people who loved it really loved. I'd tried seriously to learn it but couldn't really wrap my head around it – the weird postfix syntax, the confusing levels of meta. Why could I not use IF statements at the REPL? How was I supposed to remember all these finicky rules? I filed it away as “interesting, but not for me.”

This project was the perfect opportunity to revisit that evaluation. Forth fit the bill exactly – it was a tool that could be built quickly, using minimal resources, and made to do what I wanted, AND I already had a hazy half-remembered foundation from decades ago. I dove headfirst into it.

Relearning Forth was an altogether different experience. It turned out that once I built one myself, I understood it completely. The design of Forth is to write as little code as you possibly can, to make the computer do only as much work as it needs to. When I had to write it all myself, I had to decide – is it worth building this language feature, or can I do without it? Usually I could do without it. Usually there was a tinier way to do it. The code that I had to write wasn't really all that much uglier or worse for it, once I got used to the constraints. And I had proven designs I could pilfer; there are lots of existing open-source Forth implementations to get inspiration from. There are guides for building Forth systems. Doing Forth is not learning an existing language set in stone, it is building a language to solve your problem, and sharing ideas about useful building blocks. Chuck Moore, the inventor of Forth, hated its standardization; thought the goal of portability was absurd, thought everyone should change it as they needed, to fit their problem. He is still trying out new ideas, rebuilding, simplifying, making a system uniquely his own.

So why do I think all this is important enough to write about?

When I was a kid, I had this persistent idea in my head, that computing was a skill I could work at, get better at, and that doing so would allow me to accomplish things that were impossible for me without it. “Once I got good enough”, I could make a computer game, by myself. I could draw the graphics, I could write the code, I could make the music, I could design it all. I could make it and I could put it out into the world and it would be mine, start to finish. Every time I learned something new about computers, got some new piece of software, I gained abilities. I could do things I couldn't do before. My vision of computer literacy is that everyone has this experience, that everyone can learn the skills they want, is provided with the tools they need, to make their imagination real. I have never really let go of this idea.

I'm still trying to find ways to make it true, still trying to explore the different ways that computing can be empowering. Retrocomputing is one avenue for that – people in the past had a lot of good ideas that didn't catch on. And while emulators are wonderful, running them inside a modern computing system makes it harder to experience what using an old computing system really felt like.

When I show people my setup, they are often curious about the qualitative difference between old tools and modern tools; it must be so much harder, right? And... for me, it's really not! I write bugs at about the same rate; I fix them at about the same rate. There are many things I can't do because of resource constraints, but that keeps the scope manageable and makes for an interesting challenge to find cool stuff I can do. The biggest thing I miss is having a second editor that I can use to look at & edit code while my game is running — I have often resorted to taking a photo of some code with my phone so I can read it while I have the game up.

And I gain really valuable things from the constraints. The biggest thing is that there's no alt-tab away from the work – it's so much easier to focus without a web browser instantly at my fingertips. (I'm procrastinating at work writing this right now!) The resource constraints mean I have to focus ruthlessly on solving the problems I have, not the problems I imagine I'll have – there's no perfect, elegant, general solution if I think hard enough, there's only adding things and cleaning up what I've got, one small piece at a time. And I can take workflow seriously as one of those problems! When I'm fed up with the tools that are available for DOS on a 286 (and this happened multiple times!), I make my own that work the way I want, and I'm able to integrate them seamlessly into my engine. I'm able to intentionally craft my environment to be comfortable. I'm no artist, but multiple people have complimented my art – partly, the secret is that 16x16 sprites and tiles can only look so good with a fixed ugly 16-colour palette, so I'm able to focus on broad colour and style choices. But really, if you put me into my ugly, limited pixel editor that's two pages of code but instantly shows me what my sprite looks like in my game, I will mess around until I'm happy. Put me in front of Photoshop with 16 million colours and I will go crazy from decision fatigue; I'll avoid making more art, and I'll get myself stuck.

So for me, the tradeoffs are incredibly worth it. I've spent decades trying to make games as a hobby; I've put out reams of junk – failed prototypes, bad joke games, quick jam games, failed engines, half-finished tools. I've tried every way of making games that I can think of; coding engines from scratch, using Unity, Godot, Love2D, Klik & Play, Game Maker, Twine, Construct, Adventure Game Studio, pygame, Allegro. Some approaches I've had more success with than others, but I've not ever been as happy with anything I've made as I am with Neut Tower. Not as a retrocomputing exercise — as a game.

Neut Tower is done, for now, and I am taking a break from it. (Perhaps someday I will return to it to create the next two episodes.) I'm quickly finding myself using all of these lessons and starting to build some tools for myself in Linux. I don't quite know what they'll turn into yet, but I'm looking forward to finding out, one small piece at a time.

#neuttower #retrocomputing #essays #forth

Forth and Lisp are often spoken of as though they are similar in some deep way. In Lisp circles, you often hear “code is data.” This is generally held to mean “Lisp has macros”, more or less – a Lisp program's source code is a syntax tree made of Lisp lists, that your Lisp program can introspect into and transform into new syntax trees and execute. Your program is literally a data structure.

My Forth code has very few things I would refer to as “data structures”. There is no significant language for defining them – I write one-off words that do pointer arithmetic. I only have a handful, so I haven't felt the need to generalize. It does zero transformation of them – they have been carefully chosen to be directly useful for everything the program needs them for, in-place, as-is.

Instead, the common pattern is that everything is code, which, thanks to Forth's flexible non-syntax, can be made to look a lot like data. Often data is compiled directly into the code that uses it – instead of naming a constant that's passed into a function to do a particular thing, you name a function that takes no arguments that just does the thing. (There are lots of flexible ways to make this sort of thing easy and inexpensive in Forth.) Forth is hyper-imperative to a degree that, as modern programmers, we've largely forgotten is even possible. Even, say, the number 4 is arguably a word executed for its side effects (push the value 4 onto the current stack). Of course, this is how CPUs work, too – you don't have a concept of “4” on its own in assembly, you have the concept of moving “4” into a register, or into memory. The only thing you can tell a CPU is to do things. Forth is the same.

One consequence is that a Forth word that represents a constant is invoked in exactly the same way as a word that makes decisions. What this means is that it is virtually impossible to write yourself into a corner by “hard-coding” something. You can start with the most direct implementation, and expand it into something more flexible as you need to. I often find myself turning a word that was very static into something dynamic, and not having to change any of the code that depends on it. And my Forth has developed lots of facilities for sophisticated decision-making and dispatch. It turns out that most sophisticated decision-making is largely just indirection, and is easy to accomplish even in extremely resource-constrained environments. Many things I used to think of as modern, expensive conveniences – anonymous functions! polymorphism! green threads! – are actually extremely cheap and simple to build, they just... don't exist in C.

In “Programming a Problem-Oriented Language”, Chuck Moore defines “input” as “...information that controls a program.” Forth and Lisp share the idea that, most of the time, it's more powerful and flexible to use the language's parser to read a program's input. Before JSON, there was the s-expression, the universal data structure, and in Lisp, you usually are either using macros to turn that data into code directly, or writing an interpreter for that data. You can often think of a Lisp program as a collection of small, domain-specific virtual machines.

However, Forth doesn't really have a parser; it has a tokenizer, a symbol table, an interpreter, and a virtual machine. Parsing Forth and executing Forth are synonymous; hell, compiling Forth and executing Forth are synonymous. Forth says you don't need a domain-specific virtual machine; you already have a perfectly good machine right here! Why not just solve your problem directly, right now?

You may need sophisticated abstractions to succinctly describe the logic of how your problem is solved, and writing good Forth code is all about investing in those. But Forth makes an argument that most of the data that your program deals with is actually about controlling what your program should do, and making decisions about what your program should do is the job of code.

There are drawbacks to this approach, of course; plenty of things that are inconvenient to express as text, plenty of times I wished I had a “live” data structure I could update on the fly and persist while my program is running, rather than having to exit my program and update my code. But if you can work within the constraints, there is enormous flexibility in it. I'm writing a puzzle game, and while I have a terse vocabulary for defining levels, it's also trivial for me to add little custom setpieces to a given level, to throw in dialogue in response to weird events, to add weird constraints that only apply in that space, because at every step, I have the full power of the language at my disposal. If I'd taken a data-driven approach, I would have needed to plan everything in advance, to design my little problem-oriented VM and and hope I thought of everything I needed. But with a code-first approach, I can be much more exploratory – try to build things, and if they work well, factor them out to be used more generally. Architecture arises naturally from need, as I build.

#forth #essays

• The amount of code required to implement it
• The size of the code that is generated
• The amount of memory used
• The number of features it considers necessary for useful work

It is a language that makes complexity painful, but which reveals that a surprising amount can be accomplished without introducing any. Forth is the opposite of “bloat”. If you've ever been like “Oh my God this Electron-based chat app is taking up 10% of my CPU at idle, what the HELL is it DOING, modern computing has gone MAD”, Forth is there to tell you that computing went mad decades ago, and that programs could be doing SO MUCH MORE with SO MUCH LESS.

WHAT DO YOU MEAN, “FORTH”

There is an expression about Forth: “If you've seen one Forth, you've seen one Forth.” Forth isn't a strictly-defined language, though there is a standardized dialect; it's more a set of ideas that tend to work well together.

In the past month, I wrote a tiny Forth system on a 286 running MS-DOS using Turbo C++ 1.01. It is my first time using Forth in anger, though I read a lot about it 15 years ago. When I refer to my Forth, I am referring to a system literally thrown together in two weeks, written by someone who does not really know Forth that well. It is slow and wildly nonstandard and it doesn't do very much, but I have enjoyed the process of writing it very much. If you are a grizzled old Forth grognard, please let me know if I have misrepresented anything.

WHAT DOES FORTH NOT DO

Here is an incomplete list of things you may take for granted as a programmer that Forth, in its purest form, generally considers unnecessary waste:

• Garbage collection
• Dynamic memory allocation
• Garbage
• Memory safety
• Static types
• Dynamic types
• Objects
• Polymorphic methods
• Closures
• Lexical scoping
• The concept of global variables being in any way “bad”
• Local variables
• The ability to write “IF” statements at the REPL

Most or all of these can be added to the language – the Forth standard, ANS Forth, specifies words for dynamic memory allocation and local variables. There are lots of object systems that people have built on top of Forth. Forth is a flexible medium, if you're willing to put in the work.

But the inventor of Forth, Chuck Moore, literally said, in 1999: “I remain adamant that local variables are not only useless, they are harmful.” In the Forth philosophy, needing to use local variables is a sign that you have not simplified the problem enough; that you should restructure things so that the meaning is clear without them.

WHAT DOES FORTH LOOK LIKE

A core part of Forth is that all functions, or “words” in Forth terminology, operate on “the stack”. Words take arguments from the stack, and return their results on the stack. There are a handful of primitive built-in words that do no useful work besides manipulating the stack.

What this means is that writing an expression tree as Forth code ends up turning into postfix notation. (1 + 2) * (3 - 4) becomes 1 2 + 3 4 - *. Writing a number in Forth means “push that number onto the stack”.

Forth syntax is, with a few exceptions, radically, stupefyingly simple: Everything that's not whitespace is a word. Once the interpreter has found a word, it looks it up in the global dictionary, and if it has an entry, it executes it. If it doesn't have an entry, the interpreter tries to parse it as a number; if that works, it pushes that number on the stack. If it's not a number either, it prints out an error and pushes on.

Oops, I meant to describe the syntax but instead I wrote down the entire interpreter semantics, because it fits in three sentences.

The exception to the “whatever is not whitespace is a word” rule is that the interpreter is not the only piece of Forth code that can consume input. For example, ( is a word that reads input and discards it until it finds a ) character. That's how comments work – the interpreter sees the ( with a space after it, runs the word, and then the next character it looks at is after the comment has ended. You can trivially define ( in one line of Forth.

WHY THE HELL WOULD I USE THAT

There are practical reasons:

• You need something tiny and reasonably powerful, and you don't care about memory safety
• I'm not sure I can think of any others

And there are intangible reasons:

• Implementing a programming language that fits into a few kilobytes of RAM, that you understand every line of, that you can build one piece at a time and extend infinitely, makes you feel like a god-damn all-powerful wizard

Part of the mystique of Forth is that you can get very metacircular with it – control flow words like IF and FOR are implemented in Forth, not part of the compiler/interpreter. So are comments, and string literals. The compiler/interpreter itself is usually, in some way, written in Forth. It turns out that you can discard virtually every creature comfort of modern programming and still end up with a useful language that is extensible in whatever direction you choose to put effort into.

Forth enters that rarefied pantheon of languages where the interpreter is, like, half a page of code, written in itself. In many ways it's kind of like a weird backwards lisp with no parentheses. And it can be made to run on the tiniest hardware!

The mental model for bootstrapping a Forth system goes something like:

• Write primitive words in assembly – this includes the complete Forth “VM”, as distinct from the Forth language interpreter/compiler. The set of built-in words can be very, very small – in the document “eForth Overview” by C. H. Ting, which I have seen recommended as an excellent deep-dive into the details of how to build a Forth environment, Ting states that his system is built with 31 “primitive” words written in assembly.
• Hand-assemble “VM bytecode” for the interpreter/compiler and required dependencies – because of the extreme simplicity of the VM, you can generally program your macro assembler to do this job, and so this can meaningfully resemble the act of simply writing Forth code directly
• Write all new words using the interpreter/compiler you just got running

I say “interpreter/compiler” and not “interpreter and compiler” because they are literally mixed together; there is a global flag that determines whether the interpreter is in “compile mode” or not. It is done this way because it turns out that if you add the ability to mark a word as “always interpret, even in compile mode”, you have added the ability to extend the compiler in arbitrary ways.

WHAT SUCKS ABOUT WRITING FORTH

Any word that takes more than two or three parameters is a nightmare to read or write

Right now in my codebase I have a word that uses two global variables because I cannot deal with juggling all of the values on the stack. This word is absolutely not re-entrant and at some point I'm going to need to rewrite it so that it is, and I am not looking forward to it. If I had local variables, it would be substantially less of a problem. But there's also part of me that thinks there must be some way to rewrite it to be simpler that I haven't figured out yet.

There's another word in my codebase that takes 4 or 5 parameters that I managed to write by breaking it up into, like, 8 smaller words, over the course of writing / rewriting for like an hour or two. I felt pretty proud when I finally got it working, but honestly I think it would have been pretty trivial to write in C with local variables. I miss them.

Shit crashes

Remember the part about no memory safety? Yeah, there's all kinds of ways a wayward Forth system can go wrong. I forgot a DROP once in a frequently-used word and my computer hard-locked when the stack overflowed. (To be fair: my computer was a 286 running MS-DOS, so I was already in a situation where programming it meant rebooting it when I inevitably fucked something up.)

Nonexistent error messages

The only error message my Forth system has is, if it doesn't recognize the word “foo”, it prints “foo?” If, for example, I write an IF statement, but forget to end it with THEN, I don't get a compile error, I get — you guessed it — a runtime hard crash.

WHAT RULES ABOUT WRITING FORTH

It's compact as hell

The majority of words I write are literally one line of code. They do a small job and get out.

It's direct as hell

Building abstractions in Forth is... different than building abstractions in other languages. It's still a really core, important thing, but as building complex / expensive code is so much work, stacking expensive abstractions on top of each other is not really tenable. So you're left with very basic building blocks to do your job as straightforwardly as possible.

You are absolutely empowered to fix any problems with your particular workflow and environment

People turn Forth systems into tiny OSes, complete with text editors, and I absolutely did not understand this impulse until I wrote my own. The Forth interpreter is an interactive commandline, and you can absolutely make it your own. Early on I wrote a decompiler, because it was easy. It's like half a screen of code. There are some cases it falls down on, but I wrote it in like a half hour and it works well enough for what I need.

Everything is tiny and easy to change or extend

Remember when I said I wrote a decompiler because it was easy? Other things I changed in an evening or two:

• Added co-operative multitasking (green threads)
• Custom I/O overrides, so my interactive REPL sessions could be saved to disk
• Rewrote the core interpreter loop in Forth
• Rewrote the VM loop to not use the C stack
• Instrumenting the VM with debug output to catch a crash bug

One of the things on my todo list is a basic interactive step-through debugger, which I suspect I'll be able to get basically up and running within, like, an hour or two? When things stay tiny and simple, you don't worry too much about changing them to make them better, you just do it.

If you have ever wanted an assembly code REPL, this is about as close as you're going to get

Forth is a dynamic language in which the only type is “a 16-bit number” and you can do whatever the fuck you want with that number. This is dangerous as hell, of course, but if you are writing code that has no chance of having to handle arbitrary adversarial input from the internet (like my aforementioned MS-DOS 286), it is surprising how refreshing and fun this is.

THIS SOUNDS INTERESTING, WHAT IS THE BEST WAY TO LEARN MORE

I honestly do not know if there is a better way to understand Forth than just trying to build your own, and referring to other Forth implementations and documents when you get stuck. It's been my experience that they just don't make sense until you're neck deep into it. And it's tiny enough that you feel good about throwing away pieces that aren't working once you understand what does work.

I've found the process of writing my own Forth and working within its constraints to be far more rewarding than any time I have tried working with existing Forths, even if on occasion I have wished for more complex functionality than I'm willing to build on my own.

WHAT HAVE I LEARNED FROM ALL THIS

I'm very interested in alternate visions of what computing can look like, and who it can be for. Forth has some very interesting ideas embedded in it:

• A system does not have to be complex to be flexible, extensible, and customizable
• A single person should be able to understand a computing system in its entirety, so that they can change it to fit their needs

I find myself wondering a lot what a more accessible Forth might look like; are there more flexible, composable, simple abstractions like the Forth “word” out there? Our current GUI paradigms can't be irreducible in complexity; is there a radically simpler alternative that empowers individuals? What else could an individual-scale programming language look like, that is not only designed to enable simplicity, but to outright disallow complexity?

Forth is a radical language because it does not “scale up”; you cannot build a huge system in it that no one person understands and expect it to work. Most systems I have used that don't scale up – Klik & Play, Hypercard, Scratch, that sort of thing – are designed for accessibility. Forth is not; it's designed for leverage. That's an interesting design space I wasn't even really aware of.

The lesson that implementing abstractions as directly as possible enables you to more easily change them is a useful one. And the experience of succeeding in building a programming environment from scratch on an underpowered computer in a couple of weeks is something I will bring with me to other stalled projects – you can sit down for a couple of hours, radically simplify, make progress, and learn.

#forth #retrocomputing #essays