It didn't seem like too much of a hardship. In a previous Forth I'd built, I'd implemented do / loop using the return stack, but it was... ugly. The code to implement it was ugly, the code it generated was ugly (and large!), and I didn't find a lot of places where it was actually much nicer to use than explicit begin-based loops. I was able to implement an 8086 assembler and a Minesweeper game without bothering to build do / loop. I didn't really miss it, but I had a design percolating in the back of my mind that I wanted to try.

At some point I came across some writing that suggested that Forth had a “loop control stack”. Wouldn't it be nice if I could implement some kind of loop control stack that worked for all kinds of iteration?

The thing I built has blown me away with how flexible, composable, and useful it's turned out to be. It's way more powerful than I was expecting. And the code that leverages it is inevitably much simpler and easier to read.

The Stacks

I added two loop control stacks – what I call the i-stack, and the next-stack. The i-stack contains the current value(s) being iterated over, and is read from with the i and j words like normal. The next-stack is where the magic happens.

When iterating, the top value of the next-stack is a pointer to a small structure called an iterator. It's a very simple structure, only two cells. The first cell contains the execution token of a word that will either update the current values on the i-stack and return true, or remove its state from both stacks and return false. The second cell points to a cancellation function, that cleans up whatever state the iterator has kept on the two stacks without iterating further, and returns nothing.

Iterators

I built some simple helpers for creating iterators. It took a few tries to nail down this design, but I'm happy with it now. defiter creates a “blank” iterator, which, when called, pushes itself to the next-stack. :iter does the same thing but allows you to write some code that accepts parameters and prepares the loop stacks first. :next defines a new anonymous “next-word” and assigns it to the most-recently defined iterator. :cancel does the same thing, but for cancellation.

On its own, this is already quite nice. I've got a page or so of basically-trivial iterators. Here's one:

:iter times ( n -- ) >i ;
:next <i dup 1- >i finish? ;
:cancel idrop nextdrop ;

times keeps its state in the i-stack – it initializes itself by pushing the number of times to repeat onto it. When fetching the next value, it pops the current value off the i-stack, decrements it, and pushes it back, leaving the old value on the data stack. finish? is a simple helper word that peeks at the top of the stack and runs the current cancellation function if it's false, or in this case, if we've already hit 0. Since cleaning up after an iterator is often the same job whether you're exiting early or not, this word is very handy. Explicitly defining cancellation for this iterator isn't actually necessary in my current implementation, because idrop nextdrop is common enough that I use it as the default.

each / next

I can use these iteration words (within a compiled definition) like this:

5 times each i . next
( outputs: 4 3 2 1 0 )

All the common loop types are easy to build in this system, as well as some uncommon ones:

5 10 for each i . next ( outputs: 5 6 7 8 9 )
0 10 2 for+ each i . next ( outputs: 0 2 4 6 8 )
( pchars yields pointers to each byte in a zero-terminated string )
s" hello" pchars each i b@ emit next ( outputs: hello )

Generic cancellation, of course, allows us to trivially implement break; just cancel the iteration at the top of the stack, and then jump to the each loop exit point, after next. continue is even simpler, just jump back to the top of the loop.

5 times each i 3 < if break then i . next ( outputs: 4 3 )
5 times each i 2 % if continue then i . next ( outputs 4 2 0 )

Under the hood, each just calls the “next-word” of the iterator and jumps to the end of the loop if it returns 0 – conceptually identical to begin iterate while, with next meaning the same thing as repeat. This allows for iterators that return no values.

0 times each i . next ( outputs: )

Generators

That's nice, but it's not exactly setting the world on fire; it's a fair amount of work just to end up with a few different ways of writing “for” loops in practice, that Forth systems have had forever anyway. Is it really worth the cost of this abstraction?

Turns out, absolutely, yes, it is, because you can also build generators on it, and that blows things wide open.

First, a simple example:

: 5-2-8 (( 5 yield 2 yield 8 yield )) ;
5-2-8 each i . next ( outputs: 5 2 8 )

(( defines the start of the generator, and )) defines the end (and pushes it onto the next-stack). Any valid Forth code goes in between. yield takes the top of the stack, pushes it onto the i-stack, and then suspends the generator until the next iteration. How does this work? Essentially, yield takes the top of the return stack and pushes it onto the next-stack, then pushes an iterator that pops it off the next-stack and pushes it back onto the return stack. The details get a little messier in order to support some more advanced use cases, but that's the simple idea at the core of it.

OK, neat trick, we've built ourselves a nice little coroutine-like system. But wait! It gets better! When yield resumes, it immediately removes all of its state from the iteration stacks. This means that generators can safely interact with any iterator that might be “underneath” it. They can iterate over things and yield in the middle! They can yield different things based on those values! We've accidentally built an extremely powerful, totally generic map/filter capability!

: doubled (( each i i + map next )) ;
5 times doubled each i . next ( outputs: 8 6 4 2 0 )
: odd (( each i 2 % filter next )) ;
5 times odd each i . next ( outputs: 3 1 )

map and filter are more yield-like words – it turns out that there's a number of these that you might want to implement, with different logic for suspending, resuming, and cancelling. map saves the top of the i-stack onto the next-stack and replaces it with the input, restoring the original value after resuming (necessary since the iterator underneath might be using that value as its state). filter conditionally suspends based on the top of the data stack but otherwise doesn't touch the i-stack, leaving whatever iterator is running underneath to provide the value. Both of these words push iterators with special cancel logic that knows that there is another iterator underneath, and can cancel again recursively once they've cleaned themselves up.

Generator state

This design can almost be made to work for generators that have extra state, but it's awkward and incomplete. You must ensure the data stack is clean whenever you yield, so you're forced to manually shuffle data to and from the next stack. Consider a filter that only returns values that are divisible by a certain number:

: divisible-by ( n -- ) >next 
  (( <next each i over % 0 = swap >next filter <next next drop )) ;
5 divisible-by 21 times each i . next ( ouputs: 20 15 10 5 0 )

This works, but there's so much stack noise! And it breaks down if you need to cancel, because filter has no idea that there's extra stuff on the next-stack that it needs to clear. Ideally there would be some automatic way of keeping the state of the generator on the data stack while it's running, and push it safely away when we suspend. Could there be some way to write divisible-by like this?

: divisible-by ( n -- ) >arg (( each i over % 0 = filter next drop )) ;

In fact, this code works in my implementation. The scheme to make this happen is a little bit subtle, but it can be done efficiently with a minimum of bookkeeping noise in most cases. I define a variable, gen-arg-count, that starts at zero. >arg is an immediate word that compiles a call to >next and increments that variable. Then, any time I compile a yielding word, I append the value of gen-arg-count to the instruction stream – much like lit. When suspending, the yielding word reads that value out of the instruction stream and transfers that many values from the data stack to the next-stack. Then it moves the pointer to the instruction stream from the return stack to the next-stack, and finally pushes the yielding iterator. That iterator then pulls the instruction pointer back off the next-stack to determine how many values to move from the next-stack back onto the data stack, as well as where to resume the instruction stream. Cancellation similarly can read the arg-count byte to know how many extra values to drop from the next-stack.

Generators need to ensure the data stack is empty before exiting at )). At one point I considered having )) compile the appropriate number of drop calls automatically, but in the end I decided that it's reasonable and idiomatic to expect a generator to exit with a clean stack, like any other Forth word would.

With this extension, it's trivial to write all kinds of new iterators – we could even do away with the base iterator system entirely and just express everything as generators. There are lots nice one-line definitions of times:

( 1 ) : times ( n -- ) >arg (( begin dup while 1- dup yield repeat drop )) ;
( 2 ) : times ( n -- ) >next (( <next begin dup while 1- yield> repeat drop )) ;
( 3 ) : times ( n -- ) >arg (( -arg begin dup while 1- yield> repeat drop )) ;
( 4 ) ( suspend ) ' noop ( resume ) ' noop ( cancel ) ' idrop :yield iyield
: times ( n -- ) >i (( begin i while <i 1- >i iyield repeat idrop )) ;

Definition 1 doesn't use anything I haven't already explained. The state of the iterator is managed on the data stack, and automatically shuffled back and forth from the next-stack by yield.

Definition 2 adds a new word. yield> is a yielder that moves the yielded value from the i-stack back onto the data stack when it resumes, instead of dropping it. The state of the iterator starts on the next-stack but is moved to the i-stack once the iteration loop actually starts.

Definition 3 is virtually the same as 2, but demonstrates the ability to handle changes in the amount of state. -arg is an immediate word that generates no code, but decrements gen-arg-count so that you can express that you've consumed the argument and the next yield should preserve one less value on the data stack. (+arg is also defined, performing an increment, in case you generate more values on the stack than you started with.)

Definition 4 is built to keep all state on the i-stack from the beginning. Here we use :yield to define a new yielding word. I realized I hadn't built a yielder that left the i-stack alone when resuming, but would drop the value when cancelling, so I added one.

All of these options will correctly be cancelled if the code iterating over it calls break, with no special effort!

Final thoughts

With this scheme, generators always take up at least two spaces on the next-stack – one for the yielder's iterator, and one for the resume point. But if all iterators were defined as generators, and all yielding words had to be defined with :yield to ensure a uniform structure, we could just push the resume point. iterate and cancel could easily find the appropriate function pointer by looking next to the resume point for the address of the yielder and digging inside. I think this could be built in such a way that it would be basically as efficient as the existing scheme, at the cost of making the whole thing more complex to explain. It might be worth pursuing, because generators are so pleasant to read and write, and raw iterators are... less so. I basically never want to write a raw iterator besides the very basic ones that are built-in.

All the source for my Forth system is available online; the iteration system is defined in iter.jrt. There are some interesting examples of generators in embed.jrt, dialer.jrt and rick.jrt – some highlights:

• rle-decode – takes a pointer to some run-length encoded packed data, yields a stream of values. Uses the times iterator internally to count off the repeated values.
• menu-options – Provides a dynamic list of items to display in a menu. Yields 2 values at a time – the text to display, and the function to execute when the user selects it.
• xmit-iter – Writes text to the screen with a small delay between each character, to simulate a slow serial connection. An extremely simple loop that can be driven by complex generation logic – including streaming RLE-encoded data with embedded colour information.

#forth #code #essays

Because... functions should be re-entrant by default! They should clean up after themselves! Global variables are bad and must be avoided at all costs! Functions should be “pure” and take all of their inputs as parameters, avoiding hidden dependencies!

All of these ideas of what “good code” looks like are wrong in Forth.

It is actually extremely common for Forth words to rely on implicit context, which is globally accessible through other Forth words. This is often how you build DSLs!

Perhaps you are familiar with the JavaScript canvas API. It's based on PostScript, as are most vector drawing APIs, and PostScript, as you may know, is a Forth-like postfix language for printed graphics. The canvas API has a bunch of implicit state. When you draw a rectangle, for example, you pass in just the position and size. If you want to specify properties like the fill colour, stroke colour, stroke width, line cap style, and on and on and on, you call setter methods before calling the draw function. If you want to preserve the previous canvas state and return to it when you're done, you can explicitly push it onto a stack.

This is one secret sauce to writing small Forth words – you build little vocabularies that all work with some kernel of shared state.

Let's implement Bresenham's line algorithm

I had the idea to implement an algorithm where juggling all of the state on the stack would be a nightmare, to show an example of what this looks like in practice. I've always found Bresenham's line-drawing algorithm kind of awkward – most implementations in C switch between several nearly-identical code blocks depending on how steep the line is. But the core idea is actually very simple, and the awkward near-duplication of the standard C implementation does not have to be reproduced in Forth.

First we will define a simple textual canvas vocabulary:

80 CONSTANT SCREEN-W 
24 CONSTANT SCREEN-H
CREATE SCREEN SCREEN-W SCREEN-H * ALLOT
CREATE SCREEN-BRUSH KEY + C,

: SET-BRUSH ( -- ) KEY SCREEN-BRUSH C! ;
: FILL-SCREEN ( -- ) SCREEN-W SCREEN-H * SCREEN + SCREEN DO I SCREEN-BRUSH C@ SWAP C! LOOP ;
: SCREEN-XY ( x y -- ptr ) SCREEN-W * + SCREEN + ;
: PLOT-XY ( x y -- ) SCREEN-XY SCREEN-BRUSH C@ SWAP C! ;
: PRINT-ROW ( y -- ) 0 SWAP SCREEN-XY SCREEN-W TYPE ;
: PRINT-SCREEN SCREEN-H 0 DO I PRINT-ROW CR LOOP ;

This is ANS Forth – my personal Forths have all been lowercase, I don't usually like all the shouting.

This creates a buffer called SCREEN that is 80 columns wide by 24 rows tall. It also defines the concept of a brush, which is just an ASCII character that is put into this buffer by PLOT-XY. Our line-drawing routine will use PLOT-XY to put “pixels” on the “screen” without caring about what they look like. Kind of a canvassy idea.

Now let's clear the screen:

SET-BRUSH +
FILL-SCREEN 
SET-BRUSH $

I use the + character for “off” and the $ character for “on” because they were about the same width in the variable-width font that my browser picked when plugging this code into jsForth. The trick where SET-BRUSH reads the next character in the code directly is cute but brittle; it only works interactively and will break weirdly in a : definition. WAForth can't handle it at all, it pops up a dialog box asking for you to type a character. Feel free to use 43 SCREEN-BRUSH C! to draw with + and 36 SCREEN-BRUSH C! to draw with $ if you want to follow along in WAForth. Define little helper words for them even, like BRUSH-+ and BRUSH-$. It's not a big problem, don't overthink it, but do make yourself comfortable.

An aside: How to draw a line

So let's talk for a minute about how Bresenham's line-drawing algorithm works. The Wikipedia article has a bunch of math and symbols but at its core it's really very simple. Start with a specific kind of line, that slopes upwards and to the right, but not steeper than 45 degrees.

1. Start at the bottom-left side of the line. Draw that pixel.
2. Move your X coordinate one to the right. Now you need to decide if the Y coordinate needs to move up one or stay where it is.
3. To do that, you keep track of a subpixel fraction; ie. you start in the middle of a pixel (0.5), and increment it by the amount that the line has risen over the last pixel: (y2-y1)/(x2-x1) or dy/dx.
4. If the fraction is >1, move Y up one pixel and subtract 1 from the fraction; the fraction value is now somewhere within the bottom half of the next highest pixel.
5. Now draw the next pixel and go back to step 2 until you end up at the top-right end of the line.

This is very simple! We then layer on just a few simple tricks:

• Instead of always moving along the X axis, for lines that are taller than they are long, we need to move along the Y axis. To do this we simply always move in the direction of the longer side, and run the decision logic along the shorter axis. This way the slope is never steeper than 45 degrees.
• If, for example, the line slopes down instead of up, when we decide whether to move along the Y axis, we need to move down one pixel instead of up. We can handle this by simply incrementing instead of decrementing along the appropriate axis.
• In the olden days, floating point numbers were very slow and integers were fast. Since the “error” value (really a fractional pixel location, but everyone calls it “error”) always has the same denominator, and we don't do anything more complicated than adding more fractions with the same denominator to it, we can just keep the denominator implicit and store the numerator in an integer. We choose 2 * dx (when x is the long axis) as the denominator so that we can easily start exactly on a half pixel (ie. our starting value is dx/2dx, and we increment by 2 * dy every step). It doesn't actually make a huge amount of difference what you use for a starting value though, as long as it's smaller than your implicit denominator then you'll end up with a line that starts and ends where you expect.

That's it! That's the whole thing.

Now back to writing Forth

So, first off, let's define the state that we'll need. Starting and ending X and Y coordinates, the current X and Y coordinates, and the fractional “error” value. Definitely need to remember all that.

VARIABLE LINE-X1 VARIABLE LINE-Y1 
VARIABLE LINE-X2 VARIABLE LINE-Y2
VARIABLE LINE-X  VARIABLE LINE-Y  VARIABLE LINE-ERR

Now we can start defining helper words. Let's write a couple of words to figure out the length of the line along each axis:

: LINE-DX ( -- dx ) LINE-X2 @ LINE-X1 @ - ;
: LINE-DY ( -- dy ) LINE-Y2 @ LINE-Y1 @ - ;

No sweat; just take x2 - x1 or y2 - y1. How about some words to decide which axis is longer, and what direction each axis is moving in?

: X-LONGER? ( -- f ) LINE-DX ABS LINE-DY ABS > ;
: LINE-LEFT? ( -- f ) LINE-DX 0 < ;
: LINE-UP? ( -- f ) LINE-DY 0 < ;

Even if you're not well-practiced reading postfix, I hope it's pretty clear what these are doing.

Now let's define some words for incrementing or decrementing, depending on which direction the line is going:

: LINE-XINC ( x -- x ) LINE-LEFT? IF 1- ELSE 1+ THEN ;
: LINE-YINC ( y -- y ) LINE-UP? IF 1- ELSE 1+ THEN ;
: LINE-INC ( x|y x? -- x|y ) IF LINE-XINC ELSE LINE-YINC THEN ;

LINE-INC is our first and only word to take two values on the stack – the top is a boolean that determines if we're talking about the X or Y axis. We will soon use it in conjunction with X-LONGER? to abstract away incrementing the “long” vs. “short” axis.

: LINE-LONG ( -- p ) X-LONGER? IF LINE-X ELSE LINE-Y THEN ;
: LINE-SHORT ( -- p ) X-LONGER? 0= IF LINE-X ELSE LINE-Y THEN ;
: LINE-LONG-INC! ( -- ) LINE-LONG @ X-LONGER? LINE-INC LINE-LONG ! ;
: LINE-SHORT-INC! ( -- ) LINE-SHORT @ X-LONGER? 0= LINE-INC LINE-SHORT ! ;

LINE-LONG-INC! is a little tricky, so let's walk through it:

• LINE-LONG returns a pointer to either the LINE-X or LINE-Y variable.
• @ fetches the current coordinate along the long axis.
• X-LONGER? pushes “true” onto the stack if X is the long axis (and thus the X coordinate is what's on the stack)
• LINE-INC calls LINE-XINC if X is long, or LINE-YINC if Y is long. This increments or decrements the value, depending on the direction of the line. The new coordinate is the one value left on the stack.
• LINE-LONG ! fetches the appropriate pointer again and stores the new value.

LINE-SHORT-INC! is basically the same, except with an 0= in there as a “logical not” for X-LONGER?. (It didn't quite seem worthwhile to define Y-LONGER? on its own.)

Now let's define some useful words for the error / fractional pixel calculation:

: LINE-LONG-LEN ( -- l ) X-LONGER? IF LINE-DX ELSE LINE-DY THEN ABS ;
: LINE-SHORT-LEN ( -- l ) X-LONGER? IF LINE-DY ELSE LINE-DX THEN ABS ;
: LINE-LONG-ERR ( -- err ) LINE-LONG-LEN 2 * ;
: LINE-SHORT-ERR ( -- err ) LINE-SHORT-LEN 2 * ;
: LINE-INIT-ERR! ( -- ) LINE-LONG-LEN LINE-ERR ! ;
: LINE-ERR-ACC ( -- err ) LINE-ERR @ LINE-SHORT-ERR + ;

LINE-INIT-ERR! defines the initial error value as half a pixel (with LINE-LONG-ERR being the implicit denominator). LINE-ERR-ACC fetches the current error and adds the appropriate fraction along the short axis, leaving the new value on the stack.

: LINE-ERR-INC! ( err -- err ) DUP LINE-LONG-ERR >= IF LINE-LONG-ERR - LINE-SHORT-INC! THEN ;
: LINE-ERR-ACC! ( -- ) LINE-ERR-ACC LINE-ERR-INC! LINE-ERR ! ;
: LINE-STEP ( -- ) LINE-LONG-INC! LINE-ERR-ACC! ;

LINE-ERR-INC! takes the incremented error value, determines if we've overflown the fraction into the next pixel, and if so, decrements the error value and increments the coordinate along the short axis. The updated error value is left on the stack. This is the only place in the algorithm where I chose to use a stack-manipulation word. I could have gotten by without it by just calling LINE-ERR-ACC a couple of times, but it would have made the definition longer and arguably harder to follow.

LINE-ERR-ACC! handles accumulating the error, incrementing the short axis if necessary, and storing the new error. Finally, LINE-STEP puts all the core logic together – increment along the long axis, then decide whether we need to increment along the short axis.

All that's left is to run it in a loop:

: PLOT-LINE-STEP ( -- ) LINE-X @ LINE-Y @ PLOT-XY ;
: DO-LINE ( -- ) LINE-INIT-ERR! LINE-LONG-LEN 0 DO PLOT-LINE-STEP LINE-STEP LOOP PLOT-LINE-STEP ;

: LINE ( x1 y1 x2 y2 -- ) 
  LINE-Y2 ! LINE-X2 ! DUP LINE-Y ! LINE-Y1 ! DUP LINE-X ! LINE-X1 ! DO-LINE ;

The final definition of LINE takes four values on the stack and immediately puts them into variables that are used by all the other words.

IMO, this is what Forth enthusiasts mean when they say things like “write lots of small definitions”, or “the stack shouldn't need to be very deep”, or “you don't need local variables”. There are 24 one line function definitions up there. No individual definition is particularly complicated or hard to read. We do virtually no stack manipulation.

Let's see it in action!

0 0 0 15 LINE
0 0 15 15 LINE
30 15 0 0 LINE
60 15 0 0 LINE
79 7 0 0 LINE
79 7 60 15 LINE
0 15 60 15 LINE

PRINT-SCREEN

$$$$$$++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$$$$$$$$$$$$$$$$$+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$+$+$$+$$$$++++++$$$$$$$$$$$$+++++++++++++++++++++++++++++++++++++++++++++++++++
$++$++$$+++$$$$++++++++++++++$$$$$$$$$$$++++++++++++++++++++++++++++++++++++++++
$+++$+++$$+++++$$$$+++++++++++++++++++++$$$$$$$$$$$+++++++++++++++++++++++++++++
$++++$++++$$+++++++$$$$++++++++++++++++++++++++++++$$$$$$$$$$$$+++++++++++++++++
$+++++$+++++$$+++++++++$$$$++++++++++++++++++++++++++++++++++++$$$$$$$$$$$++++++
$++++++$++++++$$+++++++++++$$$$+++++++++++++++++++++++++++++++++++++++++++$$$$$$
$+++++++$+++++++$$+++++++++++++$$$$+++++++++++++++++++++++++++++++++++++++++$$++
$++++++++$++++++++$$+++++++++++++++$$$$+++++++++++++++++++++++++++++++++++$$++++
$+++++++++$+++++++++$$+++++++++++++++++$$$$++++++++++++++++++++++++++++$$$++++++
$++++++++++$++++++++++$$+++++++++++++++++++$$$$++++++++++++++++++++++$$+++++++++
$+++++++++++$+++++++++++$$+++++++++++++++++++++$$$$+++++++++++++++$$$+++++++++++
$++++++++++++$++++++++++++$$+++++++++++++++++++++++$$$$+++++++++$$++++++++++++++
$+++++++++++++$+++++++++++++$$+++++++++++++++++++++++++$$$$+++$$++++++++++++++++
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lovely!

Now, there is plenty to criticize about this code. It does all kinds of redundant recalculation that in any sane C implementation would have been stashed away into a local, for example. But that's fixable with a little more effort; I might do another blog post where I apply some of Forth's fun metaprogramming tricks to that problem.

#forth #essays #code