home .. forth .. colorforth mail list archive ..

[colorforth] Adventures with Building Applications


Hello All,

I have a project that has been in development for several years now, that supports a number of web applications. The project's customer base ranges from multinationals to startups, and has pretty much paid my bills for the past 5 years. The code base has been ported from C++, to Ocaml, to C. It has used Perl, Python, Ruby, Lua, and Javascript as embedded interpreters. The latest version of the project consists of 7k lines of C, and a cut down version of Mozilla's SpiderMonkey (which itself consists of 86k lines of C).

This web application server includes support for HTTP/1.1 (client & server), SSL, SMTP (client & server), SMPP (client), and IRC style protocol, and SQL through PostgreSQL. Unlike most web servers, it has a large number of resident "bots" that process requests, chat to users, and do general house keeping. All of the functionality is currently accessible through Javascript and doesn't require any special training beyond what a general Flash programmer knows.

But time doesn't stand still, and a recent project for a large multinational put serious strain on the system when the numer of users was an entire order of magnitude greater than the original project proposal! The biggest stumbling block has been the 86k lines of C code that run the Javascript engine, the same one found in Firefox. After reading through Webkit, and the new Tamarind engine from Adobe (with its 250k lines of C++ code & Javascript JIT), I pretty much gave up on using any of the existing engines. So I have begun the 4 major rewrite of the system. Over the past few months, I've been evaluating a wide range of technologies. I've played with customizing several smalltalk VMs, written PEG based language translators in several different languages, prototyped versions in the mainline scripting languages, and created 4 custom VMs in C and intel assembler. But ultimately, I went back and read through cmForth.blk, and the source listings for colorforth and got some inspiration.

The latest VM has a simple opcode set which consisted of the characters:

   !"#$%&'()*+,-./0123456789:;<=>?@[\]^_`abcdef{|}~

And for inspiration, this VM ran opcode that were very similar to Chuck's colorforth VM with a few minor modifications. Registers were allocated:

; %edx - top of stack  (doubles as the B register)
; %eax - next on stack
; %ecx - counter/utility
; %ebx - instruction pointer (now a utility register)
; %esi - data stack pointer
; %edi - memory register ( the A register )
; %esp - return stack pointer
; %ebp - free space pointer (aka here)

And the VM opcodes did slightly different things than they do in Forth, but there always were equivalents. For example while ! is xor, $ does a !a+. Also some words were repurposed:

[ pushes ebp, and starts compiling
] stops compiling
( immediately switches to interpreter context, and compiles
) evaluates the interpreter context
{ pushes ebp
} compiles a 0 cell
: takes the next word in the input buffer, and binds it to the value on the top of the stack.

Also any word not found in the dictionary, just leaves the address of the token on the stack. These changes made a Forth-like language for the VM very simple, and the implementation of the VM and bytecode interpreter, was under 500 lines of assembler. The trick being each opcode was aligned on a 32 byte boundary, and simply vectored to the byte [ebx + 8000] address. My favorite op codes are 0-9a-f which multiply the TOS by [base] and add the values they represent.

One of the biggest changes from Forth is :. Unlike traditional colon definitions the VM's Forth-like language would look like:

[ _ * ] : square

Where [ started compiling, pushing the address of the code on the stack, _ compiles a dup, * compiles a multiplication, and ] turns off compilation, : then bound the value pushed on the stack by [ to the word square. And everything else works pretty much like you'd expect. The other thing that makes
life really kinda neat was:

{ 0 : foo , "narf" : bar , }

with a minor tweak to : to mean , when between { } produced data strutures that look a lot like JSON, only with the value : key notation rather than key : value. At this point, I thought that I was almost done with the rewrite, I'd simply build a parser that reordered the Javascript into a postfix notation, and run it through the VM, and presto I'd be able to support most of the code the developers I've worked with have written over the years.

But why stop there! I've got a VM that looks a lot like Chuck's VM, and I know that he's got an optimizing native compiler for his code, so why not remove the whole redundant bytecode step? So I kept the byte codes, changed the definition of next to just be ret, and added a quick lookup table of the lengths of each VM instruction, and rewrote the _compile block to just copy VM instruction inline, compile literals as a dup; mov edx, 1234 and all dictionary calls as call instructions. Now I switched my VM from a bytecode interpreter to a native compiler! A few more tweaks, and in 630 lines of assembler, I now have a native compiler that optimizes tail calls, optimizes out some stack juggling combinations, and can still switch between interpret and compile modes (literally, compile & run vs. just compile).

And that's where it stands today. I am currently, rewriting my javascript -> forth parser, in javascript, and playing around with a colorforth -> javascript compiler in javascript as well. Colorforth in your browser? Sure why not? Your browser in colorforth? Sure why not? Chuck was right when he said a browser was simple, it just needs to be done :) By implementing javascript in forth, and forth in javascript, either can run anywhere.

Going forward, I fully expecting to have a Forth/Javascript engine powering version 4 of this web application server project. I am currently planning on rewriting the 7K lines of C in a combination of Forth and Javascript (which compiles to Forth so ultimately it is Forth, but my web guys don't need to know that), and expect to have the entire code based reduced to under 4k lines of code (down from 94k!).

In the future, since I've got the code to boot the VM from disk, I'm also planning on writing an Ethernet device driver for the couple cards I use, and a TCP/IP stack, so I can run this server on bare hardware. With the availability of virtualization software like VMWare, VirutalBox, QEMU, etc, I can easily see a migration path from FreeBSD/Linux/MacOS X to plain bare hardware.

Looking back at this development process, I could have just used the Colorforth 2.0a to implement the VM, but without the equivalent of Jay Melvin's shadow block listings of cmforth for the first 18 blocks of CF, I wasn't comfortable implementing it on top of so much black magic. Building a native javascript compiler is hard enough without knowing how the internals of the compiler work :)

Anyways, I hope to do a postmortem for this project in a few months. The source code and executables will most likely be available by the end of the year under some form of open source license. I hope this inspires some of you to dust off your CF images, and build some apps.
Dave Goehrig

--

David J. Goehrig


Email: dave@xxxxxxxxxxxxxx


---------------------------------------------------------------------
To unsubscribe, e-mail: colorforth-unsubscribe@xxxxxxxxxxxxxxxxxx
For additional commands, e-mail: colorforth-help@xxxxxxxxxxxxxxxxxx
Main web page - http://www.colorforth.com