System architecture

The plan is to start with the full system as an interpreter in Scheme, then move stepwise to a fully self-hosted compiler.

The file julia-interpreter.scm will detail exactly what the compiler and run time system need to implement. The only interface needed is functions to convert between Scheme representations of ASTs and Julia representations, to allow macros written in Julia. So the intermediate step is to keep the whole parser and front end in Scheme, and replace julia-interpreter.scm with a real code generator and RTS. At that point we would have a respectable system. Then the final, somewhat optional, step is to rewrite the front end and drop the Scheme environment.

Obviously we can't waste too much time on the interpreter. We'll only use it to finalize the language design. It will also provide the opportunity to write real working code transformations that can be kept intact when we switch to the new back end.

Gambit will be used for the Scheme environment, with code generation targeting LLVM through its C++ interface.

Putting the pieces together

How can this all fit together?

Code will execute in a C/C++ run-time system that performs code generation, maintains the object model, and provides basic library routines. The input to the system consists of fully-lowered ASTs that can be converted to code by a straightforward one-to-one process.

These ASTs are generated by Scheme code, beginning with the parser and ending with type inference and other optimizations. However, in order to do these optimizations the front end needs to know a certain amount about the state of the system, specifically the values of global constant types, functions (and their ASTs), and values (e.g. numeric constants). There may be other global variables, but if they aren't constant they are opaque (type Any) to the front end. To supply this information, the front end maintains a skeleton environment. It parses and interprets top-level definitions to a limited extent to extract the information it needs.

The back end can examine ASTs using Gambit's C API. The only other essential interface is feeding run-time types back to the front end. This can be done by converting the Type objects to Scheme representation, and passing those to the front end to request a new AST.

Julia macros can be supported by having the front end ask the back end (via Gambit's FFI) to call the needed Julia function, and convert the result from Julia objects to Scheme s-exprs.

Initially this setup will entail some duplicated code. Both pieces will need implementations of subtype?, instantiate-type, and the logic needed to process top-level forms like type definitions. Some duplication could be removed by calling the C versions using Gambit's FFI. If Julia became self-hosted all the duplication would go away and we would gain a bit of efficiency.