fixed something weird going on with the tokenizer that might be a bug in clang

4 files changed, 114 insertions, 8 deletions

diff --git a/docs/00.html b/docs/00.html
new file mode 100644
index 0000000..cd0c352
--- /dev/null
+++ b/docs/00.html
@@ -0,0 +1,61 @@
+<h2>Declarations</h2>
+
+<p>In toc, declarations have the following syntax:
+<code>
+&lt;name&gt; :[:] [type] [= expression];
+</code></p>
+
+<p>The square brackets (<code>[]</code>) indicate something optional.</p>
+
+<p>All of the following statements
+declare an new variable <code>x</code> which is an integer, and has a value of 0:
+<code>
+x : int;
+x : int = 0;
+x := 0;
+</code>
+Note that in the first of those statements, although no expression
+is specified, it defaults to 0. This is not true in C,
+and there will eventually probably be an option to
+leave <code>x</code> uninitialized.</p>
+
+<p>If you wanted x to be a floating-point number, you could use:
+<code>
+x : float;
+x : float = 0;
+x := 0.0;
+</code></p>
+
+<p>Note that <code>0</code> can be used as both a <code>float</code> and an <code>int</code>eger, but
+when no type is specified, it defaults to an <code>int</code>, whereas <code>0.0</code>
+defaults to a <code>float</code>.</p>
+
+<p>Here are all of toc's builtin types and their ranges of values:</p>
+
+<ul>
+<li><code>int</code> - A 64-bit signed integer (always), -9223372036854775808 to 9223372036854775807</li>
+<li><code>i8</code> - An 8-bit signed integer, -128 to 128</li>
+<li><code>i16</code> - 16-bit signed integer, -32768 to 32767</li>
+<li><code>i32</code> - 32-bit signed integer, -2147483648 to 2147483647</li>
+<li><code>i64</code> - 64-bit signed integer (same as <code>int</code>, but more explicit about the size), -9223372036854775808 to 9223372036854775807</li>
+<li><code>u8</code> - An 8-bit unsigned integer, 0 to 255</li>
+<li><code>u16</code> - 16-bit unsigned integer, 0 to 65535</li>
+<li><code>u32</code> - 32-bit unsigned integer, 0 to 4294967295</li>
+<li><code>u64</code> - 64-bit unsigned integer, 0 to 18446744073709551615</li>
+<li><code>float</code> - A 32-bit floating-point number, -3.40282347e+38 to 3.40282347e+38</li>
+<li><code>f32</code> - A 32-bit floating-point number (same as <code>float</code>, but more explicit about the size)</li>
+<li><code>f64</code> - A 64-bit floating-point number, -1.7976931348623157e+308 to 1.7976931348623157e+308</li>
+<li><code>bool</code> - A boolean value, either <code>false</code> or <code>true</code>.</li>
+<li><code>char</code> - A character. The specific values are technically platform-dependent, but usually there are 256 of them.</li>
+</ul>
+
+<p>At the moment, it is not technically guaranteed that <code>f32</code>/<code>float</code> is actually 32-bit and that <code>f64</code> is actually 64-bit; they are platform dependent. Perhaps someday there will be a version of toc which does not compile to C, where that could be guaranteed.</p>
+
+<p>To make declarations constant, use <code>::</code> instead of <code>:</code>. e.g.</p>
+
+<p><code>
+x ::= 5+3; <br />
+y :: float = 5.123;
+</code></p>
+
+<p>Here, "constant" means constant at compile time, not read-only as it does in C. One interesting thing about toc is that normal functions can run at compile time, so pretty much any expression is a valid initializer for a constant, e.g. doing <code>x ::= some_function();</code> runs <code>some_function</code> at compile time, not at run time.</p>
diff --git a/docs/00.md b/docs/00.md
index e4e8d41..4998684 100644
--- a/docs/00.md
+++ b/docs/00.md
@@ -1,8 +1,8 @@
-## Declarations in toc
+## Declarations
 
-Declarations have the following syntax:
+In toc, declarations have the following syntax:
 ```
-<name> : [type] [= expression];
+<name> :[:] [type] [= expression];
 ```
 
 The square brackets (`[]`) indicate something optional.
@@ -41,10 +41,19 @@ Here are all of toc's builtin types and their ranges of values:
 - `u16` - 16-bit unsigned integer, 0 to 65535
 - `u32` - 32-bit unsigned integer, 0 to 4294967295
 - `u64` - 64-bit unsigned integer, 0 to 18446744073709551615
-- `float` - A 32-bit floating-point number,
-- `f32`
-- `f64`
-- `bool`
-- `char`
+- `float` - A 32-bit floating-point number, -3.40282347e+38 to 3.40282347e+38
+- `f32` - A 32-bit floating-point number (same as `float`, but more explicit about the size)
+- `f64` - A 64-bit floating-point number, -1.7976931348623157e+308 to 1.7976931348623157e+308
+- `bool` - A boolean value, either `false` or `true`.
+- `char` - A character. The specific values are technically platform-dependent, but usually there are 256 of them.
 
 At the moment, it is not technically guaranteed that `f32`/`float` is actually 32-bit and that `f64` is actually 64-bit; they are platform dependent. Perhaps someday there will be a version of toc which does not compile to C, where that could be guaranteed.
+
+To make declarations constant, use `::` instead of `:`. e.g.
+
+```
+x ::= 5+3;  
+y :: float = 5.123;
+```
+
+Here, "constant" means constant at compile time, not read-only as it does in C. One interesting thing about toc is that normal functions can run at compile time, so pretty much any expression is a valid initializer for a constant, e.g. doing `x ::= some_function();` runs `some_function` at compile time, not at run time.
diff --git a/docs/01.html b/docs/01.html
new file mode 100644
index 0000000..633295b
--- /dev/null
+++ b/docs/01.html
@@ -0,0 +1,18 @@
+<h3>A first program</h3>
+
+<p>The <code>main</code> function in toc corresponds to the <code>main</code> function in C. This function is called when your program is run. So, this is a valid toc program which does nothing:</p>
+
+<p><code>
+main ::= fn() {
+};
+</code></p>
+
+<p>It declares a constant, <code>main</code>, which is a function with an empty body. Note that the syntax for declaring functions is the same as the syntax for declaring constants (it isn't something like <code>fn main() { ... }</code>).</p>
+
+<p>Assuming you have compiled the compiler (see <code>README.md</code> for instructions about that), you can compile it with</p>
+
+<p><code>
+toc &lt;your filename&gt;
+</code></p>
+
+<p>You will get a file called <code>out.c</code>, which you can then put through your C compiler to get an executable file which does nothing. Congratulations! You've written your first toc program.</p>
diff --git a/docs/01.md b/docs/01.md
new file mode 100644
index 0000000..816f3e6
--- /dev/null
+++ b/docs/01.md
@@ -0,0 +1,18 @@
+### A first program
+
+The `main` function in toc corresponds to the `main` function in C. This function is called when your program is run. So, this is a valid toc program which does nothing:
+
+```
+main ::= fn() {
+};
+```
+
+It declares a constant, `main`, which is a function with an empty body. Note that the syntax for declaring functions is the same as the syntax for declaring constants (it isn't something like `fn main() { ... }`).
+
+Assuming you have compiled the compiler (see `README.md` for instructions about that), you can compile it with
+
+```
+toc <your filename>
+```
+
+You will get a file called `out.c`, which you can then put through your C compiler to get an executable file which does nothing. Congratulations! You've written your first toc program.