A Tour of tiko

The last chapter got a compiler working. This one writes something with it — a program that reads a file, counts the words in it, and reports the three most common. It is small enough to hold in your head and big enough to need most of what the language is made of: the standard library, strings, a hash map, iteration, sorting, a lambda, and errors that are values rather than surprises.

Nothing here is explained completely. The point is to see the shape of a tiko program before taking it apart, and every idea below is picked up properly in a later chapter. Read it the way you would read someone else's code: get the sense of it, and let the details wait.

The file we are counting is three lines:

the quick brown fox
jumps over the lazy dog
the dog sleeps

Reaching the standard library

tiko's library comes in modules, and a program says which ones it wants:

'' step1.tiko -- read the file, and say so if it is not there.
#import "std/fs"

var loaded = fs.ReadAllText( "words.txt" )
if loaded.IsOk( ) = false then
    print "cannot read words.txt"
    end 1
end if

var text = loaded.Value( )
print "characters: " & len( text )

characters: 59

#import "std/fs" makes the filesystem module available under the name fs, and every name it provides is reached through that prefix — fs.ReadAllText. That prefix is not decoration; it is how a reader of your code knows where a function came from.

The interesting part is what ReadAllText gives back. Not a string: a Result, which is either the text or the reason there is no text. There is no exception to catch and no error code to remember to check, because the value itself will not hand over its contents until you have asked whether it has any:

• loaded.IsOk( ) — did it work?
• loaded.Value( ) — the text, once you know it worked.

A program that forgets to check does not silently read nonsense; it fails to compile in the shape you would have wanted. Chapter 18 is about designing with Result.

Splitting the text into words

'' step2.tiko -- split the text into words.
#import "std/fs"
#import "std/strings"

var loaded = fs.ReadAllText( "words.txt" )
if loaded.IsOk( ) = false then
    print "cannot read words.txt"
    end 1
end if

var flat = strings.Replace( loaded.Value( ), !"\n", " " )
var words = strings.Split( flat, " " )
print "words: " & words.Count

for each w in words
    if len( w ) > 0 then print "  [" & w & "]"
next

words: 13
  [the]
  [quick]
  [brown]
  [fox]
  [jumps]
  [over]
  [the]
  [lazy]
  [dog]
  [the]
  [dog]
  [sleeps]

Two library calls do the work. strings.Replace flattens the newlines into spaces, and strings.Split cuts the result on spaces and hands back an array. words.Count is how many it found.

for each w in words walks the array without an index, without a bound, and without the off-by-one that an index loop invites. The loop variable's type comes from the array, so w is a ustring — tiko's string type, which holds text rather than bytes and is the same on every platform.

The if len( w ) > 0 is there because splitting on a single space leaves empty pieces wherever two spaces met. Real text is untidy; the program says so rather than pretending.

Counting

'' step3.tiko -- count each word.
#import "std/containers"
#import "std/fs"
#import "std/strings"

var loaded = fs.ReadAllText( "words.txt" )
if loaded.IsOk( ) = false then
    print "cannot read words.txt"
    end 1
end if

var flat = strings.Replace( loaded.Value( ), !"\n", " " )
var words = strings.Split( flat, " " )

dim counts as containers.Map( of ustring, int32 )
for each w in words
    if len( w ) > 0 then counts[ LCase( w ) ] += 1
next

print "distinct words: " & counts.Count
print "the: " & counts[ "the" ]
print "dog: " & counts[ "dog" ]

distinct words: 9
the: 3
dog: 2

containers.Map( of ustring, int32 ) is a hash map from text to a number. The ( of … ) is tiko's generics syntax: this is not a map of "anything", it is a map whose keys are ustring and whose values are int32, checked as thoroughly as if you had written the type by hand. Chapter 17 is about writing your own.

counts[ LCase( w ) ] += 1 does the whole count. Indexing a map with a key that is not there inserts it with a zero value, so the += 1 works the first time a word appears as well as the fifth. LCase folds the case, so The and the are one word.

Ranking, with a lambda

A map has no order, so the last step copies the counts into an array and sorts it:

dim table as containers.Array( of Tally_ )
for each entry in counts
    dim as Tally_ t = ( entry.Key, entry.Value )
    table.Push( t )
next

containers.SortBy( table, _
    function( byref a as Tally_, byref b as Tally_ ) as boolean
        return a.count > b.count
    end function )

for each entry in counts walks the map, and each entry carries a Key and a Value.

containers.SortBy takes the array and a lambda — a function written where it is used, with no name. This one answers a single question: does a come before b? Sorting by descending count is a.count > b.count, and changing the ranking means changing that one line. Chapter 20 is about lambdas, including the ones that capture variables from around them.

The _ at the end of the SortBy line continues the statement onto the next line. tiko has no statement terminator, so a line break ends a statement unless you say otherwise.

Reporting

print "the three most common words"
for i as int32 = 0 to 2
    print using "  \    \ #"; table[ i ].word; table[ i ].count
next

print using formats output against a picture: \ \ is a text field of that width, and # is a digit. It is the one piece of this program that will look unfamiliar to most readers, and it is worth the surprise — lining up columns with string concatenation is miserable.

The whole program

#import "std/containers"
#import "std/fs"
#import "std/strings"

type Tally_
    as ustring word
    as int32 count
end type

var loaded = fs.ReadAllText( "words.txt" )
if loaded.IsOk( ) = false then
    print "cannot read words.txt"
    end 1
end if

var flat = strings.Replace( loaded.Value( ), !"\n", " " )
var words = strings.Split( flat, " " )

dim counts as containers.Map( of ustring, int32 )
for each w in words
    if len( w ) > 0 then counts[ LCase( w ) ] += 1
next

dim table as containers.Array( of Tally_ )
for each entry in counts
    dim as Tally_ t = ( entry.Key, entry.Value )
    table.Push( t )
next

containers.SortBy( table, _
    function( byref a as Tally_, byref b as Tally_ ) as boolean
        return a.count > b.count
    end function )

print "the three most common words"
for i as int32 = 0 to 2
    print using "  \    \ #"; table[ i ].word; table[ i ].count
next

the three most common words
  the    3
  dog    2
  quick  1

Forty lines, no memory management, no build file, and one command to build it:

tiko wordcount.tiko -o wordcount
./wordcount

The result is a native executable that depends on nothing you have to ship beside it.

What you have seen

Each of these has a chapter of its own:

Summary

• A program reaches the library with #import "std/…", and uses the module's name as a prefix.
• Fallible calls return a Result: ask IsOk( ), then take Value( ).
• var infers a variable's type; dim as T states it.
• for each walks an array without an index; the loop variable's type comes from the elements.
• Map( of K, V ) is a checked hash map, and indexing a missing key inserts it.
• A lambda is a function written where it is used — SortBy takes one to decide the order.
• One command compiles and links a native executable.

Exercises

1. Report the longest word in the file as well as the most common ones. {solution="ex03-01"}
2. List the words that appear exactly once, in alphabetical order. OrderedMap keeps its keys sorted, where Map does not. {solution="ex03-02"}
3. Make the program take the file name from the command line, falling back to words.txt when none is given. Chapter 1 showed command( 1 ).
4. The ranking prints three words. Make it print the top three counts, so that words tied for third all appear.