Last time the bot learned some text commands. Now it’s time to draw some pretty pictures. Today I’d like to add a couple of chart commands.

Using Telegram API it possible to send an image (as well as a video or any other document), it’s not limited to just text messages. All I need is a way to generate an image and push it down the wire. After a quick search, I stumbled upon go-chart. go-chart renders charts into in-memory PNG and SVG files and this is exactly what I need. Here’s an example:

I’m not so interested in visualizing stock prices though. I have my own data to deal with. The first step would be to draw something simple. Last time I added the /top command that shows 10 most logged events. It would be good to see that as a bar chart as well. Please welcome the /topChart command.

The charts look a bit too small and ugly like this, but they blow up once you click on them. The code for this chart is pretty simple:

func (c context) topChart(args string) {
    num := parseTopArgs(args)

    // Get values from the DB and convert
    values := make([]chart.Value, 0, num)
    for _, e := range c.getTopEvents(num) {
        values = append(
            values,
            chart.Value{Label: e.name, Value: float64(e.count)},
        )
    }

    // Chart settings
    response := chart.BarChart{
        Title:      fmt.Sprintf("Top %d events", num),
        TitleStyle: chart.StyleShow(),
        Background: chart.Style{
            Padding: chart.Box{
                Top: 40,
            },
        },
        Width:    num * 100,
        Height:   512,
        BarWidth: 80,
        XAxis:    chart.StyleShow(),
        YAxis: chart.YAxis{
            Style:          chart.StyleShow(),
            ValueFormatter: chart.IntValueFormatter,
            Range: &chart.ContinuousRange{
                Min: 0,
                Max: values[0].Value,
            },
        },
        Bars: values,
    }

    // Render and send
    c.sendChart(response)
}

To get the values I use the following query:

SELECT name, COUNT(name) freq FROM events
    WHERE user = ?
    GROUP BY name
    ORDER BY freq DESC
    LIMIT 10

And rendering the cart is trivial with go-chart

buffer := &bytes.Buffer{}
err := chartSettings.Render(chart.PNG, buffer)

Now the buffer holds the bytes for a PNG file. It could be saved to disk or sent over the wire via Telegram API like this:

image := tgbotapi.FileBytes{Name: "chart.png", Bytes: buffer.Bytes()}
_, err := c.bot.Send(tgbotapi.NewPhotoUpload(c.message.Chat.ID, image))

Easy-peasy.

Another idea is to use a bar chart to show daily activity for a specific event like this:

Up close it looks like this:

What I thought would be very cool is to draw a diagram similar to GitHub activity for a selected event. On GitHub it looks like this:

Easy? Well, not really. I spent a bunch of time trying to replicate it. I tried to save myself some trouble at first and use the stacked bar chart, but it didn’t work out. So I had to start from scratch, roll up the sleeves and do it myself. About five hundred lines later I have this:

Close enough for my needs. There’s quite a bit of code to be covered in this post. Maybe I make a separate post some time where I describe what it takes to draw a chart using go-chart. It is doable. The problem is testing. Especially if you want to support all kinds of different options, like configurable axis labels, legend, title, font sizes and so on. It’s quite tedious. I declare: works on my machine! Ship it!

What’s next

The bot seems to have all the basic features. Sure, they are not very robust and not polished, but they kinda work. It’s time to start using it. I need to figure out a way to deploy it somewhere and have it running. DevOps stuff. I wonder if I’m gonna need a Kubernetes cluster for that.

If you’re curious, the code is available on GitHub. This version is tagged day-6.

Also published on DEV and Medium

and I got this

That pissed me off right then and there. Why would any remote admin tell me what to do on my local machine? Even if it’s a company machine. When I do not use the VPN or maybe I’m not even connected to any WiFi, why should I have this running? It’s just silly to try to prevent me from doing something on the machine where I have the root access. When this gets pushed onto a developer’s laptop (as opposed to an accountant or an HR clerk) it simply becomes a challenge.

It has no impact on the security of the machine or the corporate network when it’s simply running in the background. This is not anti-virus software. It has an impact on the battery life and other resources like RAM and CPU though. And I need those. It’s exactly this type of programs that don’t exit, hang around and prevent me from running yet another Electron app on my laptop. Sometimes one is not enough, you know.

I guess I’d have to get my hands dirty instead of doing something I was actually going to do. Simply killing it from the command line didn’t work. It just starts over. Investigation it is, then.

First, I took a quick look at the list of the open files from the Activity Monitor. There I found a log file (bottom row):

The tool developers were very kind and dumped about half a meg of stuff on every restart cycle into the log. So it’s wasting my disk space as well then. Amongst thousands of lines, I found a reference to /Library/Application Support/Pulse Secure/Pulse/connstore.dat. It sounded promising. After poking around in that file and trying this and that, I found a parameter that is responsible for that.

ive "921sn438-qoo8-4pp4-85p7-68q5s2592o32" {
    ...
    connection-policy-override: "false"
    ...
}

When this parameter is changed to true and the tool is restarted I was able to quit. Voilà!

Yes, I do.

And here the script that automates the whole thing:

cd '/Library/Application Support/Pulse Secure/Pulse'
sudo sed -i '' \
      s'/connection-policy-override: "false"/connection-policy-override: "true"/' \
      connstore.dat
sudo killall PulseTray 'Pulse Secure' dsAccessService

Dear developers and sysadmins, please don’t do that again. You’re just wasting everybody’s time. Rather put your efforts into making the software more reliable, less resource hungry, more secure.

Also published on DEV and Medium

The go-telegram-bot-api/telegram-bot-api package provides some functions to check whether there’s a command in the received message and lets you extract it and its arguments.

for update := range updates {
    if update.Message.IsCommand() {
        command := update.Message.Command()
        arguments := update.Message.CommandArguments()
    }
}

After some light refactoring my bot is ready to accept commands as well as regular text. Actually, the plain text is simply mapped to the /add command.

if message.IsCommand() {
    switch command := message.Command(); command {
    case "a", "add":
        c.add(message.CommandArguments())
    case "e", "export":
        c.export()
    case "h", "help":
        c.help()
    case "s", "since":
        c.since(message.CommandArguments())
    case "t", "top":
        c.top()
    case "test":
        c.test()
    default:
        c.sendText(fmt.Sprintf("Eh? /%s?", command))
    }
} else {
    c.add(message.Text)
}

/test

The simplest command would be /test, it just sends some hardcoded text back. It’s pretty trivial to implement:

func (c context) test() {
    c.bot.Send(tgbotapi.NewMessage(c.message.Chat.ID, "It works"))
}

/export

Export is a very important command. I consider it so important that I implemented it first. It allows the user to download all of its data stored in the bot’s database. Say no to vendor lock-in.

This is really annoying these days. Take any app, like a fitness tracker app for example. It keeps track of your runs and stores your data somewhere on the phone or in the cloud. And one day it stops working or you decide to start using another app. And what do you do? How do you migrate your data? Usually, there’s no easy way or no way at all. You’d have to abandon all of your history and your progress and start over in the new app.

I don’t want that for my yet non-existent users. Let them take their data home whenever they want. That’s why I have the /export command. Just say /export to the bot and it’s happy to comply:

To generate the CSV file I use the built-in package encoding/csv. It’s very easy to export all the rows into a CSV file :

buffer := &bytes.Buffer{}
csv := csv.NewWriter(buffer)

// For each row
for ... {
    csv.Write([]string{name, date})
}

// Flush when done (important!)
csv.Flush()

// Send
c.bot.Send(tgbotapi.NewDocumentUpload(
    c.message.Chat.ID,
    tgbotapi.FileBytes{
        Name:  "data.csv",
        Bytes: buffer.Bytes()}))

/add and /since

The /add and the /since commands are bread and butter of this bot. As the names imply, one is for adding the events and the other is for checking when the event was last added. /add is a 2-in-1 command, like shampoo & conditioner in one bottle, it displays the time before adding.

/top

To practice SQL, I added a relatively useless command to display 10 most logged events.

SELECT name, COUNT(name) freq FROM events
    WHERE user = <user-id>
    GROUP BY name
    ORDER BY freq DESC
    LIMIT 10

On my really dumb dataset it looks like this:

I have to say that I’m pretty impressed with SQL so far. I knew theoretically what it could do. But now I get to try it and it’s really cool how I can just write a simple query instead of writing a bunch of code with loops and variables. Probably more efficient too.

What’s next

The bot is getting smarter. Now it could respond in different ways to a few basic commands. More is coming. I’d like the bot to be able to draw charts and show some statistics, like how often something happened or distribution throughout the day/week/month. Do you have any ideas?

If you’re curious, the code is available on GitHub. This version is tagged day-5.

Also published on DEV and Medium

Today, I’m going to focus on features. I’d like to make my bot intelligent. Not really AI like, rather just sounding a bit better than cmd.exe when you type in a wrong command. I’m not gonna make anything big, more like dipping the toe in the water to see how it is.

First, I’d like to find the last event with the same name, since I’m storing them in the database, and send back the date the event. Imagine the interaction like this:

me: eat
bot: Previous 'eat' happened 6 hours ago

To find the event with the same name I have to ask SQLite to give me the row where the user is the same, the name of the event it the same and I want the last row after it’s sorted by date. Converting English to SQL I get:

SELECT date FROM events
    WHERE user = ? AND name = ?
    ORDER BY date
    DESC LIMIT 1

Pretty simple, eh? Now, converting this to Go and crawshaw.io/sqlite I get the following:

// Default response
response := fmt.Sprintf("Fist time for '%s'", name)

// Get the last event with the same name and format the response
err := sqlitex.Exec(connection,
    "SELECT date FROM events "+
        "WHERE user = ? AND name = ? "+
        "ORDER BY date "+
        "DESC LIMIT 1",
    func(s *sqlite.Stmt) error {
        response = formatResponse(s.GetInt64("date"), name)
        return nil
    },
    message.From.ID,
    name)

// Send the message back to the user
go func() {
    bot.Send(tgbotapi.NewMessage(message.Chat.ID, response))
}()

Where formatResponse is very rudimentary:

func formatResponse(date int64, name string) string {
    last := time.Unix(date, 0)
    return fmt.Sprintf("Previous '%s' happened on '%v'", name, last)
}

Now the interaction looks like this:

Not exactly the smartest bot on the planet, but we’re going somewhere.

Now, I would like to make it sound a bit less dumb and bit more human, which are not always going together. In this case, they are. I don’t want the bot to say things like happened on '2019-04-04 14:13:57 +0200 CEST', but rather something like 8 minutes since or 1 year since. Believe it or not, but there’s a package for exactly that. Welcome hako/durafmt. With its help it’s very easy to turn the conversation into something like this:

It is much more readable. To make it work I just had to modify the formatResponse function a bit (now it also has to take the current message date):

func formatResponse(name string, date int64, prevDate int64) string {
    prev := time.Unix(prevDate, 0)
    now := time.Unix(date, 0)
    duration := durafmt.ParseShort(now.Sub(prev))
    return fmt.Sprintf("%s since last '%s'", duration, name)
}

Easy peasy. With not so many lines and in under an hour of work I have a bot that speaks human and can tell me when something happened the last time. Ship it! 🚢

If you’re curious, the code is available on GitHub. This version is tagged day-4.

Also published on DEV and Medium

Normally one should think, then do. Though not ideal, it’s also possible to do it the other way around when you use git. But it’s important not to forget to think at some point. Like I almost did and almost moved on to pile on more bugs on top of what I just introduced.

I remembered reading or watching something about SQLite the other day and they made it clear that I’d have to make sure not to access the database from multiple places at the same time. Go is made for concurrency, so I’m sure go-sqlite3 I was using has it under control. Not exactly, as it turned out.

From the FAQ:

Can I use this in multiple routines concurrently?

Yes for readonly. But, No for writable. See #50, #51, #209, #274.

And just like that I went down the major rabbit hole. I searched and researched, googled and binged (no, not really), read the source of go-sqlite3 and the docs for SQLite (which are pretty good by the way). All of that just to find myself in a situation that I don’t know if I can trust the go-sqlite3 package to handle my database work. Even though the FAQ states that write concurrency is not supported, I found contradicting statements. People were saying it’s fine if I use multiple connections. But I wasn’t going to open a new connection every time.

Long story short, I found a blog post which addressed exactly the same problem and offered a solution for it (and a couple of others) in a form of a Go package.

go get -u crawshaw.io/sqlite

According to David, the author of the post and the package, I should be able to trust this package to handle concurrency reliably. Now, I don’t simply establish a connection to the database, but rather create an explicit pool of those of the size I desire (16 in this case):

import "crawshaw.io/sqlite/sqlitex"

func openDB() *sqlitex.Pool {
    db, err := sqlitex.Open("./since.db", 0, 16)
    if err != nil {
        log.Panic(err)
    }

    return db
}

Now, every time I want to access the database I have to get a connection from the pool and not to forget to put it back when I’m done. Getting a connection from the pool might block until a connection becomes available.

func execSQL(db *sqlitex.Pool, sql string) {
    connection := db.Get(nil)
    defer db.Put(connection)

    err := sqlitex.Exec(connection, sql, nil)
    if err != nil {
        log.Panic(err)
    }
}

A side note on the error handling. I currently do not handle errors on purpose to not slow myself down. But I don’t ignore them either. Like any self-respecting software engineer I panic when I receive an error. For some errors, like not being able to open the database on startup, it’s totally fine to panic. But if there was an error with one of the messages, panic is not such a good choice. It’s like shutting down the server when we should have just returned HTTP/404.

Now, storing the incoming message in the database becomes this:

func store(message *tgbotapi.Message, db *sqlitex.Pool) {
    connection := db.Get(nil)
    defer db.Put(connection)

    err := sqlitex.Exec(
        connection,
        "INSERT INTO events (user, name, date) VALUES (?, ?, ?);",
        nil,
        message.From.ID,
        message.Text,
        message.Date)

    if err != nil {
        log.Panic(err)
    }
}

One small thing I don’t like about this is that I’m forced to use positional SQL arguments if I want to use sqlitex.Exec. If I wanted to use the column names like "... VALUES ($user, $name, $date)", I’d have to use a much wordier API. Prepare statement myself and then step through it. Like this:

func store(message *tgbotapi.Message, db *sqlitex.Pool) {
    connection := db.Get(nil)
    defer db.Put(connection)

    insert := connection.Prep("INSERT INTO events (user, name, date) VALUES ($user, $name, $date);")
    insert.SetInt64("$user", int64(message.From.ID))
    insert.SetText("$name", message.Text)
    insert.SetInt64("$date", int64(message.Date))

    _, err := insert.Step()
    if err != nil {
        log.Panic(err)
    }

    // Done with this query
    // TODO: Is it really needed? What happens when this isn't called?
    err = insert.Reset()
    if err != nil {
        log.Panic(err)
    }
}

If I see that positional arguments become a problem, I’ll switch to this way of doing things.

As much as I wanted to make my bot less dumb this time, I only managed to switch libraries while trying to bullet-proof my SQLite access methods. Just refactoring, no features. Here’s a typical day of a software engineer for you. Well, the next day then.

If you’re curious, the code is available on GitHub. This version is tagged day-3.

Also published on DEV and Medium

I quickly shopped around for an SQLite package for Go and found go-sqlite3. This seems to be a very popular package. It allows me to use the standard database/sql package and it acts as a driver, which allows the Go database engine to talk to SQLite databases. So far so good. I’ll use that then.

Since I’m writing everything in Go, I desperately need to use goroutines somewhere. So the first thing I did was to move the reply functionality into a goroutine.

for update := range updates {
    go reply(bot, update)
}

That should make things go FAST! Never mind the fact that Telegram will not allow frequent requests for any extended period of time. I’ve gotta try anyway.

To use the database I have to open it first. It’s done like this:

func openDatabase() *sql.DB {
    db, err := sql.Open("sqlite3", "./since.db")
    if err != nil {
        log.Panic(err)
    }

    _, err = db.Exec("CREATE TABLE IF NOT EXISTS events (" +
        "id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, " +
        "user INTEGER, " +
        "name TEXT, " +
        "date INTEGER);")

    if err != nil {
        log.Panic(err)
    }

    return db
}

I also create a table if it doesn’t exist. Since I barely use SQL, I prefer to SCREAM so people can hear me. And maybe the DB will get a sense of urgency and process my queries faster.

Once the database successfully opened and no one panicked, it’s time to start saving incoming events to it. I do it like this:

func store(message *tgbotapi.Message, db *sql.DB) {
    _, err := db.Exec("INSERT INTO events (user, name, date) VALUES ($1, $2, $3);",
        message.From.ID,
        message.Text,
        message.Date)
    if err != nil {
        log.Panic(err)
    }
}

store(update.Message, db)

Now, every time I send a message to my bot it stores its content in the database. It’s a start. The bot is still responding with the same reply as before. It’s still really dumb. We’ll get to make it smarter in the next day.

If you’re curious, the code is available on GitHub. This version is tagged day-2.

Also published on DEV and Medium

Idea

I want to be able to send simple commands to the chat and let the bot remember what happened and when. It’s a bit like a log file. I want to be able to ask the bot when or how often something happened. This is what it could look like:

me: eat
bot: Recorded 'eat'. Last 'eat' happened 8 hours ago.

me: sleep, 10 hours ago
bot: Recorded 'sleep' at 23:15, yesterday.
     Last 'sleep' happened 1 day 12 hours ago.

me: /since run
bot: Last 'run' happened 2 days 15 hours ago.

me: /stats blog
bot: So far 'blog' happened 20 times, average
     time between events is 4 days 5 hours.

The questions and answers are a bit clumsy, but I don’t think I’m going to implement the whole natural language processing thing here. It’s a simple bot, remember? I have ideas for many more commands that could be useful, but I’ll start small with recording and basic querying.

Architecture

It’s kind of a big word, eh? To talk to the Telegram servers I’m going to use Go bindings for the Telegram Bot API. I’m planning on using webhooks in the final version. I’ll start with the poll loop though since it’s possible to run the polling bot locally.

I’m planning to use SQLite to store the data. I like zero configuration and setup. I like that I could just copy the database anywhere I want. This way I could organize backups very easily. For example, I could just email it to myself every now and then. Unless I open the bot to the world, I don’t expect the database to grow much.

Code

Let’s jump straight to coding then. The Go Telegram Bot API provides a good starting point in the README. It’s an echo bot. It just sends the received text back to the user. I’ll start with that.

For the bot to connect to the Telegram network it has to have a username and an API token. Both thing you could get by talking to a special bot on Telegram. The bot is called The Botfather.

The interaction is quite trivial and pleasant. Just follow the dialog and he’ll give you what you need. Just make sure not to ask for too much.

The API token I got from the Botfather should be kept private. Who owns the token owns the bot. To keep it out of the repo I added a bit of code to read it from a simple JSON config. Besides that, it’s just the code from the README.

// Config represents the structure of the config.json file
type Config struct {
    Token string `json:"token"`
}

func readConfig() Config {
    file, err := os.Open("config.json")
    if err != nil {
        log.Panic(err)
    }

    defer file.Close()
    bytes, err := ioutil.ReadAll(file)
    if err != nil {
        log.Panic(err)
    }

    var config Config
    err = json.Unmarshal(bytes, &config)
    if err != nil {
        log.Panic(err)
    }

    return config
}

That should do it for day 1. If you’re curious, the code is available on GitHub. This version is tagged day-1.

Also published on DEV and Medium

Assert.That(_, Is.EqualTo(_))
Assert.That(_, Throws.TypeOf<_>())

or

_._(_, _)

if you’d like to go extreme and match every member function call with two parameters.

What I would like to be able to do, though, is to transform the code, not just match. I’d like to specify how to convert the patterns to the form I’m after. For example:

Assert.That(@actual, Is.EqualTo(@expected)) -> Assert.Equal(@expected, @actual)

Well, this is exactly the syntax I’m going to use. The placeholders @actual and @expected match any expression subtrees in the AST, anything that could be placed as arguments in those two positions. It’s actually a valid C# code as it allows @ at the beginning of an identifier. In my matcher I treat any identifier that starts with an @ as a placeholder.

The code in the previous post only matched the patterns (_ matches anything, a bit like .* in a regex). Now we need to store the named matches, like the match groups in a regex. The code would have to be changed from

// A placeholder matches anything
if (IsPlaceholder(pattern))
    return true;

to

// A placeholder matches anything
if (IsPlaceholder(pattern))
{
    var name = pattern.ToFullString();
    if (name != "_")
        Matches[name] = code;

    return true;
}

The rest of the matching code stays practically untouched. We have a matcher, the next step would be to write a “substituter” or “replacer”, the replace part of the find-and-replace tool.

Since I like to keep my code DRY, first I spent a bunch of time thinking about how to reuse the matching code for the substitution part. I failed there. So I decided to go with WET instead and rewrite that giant switch once again.

// _variables is populated in the constructor
public SyntaxNode Replace(SyntaxNode template)
{
    // A placeholder found. Substitute.
    if (IsPlaceholder(template))
    {
        var name = template.ToFullString();
        if (_variables.TryGetValue(name, out var v))
            return v;

        return template;
    }

    switch (template)
    {
    case ArgumentSyntax t:
        return t.Update(t.NameColon,
                        t.RefKindKeyword,
                        (ExpressionSyntax)Replace(t.Expression));
    case ArgumentListSyntax t:
        return t.Update(t.OpenParenToken,
                        Replace(t.Arguments),
                        t.CloseParenToken);
    case IdentifierNameSyntax t:
        return t.Update(Replace(t.Identifier));
    case InvocationExpressionSyntax t:
        return t.Update((ExpressionSyntax)Replace(t.Expression),
                        (ArgumentListSyntax)Replace(t.ArgumentList));
    case LiteralExpressionSyntax t:
        return t.Update(Replace(t.Token));
    case MemberAccessExpressionSyntax t:
        return t.Update((ExpressionSyntax)Replace(t.Expression),
                        t.OperatorToken,
                        (SimpleNameSyntax)Replace(t.Name));
    case GenericNameSyntax t:
        return t.Update(Replace(t.Identifier),
                        (TypeArgumentListSyntax)Replace(t.TypeArgumentList));
    case TypeArgumentListSyntax t:
        return t.Update(t.LessThanToken, Replace(t.Arguments), t.GreaterThanToken);
    default:
        return template;
    }
}

This code recursively walks the replace template AST and substitutes the placeholders (like @actual and @expected) with the matches found in the previous step. The end result of this substitution is then swapped with the matched expression node in the original source file AST. And that’s it. Works like a charm.

A side note on the choice of keywords here. I’m not an OOP aficionado. I think it has its place and in C# it’s quite natural to use classes and member functions to represent and do stuff. Though most of the time I try to stick to more of a functional style, where functions don’t have any hidden state and take all their input as parameters and return the result. Most of my functions and classes are static, actually. But common sense wins most of the time for me (or so I hope). So, in this case, I went with a class and non-static functions to not to pass around the state into every function, as there are quite many recursive calls. I kept it DRY in a sense.

I wrote a small driver application for this algorithm. It takes care of the loading, parsing, writing out the result. All the patterns are hardcoded for now, but there’s nothing but my laziness stopping me from putting that into a config file.

Here are the patterns I used to convert the bulk of my tests:

Assert.That(@actual, Is.EqualTo(true))      -> Assert.True(@actual)
Assert.That(@actual, Is.EqualTo(false))     -> Assert.False(@actual)
Assert.That(@actual, Is.EqualTo(@expected)) -> Assert.Equal(@expected, @actual)
Assert.That(@code, Throws.TypeOf<@type>())  -> Assert.Throws<@type>(@code)

Conclusion

The solution described here is not a working tool ready to be picked up and used by anyone. It’s a proof of concept that happens to work and it has value. At least for me. I was able to convert hundreds of tests across dozens of files, saving myself a couple of hours of tedious manual labor. The time I put into this tool I didn’t get back. Pure ROI is negative here. But it’s not only about time. I got plenty of satisfaction from learning something new, from automating an annoying task, from blogging about it and sharing it with the world.

The code could be found here.

Future work

Not every AST node type is covered by the pattern matching code and not every possible Assert expression is covered by the patterns. That is something to expand on. Maybe this could also be turned into a Visual Studio [Code] extension. In general this doesn’t have to be a unit test conversion tool. With this mini-DSL it’s possible to refactor/convert any code really. It’s also possible to search for the matches in the codebase. Lots of ideas and not so much time.

Also published on DEV and Medium

What would be great is to be able to specify structural patterns like this:

Assert.That(_, Is.EqualTo(_))
Assert.That(_, Is.EqualTo(true))
Assert.That(_, Is.Throws.TypeOf<_>())

And they would match the actual code:

// Matched by 'Assert.That(_, Is.EqualTo(_))'
Assert.That(account.Id, Is.EqualTo(id))
Assert.That("".ToBytes(), Is.EqualTo(new byte[] {}))

// Matched by 'Assert.That(_, Is.EqualTo(true))'
Assert.That(info.IsMd5, Is.EqualTo(true));
Assert.That(token.BoolAt(path, true), Is.EqualTo(true));

// Matched by 'Assert.That(_, Is.Throws.TypeOf<_>())'
Assert.That(() => Quad[-1], Throws.TypeOf<ArgumentOutOfRangeException>())
Assert.That(() => access(token, path), Throws.TypeOf<JTokenAccessException>())

At first it looks like a quite difficult task. But as it turns out in its simple form is not even that hard. I got the idea first when I was generating code for AST replacement with Roslyn Quoter. Looking at its source code I discovered a bunch of Parse* methods of the SyntaxFactory class.

So basically one function call will parse the snippet and return an AST for the given pattern:

var patternAst = SyntaxFactory.ParseExpression("Assert.That(_, Is.EqualTo(_))");

The one line above is equivalent to a wall of code like this:

var patternAst =
    InvocationExpression(
        MemberAccessExpression(
            SyntaxKind.SimpleMemberAccessExpression,
            IdentifierName("Assert"),
            IdentifierName("That")))
    .WithArgumentList(
        ArgumentList(
            SeparatedList<ArgumentSyntax>(
                new SyntaxNodeOrToken[]{
                    Argument(
                        IdentifierName("_")),
                    Token(SyntaxKind.CommaToken),
                    Argument(
                        InvocationExpression(
                            MemberAccessExpression(
                                SyntaxKind.SimpleMemberAccessExpression,
                                IdentifierName("Is"),
                                IdentifierName("EqualTo")))
                        .WithArgumentList(
                            ArgumentList(
                                SingletonSeparatedList<ArgumentSyntax>(
                                    Argument(
                                        IdentifierName("_"))))))})));

It feels like a total win already and we have not even done anything useful yet. But let’s find this pattern in a source AST. First, we need to parse the file we’re searching in:

var sourceAst = CSharpSyntaxTree.ParseText(File.ReadAllText(filename));

This gives us the list of all expression nodes in the AST:

var nodes = sourceAst.GetRoot().DescendantNodes().OfType<ExpressionSyntax>();

And now we find the nodes that match:

foreach (var e in nodes)
{
    if (Ast.Match(e, patternAst))
    {
        var line = e.GetLocation().GetLineSpan().StartLinePosition.Line;
        var code = e.NormalizeWhitespace();
        Console.WriteLine($"  {line}: {code}");
    }
}

Obviously the Ast.Match function is the tricky one. But not as tricky, really. We recursively traverse both ASTs in parallel and see if they match:

public bool Match(SyntaxNode code, SyntaxNode pattern)
{
    // A placeholder matches anything
    if (IsPlaceholder(pattern))
        return true;

    // Node types don't match. Clearly not a match.
    if (code.GetType() != pattern.GetType())
        return false;

    switch (code)
    {
    case ArgumentSyntax c:
        {
            var p = (ArgumentSyntax)pattern;
            return Match(c.Expression, p.Expression);
        }
    case ArgumentListSyntax c:
        {
            var p = (ArgumentListSyntax)pattern;
            return Match(c.OpenParenToken, p.OpenParenToken)
                && Match(c.Arguments, p.Arguments)
                && Match(c.CloseParenToken, p.CloseParenToken);
        }
    case IdentifierNameSyntax c:
        {
            var p = (IdentifierNameSyntax)pattern;
            return Match(c.Identifier, p.Identifier);
        }
    case InvocationExpressionSyntax c:
        {
            var p = (InvocationExpressionSyntax)pattern;
            return Match(c.Expression, p.Expression)
                && Match(c.ArgumentList, p.ArgumentList);
        }
    case LiteralExpressionSyntax c:
        {
            var p = (LiteralExpressionSyntax)pattern;
            return Match(c.Token, p.Token);
        }
    case MemberAccessExpressionSyntax c:
        {
            var p = (MemberAccessExpressionSyntax)pattern;
            return Match(c.Expression, p.Expression)
                && Match(c.Name, p.Name);
        }
    case GenericNameSyntax c:
        {
            var p = (GenericNameSyntax)pattern;
            return Match(c.Identifier, p.Identifier)
                && Match(c.TypeArgumentList, p.TypeArgumentList);
        }
    case TypeArgumentListSyntax c:
        {
            var p = (TypeArgumentListSyntax)pattern;
            return Match(c.LessThanToken, p.LessThanToken)
                && Match(c.Arguments, p.Arguments)
                && Match(c.GreaterThanToken, p.GreaterThanToken);
        }
    default:
        return false;
    }
}

So it’s basically a giant switch with every node type in it. By far not every type is covered here, just those that I needed to get my examples to work. I imagine to cover the most of C# syntax I’d have to tediously write a couple of thousand lines of repetitive code. I’m not going to do it all any time soon. Just the stuff I need to cover my use cases.

With a few more lines of code added this already becomes a useful tool for searching for code patterns in a codebase. Next time we see how we can implement the replace part. The goal was to refactor, not just to search, wasn’t it? I have some ideas on how it could be done. See you next time.

Conclusion

Thanks to Roslyn awesome API with just 172 lines of code we have a pretty advanced code grep. Surely, it’s just a toy and a proof of concept at the moment. It would take a serious effort to make it something more than that. But I’m happy with what is possible with so little effort. Amazing.

Also published on DEV and Medium

The test in NUnit

Assert.That(actual, Is.EqualTo(expected));

becomes the test in xUnit:

Assert.Equal(expected, actual);

To convert each by hand requires a lot of patience and stamina. Since I don’t have either, after doing a few dozens manually I decided to automate the whole thing. The first most obvious approach would be to use a regexp and convert one to the other like so:

/Assert\.That\((.*?), Is\.EqualTo\((.*)\)\)/ -> Assert.Equal($2, $1)

It does work for some simple cases, but throw in something a bit hairier and the whole thing goes sideways. A perfectly valid small test tears this regexp to shreds:

Assert.That("));", Is.EqualTo("));")); -> Assert.Equal(", "));");;"));

Not good.

The better way to do that would be to parse the source file into an AST (Abstract Syntax Tree) and perform source to source transformations on it. I’ve done a bit of this in the past with C++ using Clang/LLVM and with JavaScript using Acorn parser. For C# there’s Roslyn.

How difficult could this be? Let’s find out. There’s some amount of documentation out there and some samples. Also, it’s possible to generate a starter project with VS2017 that would do the file loading and minimal AST traversal. It’s a good start, we can build on it. Here’s a good starting point for source transformation, for example.

So here’s a simplest NUnit module:

using NUnit.Framework;

namespace Test
{
    [TestFixture]
    class DumpTests
    {
        [Test]
        public void One_plus_one_should_be_two()
        {
            Assert.That(1 + 1, Is.EqualTo(2));
        }
    }
}

When converted to xUnit, it becomes this:

using Xunit;

namespace Test
{
    class DumpTests
    {
        [Fact]
        public void One_plus_one_should_be_two()
        {
            Assert.Equal(2, 1 + 1);
        }
    }
}

Precisely the following needs to be done:

• change using directive
• remove TextFixture class attribute
• replace Test attribute with Fact
• change Assert.That to Assert.Equal and swap arguments

The syntax rewriter does all the work, we just need to fill in some logic:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    ...
}

Let’s start with the easiest, removing the TextFixture attribute:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitAttributeList(AttributeListSyntax node)
    {
        if (ShouldRemoveTestFixture(node))
            return null;

        return base.VisitAttributeList(node);
    }

    // Checks if the node is "[TestFixture]" and should be removed
    private bool ShouldRemoveTestFixture(AttributeListSyntax node)
    {
        return node.Attributes.Count == 1
            && node.Attributes[0].Name.ToString() == "TestFixture"
            && node.Parent is ClassDeclarationSyntax;
    }
}

The code, in this case, is quite simple. The VisitAttributeList function gets called for every attribute in the source file. We just check that the attribute list has only one attribute, that its name is TextFixture and the parent node is a class. If it’s all true, then we just return null from the visitor method to indicate that the node should be deleted from the syntax tree. Easy.

Next up is the Test attribute:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitAttributeList(AttributeListSyntax node)
    {
        var newNode = TryConvertTestAttribute(node);
        if (newNode != null)
            return newNode;

        return base.VisitAttributeList(node);
    }

    // Converts "[Test]" to "[Fact]"
    private SyntaxNode TryConvertTestAttribute(AttributeListSyntax node)
    {
        if (node.Attributes.Count != 1)
            return null;

        if (node.Attributes[0].Name.ToString() != "Test")
            return null;

        if (!(node.Parent is MethodDeclarationSyntax))
            return null;

        return
            AttributeList(
                AttributeList<AttributeSyntax>(
                    Attribute(
                        IdentifierName("Fact"))))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

What we do here is quite similar to the previous example, with one exception that we’re not deleting the node, but replacing it with something else. First, we check that it’s a single attribute named Test and it’s attached to a function. To replace it, we need to construct a new syntax node. In this case, it’s the same thing, just the name is different. To build the syntax node we use SyntaxFactory methods, like AttributeList, Attribute and so on. The small quirk is the NormalizeWhitespace and WithTriviaFrom bits. Those make sure the resulting code is formatted and has the whitespace copied from the original node. Otherwise, the output code would look out of place and would require reformatting.

The using directive change is also trivial. It’s very similar to the Fact attribute situation above:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitUsingDirective(UsingDirectiveSyntax node)
    {
        var newNode = TryConvertUsingNunit(node);
        if (newNode != null)
            return newNode;

        return base.VisitUsingDirective(node);
    }

    // Converts "using NUnit.Framework" to "using Xunit"
    private SyntaxNode TryConvertUsingNunit(UsingDirectiveSyntax node)
    {
        if (node.Name.ToString() != "NUnit.Framework")
            return null;

        return
            UsingDirective(IdentifierName("Xunit"))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

The Assert conversion is a much more complicated case. The problem that the expression we want to match is quite complex, even though it doesn’t look like that. There’s a member function access Assert.That and a function call Assert.That(...) and the argument list made up of two arguments, where the second one is a member function call as well: Assert.That(actual, Is.EqualTo(expected)). Using Roslyn Quoter tool it’s possible to generate the code that creates such an expression:

InvocationExpression(
    MemberAccessExpression(
        SyntaxKind.SimpleMemberAccessExpression,
        IdentifierName("Assert"),
        IdentifierName("That")))
.WithArgumentList(
    ArgumentList(
        SeparatedList<ArgumentSyntax>(
            new SyntaxNodeOrToken[]{
                Argument(
                    IdentifierName("actual")),
                Token(SyntaxKind.CommaToken),
                Argument(
                    InvocationExpression(
                        MemberAccessExpression(
                            SyntaxKind.SimpleMemberAccessExpression,
                            IdentifierName("Is"),
                            IdentifierName("EqualTo")))
                    .WithArgumentList(
                        ArgumentList(
                            SingletonSeparatedList<ArgumentSyntax>(
                                Argument(
                                    IdentifierName("expected"))))))})))

In the AST form this little snippet of code looks pretty huge. When we want to replace this pattern with a different piece of code, we need to find it first. And that means we need to check against the structure of every function call expression in the file and see if it’s similar:

public class NunitToXunitRewriter: CSharpSyntaxRewriter
{
    public override SyntaxNode VisitInvocationExpression(InvocationExpressionSyntax node)
    {
        var newNode = TryConvertAssertThatIsEqualTo(node);
        if (newNode != null)
            return newNode;

        return base.VisitInvocationExpression(node);
    }

    // Converts Assert.That(actual, Is.EqualTo(expected)) to Assert.Equal(expected, actual)
    private SyntaxNode TryConvertAssertThatIsEqualTo(InvocationExpressionSyntax node)
    {
        // Check it's Assert.That member
        if (!IsMethodCall(node, "Assert", "That"))
            return null;

        // It must have exactly two arguments
        var assertThatArgs = GetCallArguments(node);
        if (assertThatArgs.Length != 2)
            return null;

        // The second argument must be a `Is.EqualTo`
        var isEqualTo = assertThatArgs[1].Expression;
        if (!IsMethodCall(isEqualTo, "Is", "EqualTo"))
            return null;

        // With exactly one argument
        var isEqualToArgs = GetCallArguments(isEqualTo);
        if (isEqualToArgs.Length != 1)
            return null;

        // Grab the arguments
        var expected = isEqualToArgs[0];
        var actual = assertThatArgs[0];

        // Build a new AST with the actual and expected nodes inserted into it
        return
            InvocationExpression(
                MemberAccessExpression(
                    SyntaxKind.SimpleMemberAccessExpression,
                    IdentifierName("Assert"),
                    IdentifierName("Equal")))
            .WithArgumentList(
                ArgumentList(
                    SeparatedList<ArgumentSyntax>(
                        new SyntaxNodeOrToken[] {expected, Token(SyntaxKind.CommaToken), actual})))
            .NormalizeWhitespace()
            .WithTriviaFrom(node);
    }
}

To match the expression we have to drill down into the AST and compare node by node. It’s very tedious, but luckily after the code is written it will convert all the tests that have a similar structure. Write once, run many times. The two helper functions that are used in this matching code look like this:

private bool IsMethodCall(ExpressionSyntax node, string objekt, string method)
{
    var invocation = node as InvocationExpressionSyntax;
    if (invocation == null)
        return false;

    var memberAccess = invocation.Expression as MemberAccessExpressionSyntax;
    if (memberAccess == null)
        return false;

    if ((memberAccess.Expression as IdentifierNameSyntax)?.Identifier.ValueText != objekt)
        return false;

    if (memberAccess.Name.Identifier.ValueText != method)
        return false;

    return true;
}

private ArgumentSyntax[] GetCallArguments(ExpressionSyntax node)
{
    return ((InvocationExpressionSyntax)node).ArgumentList.Arguments.ToArray();
}

In case the expression is a match, we take the expected and actual arguments, or the AST nodes that represent them, to be exact and wrap them into a different AST that represents the xUnit equivalent: Assert.Equal(expected, actual).

Not that crazy difficult. But now we have a tool that can convert a majority of tests from NUnit to xUnit automagically. And it not only converts the Assert expressions but the whole file. Nice!

The sucky part is that the matching code is very specific to the expression we’re trying to convert. So if we have a few variations of the Assert it would take writing so much code for every case. It’s gonna very quickly get out of control. Imagine just a few very simple variations:

Assert.That(actual, Is.True());
Assert.That(actual, Is.EqualTo(true));
Assert.That(actual, Is.False());
Assert.That(actual, Is.EqualTo(false));

To cover most common NUnit cases we’d have to write hundreds of those matching functions with very repetitive code. That would be a LOT of work. Can we do better? Yes, we can! I have an idea and I’ll describe in the next post.

Conclusion

In only 175 lines of code we have a fully functional converter that does in a second what takes a lot of time to do by hand. Even though it’s just a proof of concept and doesn’t cover any significant amount of NUnit assertions, I was able to convert a few files with tests with almost no additional fixing.

Also published on DEV and Medium

After spending 30-40 minutes trying to figure out what I did wrong, I realized it wasn’t me. It was Microsoft. Apparently, I stumbled upon an ancient bug in Base64 decoding function. This bug must be present since the introduction of Convert.FromBase64String in .NET 1.1 in the year of 2003. Whoa! That’s old. And it’s not very difficult to reproduce. Here you go:

Convert.FromBase64String("abc==");

Technically this is an illegal Base64. The legal version is "abc=". Notice only one padding character =. The Base64 encoding represents every 6 bits of the binary input with one ASCII character. This means every 4 characters in the Base64 encoded string represent 3 bytes. When the encoded data is not a multiple of 3 bytes Base64 encoder adds padding characters to make the Base64 a multiple of 4 characters. This makes "abc=" a correctly padded Base64 string. Adding another = to it makes it invalid.

Base64 "abc=" decodes to two bytes [105, 183]. This is correct. Adding another padding character at the end shouldn’t really change the encoded value. It’s like adding a space at end of the sentence. Yes, it’s there, but it doesn’t change the meaning of the sentence. But .NET doesn’t think so. "abc==" decodes to one byte of [109]. Not only it got shorter, which is weird since we made the input longer. It also got different. The first byte changed from 105 to 109. And an exception didn’t get thrown either. Add another = and you’ll get an exception. Amazing!

Code:

using System;

public class Program
{
    public static void Main()
    {
        DecodeAndPrint("abc=");
        DecodeAndPrint("abc==");
    }

    static void DecodeAndPrint(string base64)
    {
        Console.WriteLine(
            "'{0}' -> [{1}]",
            base64,
            string.Join(", ", Convert.FromBase64String(base64)));
    }
}

Output:

'abc=' -> [105, 183]
'abc==' -> [109]

And what is really amazing, is that no one discovered this for so many years. Or it got discovered, but it didn’t get fixed. Base64 is quite fundamental in the information exchange over the wire. It is used all over the place. Yet, .NET got away with a buggy Base64 decoder for so many years.

At first I couldn’t believe it and started to investigate it. I googled for a while and didn’t really find much. Then I posted on StackOverflow, but didn’t get much luck there either. I had to even answer my own question once I figured out what’s going on. After searching on GitHub for a while I stumbled upon a fix in .NET Core made in July 2018. So the latest .NET Core version handles this correctly and throws an exception:

Unhandled Exception: System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64String(String s)
   at Program.DecodeAndPrint(String base64) in ./base64/Program.cs:line 13
   at Program.Main() in ./base64/Program.cs:line 8

It took them about 15 years to find this and fix it. What’s interesting is that no one really tried to fix it specifically. It happened while they rewrote the code to make it faster:

Convert.FromBase64() had a subtle bug where an illicit second padding character at the end of the string caused the decode to “succeed” by dropping the fifth to last character.

We inadvertently fixed this bug while optimizing that api in .NetCore 2.1. Adding test to document bug and ensure we don’t regress.

So this is fixed in .NET Core 2.2. But it’s still broken in the current latest version of .NET Framework 4.7.2. And it looks like it’s broken in Mono too.

A workaround for .NET 4.7.2 would be to repad incorrectly padded strings with something like this:

// This only works for base64 without spaces or linebreaks.
string Repad(string base64)
{
    var l = base64.Length;
    return l % 4 == 1 && base64[l - 1] == '='
        ? base64.Substring(0, l - 1)
        : base64;
}

Also published on DEV and Medium

I do all my file operations in the terminal. I find this faster and more convenient than clicking in the GUI file manager of some sort. Actually I’m so clumsy with those that I usually end up moving something important somewhere where it doesn’t belong. Many, many years ago, when I was a young junior developer on my first job, I moved an important project on a network share to a sibling folder. All I wanted to do is to put a mouse on the other side of the keyboard and I grabbed it with my left hand. Oh, that was a major disaster: people were running around looking for the project folder, thought we are being hacked. I found it 10 minutes later.

Not to repeat those grave mistakes of the past, I simply stick to the command line these days. The problem with the command line though, is that it’s not very visual. Especially when it comes to remote systems. Local file completion in zsh is pretty good. And once paired with a fuzzy matcher like fzf, it becomes a pleasure to do almost any file operations in the terminal.

For a few years I relied on zsh remote path autocompletion for rsync. It works almost as good as it does for the local files. Just type some characters, press TAB and it autocompletes the filename for you. If the name is ambiguous a list of matches would pop up where you can select a name with the arrow keys. A pretty standard shell experience.

There’s a problem though. Since it’s a remote system, it takes a second or two to autocomplete. It’s annoying. Plus it only autocompletes in the current directory and I would have to press TAB many times before I get the full path. And I can only download one file or folder at a time. Here’s a demo. It’s painful to watch how slow it is:

Where’s problem, there must be a solution. And how it usually happens with the command line, the solution is a Bash script:

#!/bin/bash
set -eo pipefail

HOST=${DL_HOST:-dev.somehost.to}
DIR=${DL_DIR:-files}

if [[ "$1" == "-h" || "$1" == "--help" ]]; then
    echo "USAGE: dl [dir [host]]"
    exit
fi

if [[ "$1" != "" ]]; then
    DIR="$1"

    if [[ "$2" != "" ]]; then
        HOST="$2"
    fi
fi

REMOTE="$HOST:$DIR/"

rsync -a "$REMOTE" \
    | ruby -ne 'puts $_.split(/\s+/, 5).last' \
    | fzf -m --height 50% \
    | rsync -avP --no-relative --files-from - "$REMOTE" .

It works like this:

First it uses rsync to list all the files in the remote folder in the format similar to ls -l:

drwxr-xr-x          4,096 2019/03/01 23:23:25 src
-rw-r--r--            261 2019/02/20 23:11:49 src/PasswordManagerAccess.csproj
drwxr-xr-x          4,096 2019/03/01 23:23:25 src/Common
-rw-r--r--            379 2019/02/20 23:11:49 src/Common/BaseException.cs
-rw-r--r--            906 2019/03/01 23:23:25 src/Common/ClientException.cs
-rw-r--r--          3,454 2019/03/01 23:23:25 src/Common/Crypto.cs
-rw-r--r--            241 2019/02/20 23:11:49 src/Common/ExposeInternals.cs
-rw-r--r--          6,868 2019/03/01 23:23:25 src/Common/Extensions.cs
-rw-r--r--          1,178 2019/02/20 23:11:49 src/Common/HttpClient.cs
-rw-r--r--            487 2019/02/20 23:11:49 src/Common/IHttpClient.cs
-rw-r--r--          7,267 2019/02/20 23:11:49 src/Common/JsonHttpClient.cs
...

That gets piped into an inline Ruby script that extracts the 5th column and all the spaces it might have. And the result of that goes into fzf for interactive selection. Once finished, that is now finally piped into another instance of rsync, which is receiving a list of files on the standard input (--files-from -). Pretty cool. The added benefit: now I can pick multiple files easily.

The script takes the remote folder name and the host optionally from the command line parameters or the environment variables. The most commonly used path I decided to hardcode to push back my RSI by a couple of years.

So I though, I could probably write a simple text config file parser in a few minutes. Why not give it a try. All I need is string keys and values. Comments would be good to have. Something like this:

# Login username
username: dude@lebowski.com
# User password
password: no one will guess
# URL
url: https://lebowski.com:443/index.html

This is a subset of YAML actually. Very clean and readable. How difficult would it be to write a parser for that. Normally, every time I say something like this to myself, I mentally prepare myself for a huge underestimation. What looks like a ten minute task, could turn out to be a week long project. Strangely, not this time. Thanks to pretty great runtime library and awesome LINQ support in 3 minutes I had a fully working solution:

Dictionary<string, string> ReadConfig(string filename)
{
    return File
        .ReadAllLines(filename)
        .Select(line => line.Trim())
        .Where(line => line.Length > 0 && !line.StartsWith("#"))
        .Select(line => line.Split(new[] {':'}, 2))
        .Where(parts => parts.Length == 2)
        .ToDictionary(parts => parts[0].Trim(), parts => parts[1].Trim());
}

This function is not crazy efficient, but who cares. It’s pretty robust, it wouldn’t fail with an error as long as it’s possible to read a file. It doesn’t have any error reporting in case there’s a syntax error, though. It would simply ignore it. In my case it’s good enough.

Let’s see how this works. First, I read the file. This call would return an array of strings, one per line:

File.ReadAllLines(filename)

Next, I trim all the whitespace on both ends. Select in LINQ is the same as map almost everywhere else, it transforms the sequence by applying a function to each element:

.Select(line => line.Trim())

Next, I filter out all lines that are blank or start with #. Where filters out the sequence by keeping the elements that satisfy the given predicate:

.Where(line => line.Length > 0 && !line.StartsWith("#"))

Next, I split each line on the first colon. If the rest of the line has more colons they will not be split and become part of the value. That’s intentional:

.Select(line => line.Split(new[] {':'}, 2))

Next, I filter out all the lines that didn’t get split into exactly two parts. This is the place where syntax errors would get ignored and thrown out:

.Where(parts => parts.Length == 2)

And in the last step I convert the array of two element arrays to a dictionary. What in C# is called a dictionary in other languages might be called object, map or hash map. It’s a key-value storage or an associative array. In this step I also trim any trailing whitespace on the key and leading whitespace on the value (other ends are trimmed already):

.ToDictionary(parts => parts[0].TrimEnd(), parts => parts[1].TrimStart());

Done. In a few lines and one statement I’ve read and parsed a config file.

JavaScript has petty similar functional programming capabilities, so it would be possible to mirror this solution in JS. Like always, there are some gotchas. In this case JS String.split function is acting weird. The limit parameter works differently compared to all the other languages I tried. Instead of returning the rest of the line in the last element, split in JavaScript truncates the input. WAT?! To fix that I have to join the split tail back together in the line before the final reduce that converts the array to object.

function readConfig(filename) {
    return require("fs")
        .readFileSync(filename, "utf-8")
        .split("\n")
        .map(x => x.trim())
        .filter(x => x.length > 0 && !x.startsWith("#"))
        .map(x => x.split(":"))
        .filter(x => x.length > 1)
        .map(x => [x[0], x.slice(1).join(":")])
        .reduce((a, x) => (a[x[0].trimEnd()] = x[1].trimStart(), a), {})
}

JavaScript has native support for JSON, so it’s probably stupid to roll your own config format, when JSON could be read in one short statement. The comments are not supported though.

I think the Ruby version is the cleanest, though it’s practically the same:

def read_config filename
    File
        .readlines("config.yaml")
        .map(&:strip)
        .reject { |x| x.empty? || x.start_with?("#") }
        .map { |x| x.split ":", 2 }
        .select { |x| x.size == 2 }
        .map { |k, v| [k.rstrip, v.lstrip] }
        .to_h
end

Ruby supports both YAML and JSON out of the box. It would be easier to just do

YAML.load_file "config.yaml"

but then I’d have to quote some of the values as YAML is not that flexible with the whitespace and special characters.

How would I do it Go? I wouldn’t! I don’t want to drown in ifs, fors, errs and nils. Just say no to writing code and go get some packages.

The naive solution anyone can write as soon as they grasp the very basic concepts of the language and understand the problem. The elegant solution requires a better knowledge of the standard library. The code becomes clearer and more succinct. But that sometimes comes with a performance penalty that is not very obvious at the first glance. In this post I wanted to go over some of those less obvious points.

Reverse a string

Take the first problem: reverse a string. To make it simpler, let’s first assume we have only basic ASCII or single code point UTF-16 characters in our string we want to reverse. I’ll touch on what to do if it’s not the case later.

function reverseString(str) {
    let reversedString = '';

    for (let i = str.length - 1; i >= 0; i--) {
        reversedString += str.charAt(i);
    }

    return reversedString;
}

Here we append the characters from the back of the original string to the begging of an empty string one by one. This way we get the reversed string. A non obvious thing that happens here is the string reallocation. Strings in JavaScript are immutable which means every time we modify a string a copy is created. I’m sure there are some clever optimizations to not to reallocate too much and not to create too many useless copies. But the fact is, some array has to grow and some reallocations and copying must be happening. Every time a reallocation happens some new memory gets reserved and all the bytes of the old string get copied to the new place. This takes extra time which is invisible if you look at the code. It would be best to preallocate the memory, since we know how many characters we are going to need ahead of time. As far as I know there’s no way to do that in JavaScript.

function reverseString(str) {
    return str.split('').reverse().join('');
}

This solution is for sure much clearer and more elegant. It’s much easier to reason about the code. It’s instantly obvious what the function does.

The hidden danger here is the additional memory the VM has to allocate to keep the temporaries. This code is roughly equivalent to the following:

function reverseString(str) {
    let chars = str.split('');
    let reversed = chars.reverse();
    let result = reversed.join('');
    return result;
}

When it’s written like this it becomes clear that during the execution we’d need to keep at least two copies of the string in memory. First chars. It has to coexist along with the original str. Next reversed. For a short time it has to coexist along with chars. And then result has to coexist along with reversed. So worst case 4x the memory. Imagine now the string is 1GB long. And of the garbage collector kicks in in between some of these calls, the total run time is gonna a lot longer than it looks.

Here’s some quick and dirty profiling (x-axis is string length, y-axis is milliseconds):

The elegant solution here is indeed optimized. I’m guessing that is due to all the work being done in the native functions that are dealing with the string as opposed to JavaScript code that runs on the virtual machine. And as you can see the first two solutions are not quite linear because of the intricacies of the underlying VM and CPU architecture. This would take a separate article (or a few) to explain that.

I briefly mentioned above the character encoding problem. Really it’s not safe to just reverse the utf-16 code points. There are some really complicated rules on how Unicode text is formed and some characters or graphemes could be up to 6 code points together. Look at this answer for some details. Long story short: you need to use a special library to deal with characters. Don’t reinvent the wheel.

The longest word

The second problem from the original post is about finding the length of the longest word in a string where words are separated by spaces. The original solution is to split the original string and then loop over the resulting array to find the longest string:

function findLongestWordLength(str) {
    let maxVal = 0;

    const wordArr = str.split(' ');

    for(let i = 0; i < wordArr.length; i++) {
        let word = wordArr[i];
        if (word.length > maxVal) {
            maxVal = word.length;
        }
    }
    return maxVal;
}

This version creates a temporary array to store all the words extracted from the original string. The words themselves also need to be stored somewhere. Since the strings are immutable it’s very likely the actual bytes are not really duplicated but rather referenced in the original array. But that’s hard to guess without looking at the VM source code. The fact is a bunch extra memory is needed.

The optimized version uses more functional approach:

function findLongestWordLength(str) {
    const arrOfWords = str.split(' ');
    const arrOfLengths = arrOfWords.map(item => item.length);

    return Math.max(...arrOfLengths);
}

In this version there’s even more memory used. arrOfLengths also has to be kept around. And in a situation when we have 1GB of input with only 1 letter words (I know, it’s kinda extreme), we would have 3GB total roughly wasted on this.

Math.max thing is kinda broken actually. The spread operator ... is substituting the array elements for the function arguments. Which makes a function call with a boatload of parameters. Which in turn causes stack overflow exception with already 200k elements. Not a very big number. In node REPL:

> Math.max(...Array(200000))
Thrown:
RangeError: Maximum call stack size exceeded

For the benchmarking I fixed this version like this:

function findLongestWordLengthFixed(str) {
    const arrOfWords = str.split(' ');
    const arrOfLengths = arrOfWords.map(item => item.length);

    let maxLength = 0;
    for (let i = 0; i < arrOfLengths.length; ++i) {
        if (arrOfLengths[i] > maxLength) {
            maxLength = arrOfLengths[i];
        }
    }

    return maxLength;
}

The really optimized solution would be not to create any temporary arrays, but iterate over the original string in-place and find the longest word:

function findLongestWordLengthFast(str) {
    let maxLength = 0;
    let currentLength = 0;

    for (let i = 0, l = str.length; i < l; ++i) {
        if (str.charCodeAt(i) === 32) {
            if (currentLength > maxLength) {
                maxLength = currentLength;
            }
            currentLength = 0;
        } else {
            ++currentLength;
        }
    }

    // Account for the last word
    return currentLength > maxLength ? currentLength : maxLength;
}

This function does zero heap allocations and is very cache friendly. Here a quick profile (x-axis is string length, y-axis is milliseconds):

As you can see the brute force/naive versions are performing much better than the optimized/elegant one.

Conclusion, sort of

It’s always tempting to rewrite the code in a more elegant, cleaner and often shorter way. Sometimes it comes with a cost. It’s important to understand that cost. Sure, when you’re reversing a string of 3 characters any solution would do and none of this matters. But often enough we don’t know ourselves in the beginning how big the date will be when the system hits production. Big-O analysis is important, but it’s not everything. Those pesky constants in O(n) are important as well. Very important for real systems. They could make a big difference, like double you AWS bill or make response time 5 times larger. Keep that in mind next time you reverse a string in production =)

I used this code to profile the functions above. I used node on macOS. The number may vary across hardware, browsers, OSes. Keep that in mind as well.

DEV has a similar problem. I’m easily distracted by the visual noise and I find it difficult to concentrate on reading when there I see something but text. I can ignore the sidebar, but ignoring the horizontal bar on the bottom is difficult, especially when it cuts a line of text in the middle.

So this time I decided to automate the process and make a userscript that removes the top, bottom and sidebars with a keyboard shortcut. Welcome DEV zen mode. Install it, press Shift-Z while in the article section and all the boxes go away. Press the same key again to bring them back.

To install the script you’d need a userscript manager extension installed in your browser. That would be Tampermonkey for Chrome or Greasemonkey for Firefox.

The script itself is available on openuser.js. Alternatively it’s possible to create a new script with Tampermonkey/Greasemonkey and paste the file form GitHub into it.

The core of the feature was not that difficult to put together. Adding a keyboard shortcut and hiding some elements is pretty trivial with vanilla JavaScript. I’m sure the code is not very robust yet and could benefit from some cleaning up. For one thing, I’m not saving the original display property, just assuming it’s blank. It works not, but might get broken when CSS changes.

There was one problem I ran into though. I wanted to be able to bring back the hidden elements when the user navigates away from the page. To make this happen I tried to find an event that is fired when the URL changes. To my surprise everything I found online didn’t work. I tried to add a listener for hashchange event, but couldn’t get any callbacks to trigger. After a while I gave up and used some hack I found on StackOverflow.

I would really love to see this becoming a feature of DEV. It doesn’t have to be exactly like this, but some kind of a reading mode would be really nice to have. I’m not a web developer and I don’t think I have the chops to contribute a feature like this to the codebase. Anyone interested? =)

I spent a bunch of time going through the giant documentation page, which seems to be both detailed and vague at the same time. It feels too formal and yet doesn’t give you a good idea what the whole thing is about. There’s a lot of terminology that is unique to this tool and not a lot of introduction into the lingo. There are very few examples of anything. No quick start section with trivial cases covered. Quite difficult to figure out where to start. I also didn’t find a lot of resources online. Hardly any, actually. Not many people seem to use this tool.

I’m sure reposurgeon is a beast. It seems to have more commands and switches than git, openssl and mogrify combined. It has a query language that makes Oracle green with envy. But for the life of me, I couldn’t figure out how to use any of it. It’s just not intuitive at all. Some commands are pretty arbitrary, like append, that lets you append text to the commit message. Yet, there’s no prepend.

The query language is quite obscure and doesn’t resemble anything I’m familiar with. Here’s an example from the docs:

define lastchange {
    @max(=B & [/ChangeLog/] & /{0}/B)? list
}

And the description:

List the last commit that refers to a ChangeLog file containing a specified string. (The trick here is that ? extends the singleton set consisting of the last eligible ChangeLog blob to its set of referring commits, and listonly notices the commits.)

That makes zero sense to me, though I’ve spent quite a bit of time trying to understand the docs.

To me this looks like a well run project. It seems to have good pace of releases, vast and detailed documentation, clean and documented source code (I wouldn’t put everything in one 21k line file though).

What I think went wrong is that it’s got too many very specific features and it tries to handle everything. Probably the author(s) have been spending too much time in their silo perfecting and adding features to their software.

I think they should have been spending more time on Stack Overflow answering questions about reposurgeon, writing introduction to the tool and blogging about it. I understand this tool is pretty niche and is designed for some very special use cases, but that should not mean “experts only”. A casual user should not be intimidated by the advanced features right away. It should be possble to do a trivial conversion with some minimal command set.

Look at Git. I know it’s a train wreck of a user experience. It’s got commands for everything. Mastering Git could take a lifetime. But if you stick to pull-add-commit-push workflow, it’s not that bad. Powerful, shouldn’t mean unapproachable.

$ git add .
$ git commit -m 'All together now!'

Done!

No, not really. This would eliminate the history. And I really wanted to keep it. I often go back to see what changes have been made to a file, I use blame to see when and why certain code was modified. I looked around to see how to get it done quickly. I found a whole bunch blog posts describing similar undertakings. I also found code snippets, shell and Python scripts and even Java programs to do just that.

After trying some of those (not the Java though, no thank you) I realized they don’t do exactly what I want. Some of the authors tried to keep the original commit hashes. Some authors wanted to have the original file paths. I wanted to be able to track changes from the beginning of the history.

Most of the approaches I found were not compatible with my goals. Usually people try to import a repo into a branch (usually by adding a remote from another repo), move all the files into a subfolder in one commit and then merge that branch into master. This creates one big commit where all the files get moved to a subfolder. And then another giant merge commit, where all the changes from one branch get copied to another branch. When you view such a repo on GitHub, you’d see that file history gets broken (blame still works though).

I also discovered a built-in command git subtree and it turns out it suffers from the same problems as all the third party tools I tried before that. So no go! Need to reinvent the wheel here and come up with my own solution.

So, basically, I needed to find a way to merge all the repos without creating any merge commits. And I needed to move the original files into subfolders. Two Git tools come to mind: cherry-pick and filter-branch.

A sidenote. I used to use Mercurial at work a few years back and it was great! The user experience on Mercurial is amazing. I kinda wish it didn’t die a slow death and let the inferior product to take over the dev scene. As Mercurial was intuitive and user friendly as Git is powerful and versatile. Git is like a really twisted Lego set: you can build whatever you want out of it.

So here’s the plan:

• put each repo in its own branch
• rewrite the history to move all the files in each commit into a subfolder
• rewrite the history to prepend the repo name to the commit message
• cherry pick all the commits from all the branches in chronological order into master
• delete branches
• garbage collect to shrink the repo

Easy peasy. Feel kinda masochistic today, so let’s do in Bash.

First, like always we need to make sure the whole script fails when any command fails. This usually saves a lot of time when something goes wrong. And it usually does.

#!/bin/bash
set -euo pipefail

List of repos I’d like to join:

repos="1password bitwarden dashlane lastpass opvault passwordbox roboform stickypassword truekey zoho-vault"

Make sure we start form scratch:

rm -rf joined
git init joined
cd joined

Now, here’s a tough one:

for repo in $repos; do
    git remote add $repo $REPO_DIR/repo
    git fetch $repo
    git checkout -b $repo $repo/master
    echo -n "$repo: " > prefix
    git filter-branch \
        -f \
        --tree-filter "mkdir -p .original/$repo && rsync -a --remove-source-files ./ .original/$repo/" \
        --msg-filter "cat $(pwd)/prefix -" \
        --env-filter 'GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"' \
        $repo
    rm prefix
done

Let’s go over it piece by piece. First, I import a repo into its own branch. The repo named lastpass end up in a branch named lastpass. Nothing difficult so far.

git remote add $repo $REPO_DIR/repo
git fetch $repo
git checkout -b $repo $repo/master

In the next step I rewrite the history for each repo to move files into a subfolder for each commit. For example, all the files coming from the repo lastpass would end up in the .original/lastpass/ folder. And it would be changed for all the commits in the history, like all the development was done inside this folder and not at the root.

git filter-branch \
    -f \
    --tree-filter "mkdir -p .original/$repo && rsync -a --remove-source-files ./ .original/$repo/" \
    --msg-filter "cat $(pwd)/prefix -" \
    --env-filter 'GIT_COMMITTER_DATE="$GIT_AUTHOR_DATE"' \
    $repo

The filter-branch command is a multifunctional beast. It’s possible to change the repo beyond any recognition with all the possible switches it provides. It’s possible to FUBAR it too. Actually it’s super easy. That’s why Git creates a backup under refs/original/refs/heads branch. To force the the backup to be overwritten if it’s already there I use the -f switch.

When the --tree-filter switch is used, every commit is checked out to a temporary directory and using regular file operations I can rewrite the commit. So for every commit, I create a directory .original/$repo and move all the file into it using rsync.

The --mag-filter switch allows me to rewrite the commit message. I’d like to add the repo name to the message, so that all the commits that are coming from the lastpass repo would look like lastpass: original commit message. For each commit the script would receive the commit message on stdin and whatever comes out to stdout would become the new commit message. In this case I use cat to join prefix and stdin(-). For some reason I couldn’t figure out why simple echo -n wouldn’t work, so I had to save the message prefix into a file.

And the last bit with --env-filter is needed to reset the commit date to the original date (author date in Git terminology). If I didn’t do it, Git would change the timestamp to the current time. I didn’t want that.

Next step would be to copy all those commits to the master branch to flatten the history. There’s no master branch yet. Let’s make one. For some reason Git creates a branch with all the files added to the index. Kill them with git rm.

git checkout --orphan master
git rm -rf .

To copy the commits, I need to list them first. That is done with the log command:

git log --pretty='%H' --author-date-order --reverse $repos

This command produces a list of all the commit hashes sorted from the oldest to the newest across all the branches I created earlier. The output of this step looks like this:

7d62b1272b4aa37f07eb91bbf46a33609d80155f
a8673683cb13a2040299dcb9c98a6f1fcb110dbd
f3876d3a4900e7f6012efeb0cc06db241b0540d6
7209ecf519475e59494504ca2a75e36ad9ea6ebe

Now that I have the list, I iterate and cherry-pick each commit into master:

for i in $(git log --pretty='%H' --author-date-order --reverse $repos); do
    GIT_COMMITTER_DATE=$(git log -1 --pretty='%at' $i) \
        git cherry-pick $i
done

The GIT_COMMITTER_DATE environment variable is again used to reset the commit date to the original creation time, which I get with the log command again like this:

git log -1 --pretty='%at' <COMMIT-HASH>

After these steps I have a repo with flat history, where each original repo lives in its own subdirectory under .original/. I can use GitHub file history and blame to see all the changes that happened to the original files since their birth. And since Git tracks renames I could just move these files to their new home inside the megarepo and I would still get the history and blame working.

The only thing left to do is to clean up the repo, delete all the branches I don’t need anymore and run the garbage collector to take out the trash.

for repo in $repos; do
    git branch -D $repo
    git remote remove $repo
    git update-ref -d refs/original/refs/heads/$repo
done

git gc --aggressive

The resulting repo lives here. I feel like it was worth the time I put into that. It gave me an opportunity to learn more about the Git low level commands. And now I have a repo that I can browse with ease and don’t need to jump between the branches every time I want to check some file history.

The script I used could be found here.

First of all, I’m trying to recreate a piece of software that I’m mostly happy with. I’d say 99%. I realized that while using it more and more. I should have rather made some pull requests to the original project instead of trying to rewrite it. But then I wouldn’t learn Go, would I?

Second, Go doesn’t seem to be a good fit for this kind of project. Text processing, no concurrency, no network access. Writing some sort of server or network crawler in Go would be a better fit. I’ll try something like this next.

So far I could say things mainly went well. It was not difficult to get productive in a matter of a couple of days. Writing Go feels like writing C most of the time without some of the C headaches. I keep typing the types first, though, and then wonder why it doesn’t compile. C habits die slow.

The thing that tripped me up all the time is the error handling. Most often peope complain about the necessity to type if err != nil ... after every function call. I’d say it doesn’t even bother me that much. It makes error handling explicit with a clear control flow path, versus exception-like implicit secondary control flow.

I think the most difficult part for me is that there’s no idiomatic way of dealing with errors. The error itself is too generic:

type error interface {
    Error() string
}

That’s it. Just something that returns a string. So anything could be an error. And most of the time it’s just a string. For example, os.IsNotExist. It checks if the error returned from any file.* function means that file wasn’t found:

func IsNotExist(err error) bool {
    return isNotExist(err)
}

In most systems I know of the errors are encoded as enumerations, types or a combination of the two. Something the compiler can actually work with. In Go it’s often enough just a string created with fmt.Errorf. Handling an error like this is difficult and feels kinda dirty, because the error message is the error value and its encoding. This is how it’s handled in the standard os package:

func isNotExist(err error) bool {
    return checkErrMessageContent(err, "does not exist", "not found",
        "has been removed", "no parent")
}

func checkErrMessageContent(err error, msgs ...string) bool {
    if err == nil {
        return false
    }
    err = underlyingError(err)
    for _, msg := range msgs {
        if contains(err.Error(), msg) {
            return true
        }
    }
    return false
}

So to see if the file didn’t open because it doesn’t exist, I’d have to check the error message for “does not exist” or “not found”. What if it’s “doesn’t exist” or if it’s something else that wasn’t found? Luckily there are convenience methods provided for this purpose: os.IsExist and os.IsNotExist. Nice!

This means to handle my own errors, I’d have to create similar methods if I don’t want to compare strings all over my program. That’s kinda tedious.

Another approach is to create my own error types. Which is also done in many places in the standard lib. At least is then possible to have a type assertion to see what type of error I got:

func underlyingError(err error) error {
    switch err := err.(type) {
    case *PathError:
        return err.Err
    case *LinkError:
        return err.Err
    case *SyscallError:
        return err.Err
    }
    return err
}

Often enough when the error is handled it’s wrapped into another error by simply gluing the error messages together. It’s a normal thing to get this from a Go program:

Error: Error building site: failed to render pages: render of "page" failed: execute of template failed: template: _internal/opengraph.html:31:19: executing "_internal/opengraph.html" at <.>: range can't iterate over Making a time tracking tool in Go

Notice how many colons (:) are in that error message. Practically each : is a callstack entry, because the typical pattern is to handle an error and wrap it again:

_, err = blah()
if err != nil {
    return fmt.Errorf("blah failed: %v", err)
}

There’s a library pkg/errors to handle this in a bit better way:

_, err = blah()
if err != nil {
    return errors.Wrap(err, "blah failed")
}

It’s a step forward. A very small one, but a step forward. The errors are still strings, but they have some idea of structure. Handling such errors is still painful. So far I opted not to handle them properly, just pass along to the poor user to deal with.

To summarize: I have not yet gotten a good feeling on how to return and handle errors in Go. I’ll experiment more and see what works for me better. I wish there was something like std::result in Rust or Try in Scala.

• Total time spent: about 20 hours

So far I’ve been just editing offline and pasting into the browser every 5 minutes to see what it looks like. When I’m done, I commit and then paste the final version and publish. When I find a typo, I fix, commit and publish again. I wanted to have a slightly different workflow.

I used to have a Jekyll blog and it was nice overall, but I had some gripes with it. I wanted to try something else. I was sure many new exciting tools came out while I wasn’t looking. And after I looked… I was shocked and overwhelmed with choice. Hugo, Hexo, Gatsby, Next and more. There’s even a dedicated website to track their GitHub stars, forks and other stuff. Wow!

And this is not it. I’d have to find a theme as well. Each of those projects has a theme shop or two. After browsing some of them and reading a bunch about what is better, easier, more powerful or what have you, I decided to go with Hugo and some theme I picked.

What was cool about Hugo, that it’s a one stop shop. It processes everything, from Markdown down to the compiled minified resources for my site. No extra plugins needed. Installing it was super easy. Setting up the blog was even easier:

$ hugo new site blog

Done!

Adding a theme was not much more difficult:

$ hub clone Track3/hermit themes/hermit

Then I deployed that on Netlify. And it was easy too. Just tell Netlify to use the GitHub repo and publish it with Hugo. Now when I push to GitHub, Netlify picks it up, builds it and it’s live in under a minute. Amazing!

Can it be all sunshine and rainbows? Sure not. The shit started happening after that. First, I tried to fix myself some CSS. I’m not sure how we as humanity arrived to this technology. I guess, one wrong wrong turn here, another wrong turn there and now we’re stuck with this mess. Have you ever tried to center a div?

Luckily I didn’t need to do that. But I wanted to fix some minor annoyances. I spent way too many hours on that, along with creating pull requests to the original theme and trying to fix the git repo on multiple occasions. Pro tip: use a new branch for each pull request.

Another major breakdown happened with Netlify not picking up my changes in the theme git submodule. I spent again way too many hours tracking this down. I even emailed the support, and they were very responsive and helpful, though they didn’t really help with the actual problem. It wasn’t obvious. After screwing around for few more hours I arrived at a solution: put generated CSS into the repo and commit after each change.

Apparently, Netlify doesn’t support Hugo pipelines at the moment. Which means the generated folder resources must be included in the repo as a workaround. Since the theme I picked had that in the repo, Hugo was just copying it to the public folder and was done with it, skipping all the SCSS processing on the server. Silly me, trying to fix that CSS.

If some poor soul is reading this and has the same problem: commit your top level resources folder into the repo.

Criss from Netlify wrote to me that I can join the beta and have my SCSS build on the server. I’m gonna give it a try a bit later. Now I want to enjoy stuff working for a little bit.

The preview of my new blog is here. It’s still got many problems and doesn’t have a proper home yet. I also would like to merge it with my dormant photo blog before I make it public. There’s always twice as much work still to do, no matter how much is done already.

Id  Day                Start      End        Duration   Notes
40  Sun Jan 20, 2019   22:00:00 - 01:00:00   3:00:00    initial setup
                                             3:00:00
41  Mon Jan 21, 2019   22:37:23 - 00:37:26   2:00:03    cli and cobra
                                             2:00:03
43  Tue Jan 22, 2019   11:40:14 - 12:01:56   0:21:42    tags
44                     12:02:08 - 12:34:00   0:31:52    parse tags from the comment
                                             0:53:34
48  Wed Jan 23, 2019   00:57:40 - 01:34:35   0:36:55    out command
49                     01:34:53 - 02:06:28   0:31:35    fix tag parsing
                                             1:08:30
59  Sat Jan 26, 2019   21:52:17 - 22:42:48   0:50:31    sqlite
60                     23:37:25 - 00:11:05   0:33:40    sqlite
                                             1:24:11
61  Sun Jan 27, 2019   15:12:51 - 15:29:04   0:16:13    sqlite
                                             0:16:13
62  Mon Jan 28, 2019   00:13:13 - 00:47:54   0:34:41    sqlite
63                     22:30:26 - 22:53:47   0:23:21    backend
64                     23:24:25 - 00:37:21   1:12:56    backend
                                             2:10:58
66  Tue Jan 29, 2019   11:19:19 - 11:20:36   0:01:17    display
67                     11:24:10 - 11:42:07   0:17:57    display
68                     11:51:51 - 11:52:53   0:01:02    display
69                     12:57:04 - 13:01:14   0:04:10    display
70                     13:10:37 - 14:03:54   0:53:17    display
71                     15:00:21 - 15:31:03   0:30:42    display
72                     21:11:52 - 21:16:23   0:04:31    display
73                     21:16:29 - 23:18:54   2:02:25    display
                                             3:55:21
74  Wed Jan 30, 2019   18:38:29 - 18:43:57   0:05:28    ids
75                     20:27:47 - 21:42:52   1:15:05    ids
                                             1:20:33
    -------------------------------------------------------
    Total                                   16:09:23

This, by the way, is how much time I’ve dedicated so far to this project. And all those tiny little bits of time I get to work on this between dealing with small kids, changing diapers, feeding them and putting out other fires at home. Talking about programmer productivity and interruptions.

This feature turned from a quick one into a time sucking endeavor. First the Backend interface had to be changed to support retrieval of completed and open entries.

// Backend ...
type Backend interface {
    OpenEntry(comment string, startTime time.Time, tags []string) error
    CloseEntry(endTime time.Time) error

    GetEntries() ([]Entry, error)
    GetOpenEntry() (Entry, error)
}

Oh, the error handling! Why not just have error a part of every function in Go? I’ll talk about this in a separate post. So far I could say I’m happy with the simplistic nature of Go. The only thing I’m suffering from so far is the error handling. if err != nil { return err }.

My database format now spans three different files. entries.json to store the entries themselves. current.json to store the currently open entry, which is later moved to entries.json once it’s closed. And id.json which stores the last used index. I’m starting to regret the decision not to use SQLite, which would provide me with all that plumbing and free up some time to learn and google SQL queries. For now I’m also ignoring the problem of preventing two instances of the program trying to modify the database at the same time and turning it all into smoking rubble.

Briefly I tried to return a channel instead of an array of entries. I tried to be smart and find some use of Go channels in my program. Though it was a breeze to write, it was not really fitting into my usage patterns. So I decided to use simple arrays and worry about looking pro and hip later.

To display the entries I shopped around for a library again. And this, I think, is a really strong point of using Go. There’s a library for anything. Even though C++ has been around for a lot longer, it’s often impossible or really difficult to find libraries that do half of what Go libraries do. Writing a tool like this in C++ would be difficult. Anyhow, the library I found is called tablewriter. It’s pretty flexible and draws nice looking tables.

$ klk display --grep 'code|test'
+----+-------------+----------+----------+-----------------------------+-------------+
| ID |     DAY     |   TIME   | DURATION |           COMMENT           |    TAGS     |
+----+-------------+----------+----------+-----------------------------+-------------+
|  2 | 30 Jan 2019 | 21:34:17 |    15:14 | Writing tests               | #test       |
|  3 |             | 21:55:04 |    20:08 | Writing code                | #code       |
|  4 |             | 22:20:38 |    25:05 | Writing more code and tests | #code #test |
+----+-------------+----------+----------+-----------------------------+-------------+
|                     TOTAL   | 1:00:27  |
+----+-------------+----------+----------+-----------------------------+-------------+

I wasted a huge amount of time trying to make the report look nicer. In the process I figured out I’m not a good UI designer, even if it’s just a basic text UI. I think I’ll just copy what timetrap prints. For now this will do.

Now my application is almost 500 lines of code. That doesn’t sound much, but it could already do the most basic time tracking. I’d say 90% of what I use in timetrap. Next step would be to worry about U and D. And more sophisticated tag filtering, the whole reason I started this.

One thing that worries me a little is the executable size. I’m at 11 megabytes now and the app doesn’t do very much. When I tried SQLite, it was 16 MB. Sounds a bit excessive. I’ll investigate later, but I think the most of the bulk is coming from Cobra. I might consider getting rid of, since I don’t use any of the advanced features anyway.

Google searches that went into getting this to work:

• golang channel
• golang make empty channel
• golang close channel
• golang read from channel
• golang for range
• golang for range channel with index
• golang display table
• golang read line
• golang unmarshal string
• golang tostring
• golang create empty slice
• golang regexp flags
• golang int to string
• golang golang cannot define new methods on non-local type
• golang format duration
• golang init primitive type
• golang read text file as string
• golang sizeof
• golang sizeof int
• golang log fatal
• golang exit vs panic
• golang recover
• golang fallthrough case
• golang atexit

• Time spent: 5:15 hours
• Total time spent: 16:15 hours

I though it would be a good start to store the entries in a simple text file, where each line is a JSON object. I started using this format a long time ago as I find it a much better alternative to a giant JSON array.

Consider JSON-per-line:

{"id": 0, "comment": "writing tests", "tags":[]}
{"id": 1, "comment": "writing more tests", "tags":[]}

vs. regular JSON:

[
    {"id": 0, "comment": "writing tests", "tags":[]},
    {"id": 1, "comment": "writing more tests", "tags":[]}
]

Yes, it’s possible to load the regular JSON file with one line of code and it’s nice. But generating those files is painful. You’d have to worry about surrounding [] and trailing commas on every line but the last one. Where the JSON-per-line format is much simpler. Just append a line with a JSON object and you’re done. Usually parsing is faster too. Probably due to some non-linearities in JSON parsing or reallocation. Basically, it’s faster to parse a million small JSON objects than a single JSON array with a million entries.

It would be totally unprofessional of me to write any functions and call them directly. I’d need an interface first. After some googling and fighting with squiggly red and green lines in VS Code I was able to write my first interface in Go.

// Backend ...
type Backend interface {
    OpenEntry(comment string, startTime time.Time, tags []string) error
    CloseEntry(endTime time.Time) error
}

Notice the // Backend ... comment. It’s there for a reason. It makes the linter happy, as I don’t want to have green squigglies everywhere. Apparently the linter is not happy about the exported type without a comment. The type is exported if it starts with a capital letter, by the way. Some interesting design decision.

In Go anything that implements these two functions is considered a backend. No explicit interface implementation is needed. Neat and scary. So the simplest form would be a null backend:

// NullBackend ...
type NullBackend struct {
}

// OpenEntry ...
func (backend *NullBackend) OpenEntry(comment string, startTime time.Time, tags []string) error {
    return nil
}

// CloseEntry ...
func (backend *NullBackend) CloseEntry(endTime time.Time) error {
    return nil
}

The next step was to write the actual file backing storage. That wasn’t that difficult ether. I had to figure out how to serialize a struct into JSON. It’s pretty easy in Go:

encoded, err := json.Marshal(entry)

And then append it to a file:

f, err := os.OpenFile("entries.json", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
if err != nil {
    return err
}
defer f.Close()

f.Write(encoded)
f.WriteString("\n")

And our hand rolled NoSQL database is ready. Might even compete with MongoDB soon, just need to add a couple more features and a bunch of security holes. Ship it! 🚢

Google searches that went into getting this to work:

• go sqlite
• go check if file exists
• go implement interface
• go pointer receiver
• golang pointer receiver interface
• go interface
• golang ISO8601 date
• golang write file
• golang serialize to json
• golang marshal unexported fields
• golang writeline

• Time spent: 3:50 hours
• Total time spent: 11:00 hours

I started googling around and found out that all the files in a package could be essentially treated as one single file. There’s no such thing as a variable or a function local to a file. Can have many inits though. Anything starting with a capital letter is public and exported from the package (not file). Identifiers starting with a lowercase letter are private or hidden and are not exposed to the users of the package.

I normally refactor all the time. I move code around. I add, delete and re-add functions all the time. I rename everything constantly until I’m happy with the result. Go in general doesn’t prevent this style of developing, but it makes it a bit harder by not allowing to have any unused variables or imports. The imports are easily fixed the goimports or goreturns formatting tools. The local variables have to be removed by hand. This slows down the progress for me quite a bit. I’d rather have the compiler show me a warning instead. I’d get stuff to work first and then I’d clean up the warning. Go doesn’t forgive or forget.

As a result of this session I have all shared code extracted and tucked away in a separate file. I have two commands now: in and out. Things are cleaned up and are ready to be made into a great app.

Next time I start coding something more serious. Enough with the baby stuff, I’m a senior developer after all.

Google searches that went into getting this to work:

• go private function
• go function local to a file

• Time spent: 35 minutes
• Total time spent: 7:10 hours

$ klk in 'Writing some #tests to find a #bug'

My first impulse is to look for a library. And there is one for exactly this purpose. But who wants to introduce a left-pad timebomb into their own codebase? (Answer: almost anybody). I’d rather reinvent the wheel and solve the problem with a regex. Two problems are better than one.

Go has the regexp package in its standard library just for this purpose. So it’s really a two-liner:

func extractTags(text string) []string {
    re := regexp.MustCompile("#\\S+")
    // TODO: Strip out #
    return re.FindAllString(text, -1)
}

Notice a TODO? That one of those cases where you’d have to take a dive from Python heights into C depths and roll you own loop. The two-liner becomes a six-liner. This where I start missing Ruby, Python, C#, Scala, Kotlin, hell, even C++.

func extractTags(comment string) []string {
    re := regexp.MustCompile("#\\S+")
    tags := re.FindAllString(comment, -1)

    // Strip #s
    for i, tag := range tags {
        tags[i] = strings.TrimLeft(tag, "#")
    }

    return tags
}