Pages

Thursday, March 29, 2012

Migrated WordGap from Python to Go 1

WordGap, an anagram search tool running on Google AppEngine, was originally written in Python. I recently rewrote it in Go 1 and added support for the SOWPODS word list. In terms of number of lines, the code size is roughly the same as the Python version, but the run-time performance has gained a considerable boost: the average latency of anagramming requests has dropped from ~500ms to ~30ms! Go has strongly exceeded my expectation and is now my language of choice for AppEngine app development.

Sunday, March 18, 2012

Initializing large arrays in Go for AppEngine

Go supports composite literals to construct arrays, slices, or maps. A list of strings, for example, may be created as follows:
words := []string{"AA", "AAH", "AAHED"}
This is convenient and works really well for small lists. If you try to initialize an array with hundreds of thousands of elements, though, Go compilation will significantly slow down even if the entire list can comfortably fit into the memory. Worse yet, you won't be able to deploy your app to AppEngine. A typical error message looks like this:
Error 422: --- begin server output ---
Compile failed:
2012/03/18 00:00:00 go-app-builder: Failed building app: failed running 6g: signal 9
--- end server output ---
Rolling back the update.
Error 422: --- begin server output ---
The solution is to construct large data structures at run-time by parsing data encoded in external files. For example, store a long list of words in a text file, one word per line; during initialization, convert them into a string slice:
import (
  "io/ioutil"
  "strings"
)

func loadStringList(filename string) []string {
  content, err := ioutil.ReadFile(filename)
  if err != nil {
    panic(err)
  }
  return strings.Split(string(content), "\n")
}

func init() {
  words := loadStringList("words.txt")
  ...
}
For more complicated data structures, you may want to use the gob package to do efficient deserialization.

Thursday, February 2, 2012

AppEngine Go SDK's 500-byte string limitation

AppEngine Go SDK has an undocumented limitation of datastore strings: they must not exceed 500 bytes. The dev_appserver will happily accept Go structs with strings longer than this limit, but when you try to deploy your app or list datastore entities from the development console, you will encounter an error like this:

BadValueError: Property Value is 501 bytes long; it must be 500 or less. Consider Text instead, which can store strings of any length.

To store long strings in the datastore, you'll need to manually convert a string type to a byte slice ([]byte), which will be stored as an unindexed blob. For example:

type T struct {
  LongString []byte
}

func store(longString string, r *http.Request) os.Error {
  ctx := appengine.NewContext(r)
  key := datastore.NewIncompleteKey(ctx, "T", nil)
  t := new(T)
  t.LongString = []byte(longString)
  _, e := datastore.Put(ctx, key, t)
  return e
}

Sunday, January 1, 2012

WordGap Chrome Web Store experiment

As a Chrome browser user, I don't see much value in Chrome Web Store, which is essentially a modern day Dmoz directory of web pages wrapped in a thin layer and marketed as "Apps". But my judgment might be clouded by my general distaste for the Apps/AppStore hype. Perhaps many Chrome users do enjoy browsing the flashy App directory, in which case, Chrome Web Store might be a great channel for web apps to get more exposure.

To test this hypothesis, I skimmed through the well-documented Developer's Guide, packaged WordGap as an app within two hours (mainly spent on struggling on artworks with my limited Photoshopping skills), and published it on Chrome Web Store under the "Games" and "Utilities" categories. No additional efforts were made at all to promote the app.

One month later, my Developer Dashboard reports that the WordGap app has garnered 32 weekly users and zero reviews.


Terribly pathetic numbers, if you ask me. Meanwhile, considering that Chrome Web Store doesn't offer an RSS feed or category for newly published apps, I'm curious to learn how these 32 weekly users discovered the WordGap app which is buried under heaps of blinding images. Did they navigate through pages after pages of apps until stumbling upon mine? Did they search for "anagram" or "scrabble" and somehow were dissatisfied by all the results ranked above my app? Did they find it under the "related apps" list? Or does Chrome Web Store randomize its directory ranking so that new apps get a chance to surface above the fold on a lucky day? Unfortunately, none of the questions can be answered by existing tools and APIs yet.