The entries in the table that used to be on this page were obtained via statistical analyses of the text corpora assembled by the crawler. Each entry is the cosine of the angle between the 3-gram profile vectors for the languages in the (large-dimensional) space of all possible 3-grams.
Now that I'm tracking over 1000 languages, displaying the data in an HTML table has become very slow (> 15MB HTML file) and has even been known to crash some people's browsers. So I've moved the raw data, along with some scripts for generating cool visualizations, to a github repository. Enjoy!