A Couple of my Fantastical Devices

with the recent news about the Voynich Manuscript, as mentioned in my last post, I thought it opportune to share a couple of pieces of code I'd written. First off, as I mentioned earlier, a couple of years ago I wrote a Python implementation of Montemurro and Zanette's algorithm for calculating the entropy of words in documents. If you're interested in using the technique yourself, you may want to have a look. Secondly, my own attempts to uncover the syntax use a Python library for Hidden Markov Models that I created. It probably still has a few bugs in it, but it's attracted a bit of interest online, and I'm hoping to develop it further. So, if you're at all interested in AI, computational linguistics, or analytics, please have a look at these. Feedback is welcome, as is anybody who wishes to contribute further to these projects.

Quantum Thinking and Function Words

Two articles in New Scientist caught my eye this week - one about a quantum mechanics like system of logic and the other about function words. According to the first article, many of our thought processes don't follow the rules of classical logic, but a system of inference that can be described in terms of Hilbert Space, which is a vector space with an arbitrary number of dimensions. Quantum mechanics uses Hilbert spaces to describe the states of quantum systems, and the mathematics of Hilbert space allows quantum states to interact in counter-intuitive ways. The same logic apparently allows human minds to combine ideas in ways that don't necessarily follow the rules of classical logic, but do allow greater flexibility. To quote from the article -

If you want to research a topic such as the "story of rock" with geophysics and rock formation in mind, you don't want a search engine to give you millions of pages on rock music. One approach would be to include "-songs" in your search terms in order to remove any pages that mention "songs". This is called negation and is based on classical logic. While it would be an improvement, you would still find lots of pages about rock music that just don't happen to mention the word songs. Widdows has found that a negation based on quantum logic works much better. Interpreting "not" in the quantum sense means taking "songs" as an arrow in a multidimensional Hilbert space called semantic space, where words with the same meaning are grouped together. Negation means removing from the search pages that shares any component in common with this vector, which would include pages with words like music, guitar, Hendrix and so on. As a result, the search becomes much more specific to what the user wants.

Obviously, if you're interested in Artificial Intelligence, where a key aim is to enable computers to emulate the flexibility of human thought, this is a useful approach. The second article, by James W. Pennebacker, concerns his work on the importance of function words. These are things like pronouns, conjunctions and prepositions, the words that don't seem to mean very much, but act like glue holding the sentence together. Professor Pennebaker has discovered that there's a lot of psychological information hidden in these apparently insignificant words - for example, in a conversation between two people, the more socially dominant one will tend to use the word "I" less than the other one. Most natural language processing software treats words like this as stop words and ignores them, but for some applications (eg sentiment analysis, social network analytics) it could be just the data you need.

Delicious has a new owner

As you can see, I'm a user of the social bookmarking site Delicious, or to use its original name, del.icio.us. It was previously owned by Yahoo!, which announced last December that they didn't want it anymore. This led to panic amongst Delicious users, as a rumour went round that the site was going to close. Yahoo! quickly issued a statement that they weren't closing it, they were selling it, and now the new owners have been announced - AVOS, a company founded by the users of YouTube. Hopefully they might develop the site a bit more - Delicious has had the feel of a resource with untapped potential for a while.

And while we're at it, can we have the cool URL back please?