Back into the Clouds

So, it has been a while since I messed around with Hadoop. Just this last week, I had a co-worker over for a "pizza and beer night" that ended up being a family-oriented lasagna night that had just a bit of time for the guys to sneak away and take a look at the cloud.

Of course, as soon as I powered up my Linux box (running Ubuntu), the problems started. The machine had issues loading the U/I, and it took a while to dive into the cloud.

Once we did, though, we started to look at how the cloud (specifically map/reduce) could be used to do "analysis" on the data. I put analysis in quotes because too often people think that analysis comes for free in the cloud. In reality, it is quite the opposite... The concept behind cloud is great for data storage and access to data. It is also good for processing large data sets (looking for something specific in an entire data set).

So, I have decided to pick back up on the cloud (Hadoop for now), while I am out of class for my so-called Summer Break. I ordered "Hadoop: The Definitive Guide" (ISBN - 978-0-596-52197-4) to kick things off. In fact, as I just typed that out (the ISBN number), it triggered a thought about specific types of data (the ISBN number in this case). If you were to take numbers that appeared anywhere and try to look them up to check for valid ISBN numbers, it would take a significant amount of resources. But if we setup a standard on how to reference these numbers (ISBN:xyz), where xyz is the ISBN, it would then be a perfect task for map/reduce to identify any location that references a given book. In that same line of thought, what if we had similar identifiers for common types of information. With current databases and web-pages, it hasn't really been effective for people to "tag" information like this, but with map/reduce jobs, it might be more effective to address this problem systematically.

For example, I write a blog post, and talk about my friend John Smith. I know which John Smith this is because I am writing the article. Imagine if, as I was typing, the blog post was processed through an entity match/extraction service and identified that John Smith was ambiguous. I would then be prompted to disambiguate the entity John Smith by adding additional information. Perhaps an e-mail address. Once the system has a unique identifier, map/reduce could then be used to extend that entity with any additional information known about the entity.

In a traditional database, this would be a nightmare to keep track of, perform the searches, and run the updates. With map/reduce, though, this process begins to become much more straight forward. Let's say I take the sentence: "I met up with an old friend John Smith."

After the disambiguation process above and numerous passes through map/reduce, the sentence might look like: I {entity:{id:chrisvensko@gmail.com,e-mail:chrisvensko@gmail.com,e-mail:chris@texeltek.com,phone:555-1212,firstName:Chris,lastName:Vensko}} met up with John Smith {entity:{id:johnsmith@example.com,e-mail:johnsmith@example.com,phone:555-3210,country:USA,state:Iowa}}.

Although this new sentence is almost unreadable by a human, map/reduce can then uniquely identify this John Smith in any other blog post, e-mail transaction, etc. This would then make searching extremely easy... I could then search for (or index on) e-mail address, so that the johnsmith@example.com would come up, not only for e-mail correspondence with John Smith, but also any blog post, tagged photo, etc that referenced the same John Smith.

This definitely presents some issues with regards to security, privacy, and unique namespacing and ID generation.

Should all of the information about an entity be pushed into every document? Should map/reduce then break out the entities that describe the given entity? Should this information be contained in a separate data space as a look up value? So that chrisvensko@gmail.com provides all of the information in my "profile" phone numbers, addresses, contacts, schools, accounts, medical records, etc? Should that information get stored everywhere? How much cross-pollination should occur in the data? In the google-like approach with indexing every word, should we the break out every word? Every sentence? Where is that line?

Regardless of how this works out, my next goal is to start testing this with an extremely limited data set to see what all is possible. Hopefully my next post wont be so far off...

Until then, thanks for reading,
Chris