A Break in the Clouds?

So, after reading the first couple chapters of "Hadoop: The Definitive Guide" I have come to the realization that most of the discussion about cloud is about the architecture itself. I know I haven't gotten very far in the book, and the author is probably providing the necessary background information, but I think too much time is being spent on the framework.

I want to focus on thinking in terms of map and reduce. I still don't have any good examples yet, but my goal is to have that one map/reduce job that makes it click. Every time I feel like I am on to something, it seems like whatever article I am reading goes back to the framework. I keep asking myself if I am just missing something, or is this process really that complex, but I hope that it just takes a real-world example to get me started.

The problem I have run into with examples so far is that they have little value by themselves. Take word count; creates a count of words in an entire series of documents. Although this is handy, the first question out of a user's mouth will be... wow, word x was used n times. I wonder what documents those words were used in?

I really just need a sample map/reduce task that spells out a valuable map reduce. Perhaps I just need to shift my focus from trying to identify valuable information to simple, key/value pairs. What series of key/values would provide value? One thing that makes me feel like I am getting close is the concept of multiple-field keys. Take the word count example. Lets say I have 3 documents:

Doc 1:
Hello world!
Doc 2:
I love perl.
Doc 3:
Perl is the most used language in the world.

Ignoring how we parse out words (to separate out world! from world. from world), a possible set of key/value pairs for word counts could be:

Doc1//hello 1
Doc1//world 1
Doc2//I 1
Doc2//love 1
Doc2//perl 1
Doc3//perl 1
Doc3//is 1
Doc3//the 2
etc

I know, this isn't rocket science, but I think that this is where the key to using map/reduce will come from. With these results, I will now be able to identify how many times a given word is used within a single document. Perhaps I should look at it a little different... perhaps word, and then a list of documents it came from:
1 - Doc1, etc
hello [1]
perl [2,3]
love [1]
world [2]
etc

Maybe this works, but I still feel like there is something missing. It seems like it would be extremely difficult to break down every problem into this type of solution (especially the more complex problems).

Maybe a better way to look at map/reduce is to start with a problem, and then try and find the map/reduce (or series of map/reduce) that can be used to solve the problem.

Scenario:
Lets say that we have all of the GPS position reports for all of the UPS drivers for a given city. Our goal is to identify where the driver has lunch each day (and eventually identify a pattern around where he has lunch).

Here are my initial thoughts:
Initially, we would want to parse out the GPS updates by driver. For each driver, we would then want to identify any time the location remained the same for more than 20 minutes (with the assumption that the drivers are required to take a 30 minute lunch).

So, here are my thoughts on the outputs:
Driver/DTG/ Location
Driver/Date/Location Dwell Time

If we sort these results by dwell time, this should give us locations where the driver stopped for more than 20 minutes.

Then, we want to put a list together of all of the locations where the driver has stopped for more than 20 minutes (to idenitfy all the lunch locations):
Driver/Date/ Lunch Location

Does this make sense like this? I guess I just need to work on breaking the problem into a series of key/value pairs, but I am not sure if I am going the right direction or not.

Well, it is getting late, and I don't think I can handle any more. I will try and read through this again and see if I can come up with a better idea.

Back into the Clouds

So, it has been a while since I messed around with Hadoop. Just this last week, I had a co-worker over for a "pizza and beer night" that ended up being a family-oriented lasagna night that had just a bit of time for the guys to sneak away and take a look at the cloud.

Of course, as soon as I powered up my Linux box (running Ubuntu), the problems started. The machine had issues loading the U/I, and it took a while to dive into the cloud.

Once we did, though, we started to look at how the cloud (specifically map/reduce) could be used to do "analysis" on the data. I put analysis in quotes because too often people think that analysis comes for free in the cloud. In reality, it is quite the opposite... The concept behind cloud is great for data storage and access to data. It is also good for processing large data sets (looking for something specific in an entire data set).

So, I have decided to pick back up on the cloud (Hadoop for now), while I am out of class for my so-called Summer Break. I ordered "Hadoop: The Definitive Guide" (ISBN - 978-0-596-52197-4) to kick things off. In fact, as I just typed that out (the ISBN number), it triggered a thought about specific types of data (the ISBN number in this case). If you were to take numbers that appeared anywhere and try to look them up to check for valid ISBN numbers, it would take a significant amount of resources. But if we setup a standard on how to reference these numbers (ISBN:xyz), where xyz is the ISBN, it would then be a perfect task for map/reduce to identify any location that references a given book. In that same line of thought, what if we had similar identifiers for common types of information. With current databases and web-pages, it hasn't really been effective for people to "tag" information like this, but with map/reduce jobs, it might be more effective to address this problem systematically.

For example, I write a blog post, and talk about my friend John Smith. I know which John Smith this is because I am writing the article. Imagine if, as I was typing, the blog post was processed through an entity match/extraction service and identified that John Smith was ambiguous. I would then be prompted to disambiguate the entity John Smith by adding additional information. Perhaps an e-mail address. Once the system has a unique identifier, map/reduce could then be used to extend that entity with any additional information known about the entity.

In a traditional database, this would be a nightmare to keep track of, perform the searches, and run the updates. With map/reduce, though, this process begins to become much more straight forward. Let's say I take the sentence: "I met up with an old friend John Smith."

After the disambiguation process above and numerous passes through map/reduce, the sentence might look like: I {entity:{id:chrisvensko@gmail.com,e-mail:chrisvensko@gmail.com,e-mail:chris@texeltek.com,phone:555-1212,firstName:Chris,lastName:Vensko}} met up with John Smith {entity:{id:johnsmith@example.com,e-mail:johnsmith@example.com,phone:555-3210,country:USA,state:Iowa}}.

Although this new sentence is almost unreadable by a human, map/reduce can then uniquely identify this John Smith in any other blog post, e-mail transaction, etc. This would then make searching extremely easy... I could then search for (or index on) e-mail address, so that the johnsmith@example.com would come up, not only for e-mail correspondence with John Smith, but also any blog post, tagged photo, etc that referenced the same John Smith.

This definitely presents some issues with regards to security, privacy, and unique namespacing and ID generation.

Should all of the information about an entity be pushed into every document? Should map/reduce then break out the entities that describe the given entity? Should this information be contained in a separate data space as a look up value? So that chrisvensko@gmail.com provides all of the information in my "profile" phone numbers, addresses, contacts, schools, accounts, medical records, etc? Should that information get stored everywhere? How much cross-pollination should occur in the data? In the google-like approach with indexing every word, should we the break out every word? Every sentence? Where is that line?

Regardless of how this works out, my next goal is to start testing this with an extremely limited data set to see what all is possible. Hopefully my next post wont be so far off...

Until then, thanks for reading,
Chris

Stratus, Cumulous, Cirus

So, I have finally gotten around to starting to learn about "the cloud." No, not the clouds mentioned in the title, but the distributed-computing "cloud."

I decided that I would start with Hadoop. It took a day to get my linux box up and running, but once I got Ubuntu updated to the latest version , it was time to install Hadoop. The install was actually quite simple, and I was ready to head right into mapreduce.

So, I ran the example from the command line (word count over the example XML files). Success, I was able to get hadoop to aggregate the counts of words from a couple files! So, my next step was to move on to something exciting: mapreduce with Hadoop using PHP - Hadoop Streaming!

I was able to get the word count example working with PHP. I decided I would try and do something a little more complicated, though, and used my PHP reducer to push the results into a MySQL database. It worked, but I don't think it got the correct results, because I was doing an "INSERT IGNORE" and the reduce ran several times (therefore ignoring each successive time the reduce job ran).

So, I want/need to mess around with it some more. I still don't feel like I have an understanding of how to take advantage of the map/reduce approach. Using word count, we aren't linking back to the original document. I need to find a slightly more advanced example. It doesn't have to be complex, but something that gets me back to the original document so that I can start "using" map/reduce to get a better feel for how it works in the real world.

Links:
Hadoop Home page - http://hadoop.apache.org/core/
Hadoop Wiki - http://wiki.apache.org/hadoop/
Hadoop Streaming - http://hadoop.apache.org/core/docs/r0.15.2/streaming.html