So, after reading the first couple chapters of "Hadoop: The Definitive Guide" I have come to the realization that most of the discussion about cloud is about the architecture itself. I know I haven't gotten very far in the book, and the author is probably providing the necessary background information, but I think too much time is being spent on the framework.
I want to focus on thinking in terms of map and reduce. I still don't have any good examples yet, but my goal is to have that one map/reduce job that makes it click. Every time I feel like I am on to something, it seems like whatever article I am reading goes back to the framework. I keep asking myself if I am just missing something, or is this process really that complex, but I hope that it just takes a real-world example to get me started.
The problem I have run into with examples so far is that they have little value by themselves. Take word count; creates a count of words in an entire series of documents. Although this is handy, the first question out of a user's mouth will be... wow, word x was used n times. I wonder what documents those words were used in?
I really just need a sample map/reduce task that spells out a valuable map reduce. Perhaps I just need to shift my focus from trying to identify valuable information to simple, key/value pairs. What series of key/values would provide value? One thing that makes me feel like I am getting close is the concept of multiple-field keys. Take the word count example. Lets say I have 3 documents:
Doc 1:
Hello world!
Doc 2:
I love perl.
Doc 3:
Perl is the most used language in the world.
Ignoring how we parse out words (to separate out world! from world. from world), a possible set of key/value pairs for word counts could be:
Doc1//hello 1
Doc1//world 1
Doc2//I 1
Doc2//love 1
Doc2//perl 1
Doc3//perl 1
Doc3//is 1
Doc3//the 2
etc
I know, this isn't rocket science, but I think that this is where the key to using map/reduce will come from. With these results, I will now be able to identify how many times a given word is used within a single document. Perhaps I should look at it a little different... perhaps word, and then a list of documents it came from:
1 - Doc1, etc
hello [1]
perl [2,3]
love [1]
world [2]
etc
Maybe this works, but I still feel like there is something missing. It seems like it would be extremely difficult to break down every problem into this type of solution (especially the more complex problems).
Maybe a better way to look at map/reduce is to start with a problem, and then try and find the map/reduce (or series of map/reduce) that can be used to solve the problem.
Scenario:
Lets say that we have all of the GPS position reports for all of the UPS drivers for a given city. Our goal is to identify where the driver has lunch each day (and eventually identify a pattern around where he has lunch).
Here are my initial thoughts:
Initially, we would want to parse out the GPS updates by driver. For each driver, we would then want to identify any time the location remained the same for more than 20 minutes (with the assumption that the drivers are required to take a 30 minute lunch).
So, here are my thoughts on the outputs:
Driver/DTG/ Location
Driver/Date/Location Dwell Time
If we sort these results by dwell time, this should give us locations where the driver stopped for more than 20 minutes.
Then, we want to put a list together of all of the locations where the driver has stopped for more than 20 minutes (to idenitfy all the lunch locations):
Driver/Date/ Lunch Location
Does this make sense like this? I guess I just need to work on breaking the problem into a series of key/value pairs, but I am not sure if I am going the right direction or not.
Well, it is getting late, and I don't think I can handle any more. I will try and read through this again and see if I can come up with a better idea.
No comments:
Post a Comment