So, I have finally gotten around to starting to learn about "the cloud." No, not the clouds mentioned in the title, but the distributed-computing "cloud."
I decided that I would start with Hadoop. It took a day to get my linux box up and running, but once I got Ubuntu updated to the latest version , it was time to install Hadoop. The install was actually quite simple, and I was ready to head right into mapreduce.
So, I ran the example from the command line (word count over the example XML files). Success, I was able to get hadoop to aggregate the counts of words from a couple files! So, my next step was to move on to something exciting: mapreduce with Hadoop using PHP - Hadoop Streaming!
I was able to get the word count example working with PHP. I decided I would try and do something a little more complicated, though, and used my PHP reducer to push the results into a MySQL database. It worked, but I don't think it got the correct results, because I was doing an "INSERT IGNORE" and the reduce ran several times (therefore ignoring each successive time the reduce job ran).
So, I want/need to mess around with it some more. I still don't feel like I have an understanding of how to take advantage of the map/reduce approach. Using word count, we aren't linking back to the original document. I need to find a slightly more advanced example. It doesn't have to be complex, but something that gets me back to the original document so that I can start "using" map/reduce to get a better feel for how it works in the real world.
Links:
Hadoop Home page - http://hadoop.apache.org/core/
Hadoop Wiki - http://wiki.apache.org/hadoop/
Hadoop Streaming - http://hadoop.apache.org/core/docs/r0.15.2/streaming.html
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment