Hadoop, mongoDB: MapReduce tools

This blog post will be eventually written when time permits :)

- Is there any use case of using monogDB as a data server for Hadoop?

Finished... 23 Months Gone!!! Got my M.S. degree

Finally, it is over!!! I got my M.S. degree in Computer Science at SFSU. It was a long, fun, and sometimes tiring journey, but I finally accomplished another dream of mine...

The thesis write up was a big rush... 4 Months, no sleep... Got lots of experience in Data Persistence for Sensor Networks, Cloud-computing techniques in Data Persistence using Key-Value-Pair data model, Database Shards and Partition...

Well, that is all...

Presentation of a Peer-Reviewed Paper...

In the middle of my dissertation write-up, I went to present a peer-reviewed paper related to my M.S. dissertation... It was fun to get feedback from other researchers in the area of Sensor Networks...

Arno Puder, Teresa Johnson, Kleber Sales, Marcello de Sales, and Dale Davidson. A Component-based Sensor Network for Environmental Monitoring. In SNA-2009: 1st International Conference on Sensor Networks and Applications, pages 54–60, San Francisco, CA, USA, 2009. The International Society for Computers and Their Applications - ISCA.

Below is a picture of the event that take place in November 4th, 2009.

Building a Clustered Shared SubNetwork with VirtualBox 3.0.2 (Ubuntu 9.04 Host + Ubuntu 9.04 and Windows XP Guests)

VirtualBox 3.0.2 (Ubuntu 9.04 Host + Ubuntu & Windows Guests)

Problem: to have a cluster of machines to work for development using VirtualBox guests. That is, I want to be able to have different operating systems running on my own development box, having each of the machines sharing the same Internet connection (my main one) and acquiring IP address from the same DHCP server (aka wireless router, or whatever...)

Description:

Finally, something to make all of us happy! VirtualBox 3.0.2 delivers smooth support to shared Nat and Subnetwork outside-the-box!!! This has always been a dream of mine to use VirtualBox for such agility of having as much development machines as I can (your host memory).

What I could accomplished tonight was the setup of a shared subnetwork between my host Ubuntu 9.04 and my guests on Ubuntu 9.04 and Windows XP SP2. Everything I did was to setup network cards from each of my guests to use the network interface as a Bridged Adapter, and choose the one I have Internet connectivity. By the way, the connectivity I am using is my wireless card.

This can help anyone who needs different boxes and installations of machines to develop and test applications that depend on the network, shared resources, etc. For example, I need to build a Sensor Network that sends collected data to the main data center as my thesis simplest workspace. On a single box, my development environment is contains source-code for both the Sensors and the Main node. However, with the use of Subversion and SCP I can transfer artifacts over between the boxes and still test the transport layer of my sensor network.

Setup:

To better describe what I have, here's the description of each member of my network made with VirtualBox 3.0.2:

* Install VirtualBox 3.0.2 from http://www.virtualbox.org/wiki/Downloads

* Host: Ubuntu 9.04 on an IBM ThinkPad T42, with 2GB (yes, my personal HP with 4GB recently burned out... NEVER BUY HP LAPTOPS).

mdesales@mdesales-laptop:~$ ifconfig

eth0 Link encap:Ethernet HWaddr 00:11:25:81:48:6d

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth1 Link encap:Ethernet HWaddr 00:0e:35:c6:5a:0f

inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20e:35ff:fec6:5a0f/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:583718 errors:235 dropped:235 overruns:0 frame:0

TX packets:316625 errors:0 dropped:5 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:763972253 (763.9 MB) TX bytes:30157656 (30.1 MB)

Interrupt:11 Base address:0x4000 Memory:c0214000-c0214fff

Note that you will need to know which device you currently have Internet connectivity. In my case, as you can see, I have it through the eth1. I acquire IPs through my private Wireless Router DHCP, which is connected to the Cable modem, in the range of 192.168.0.100 - 192.168.0.199.

* Guest 1: Ubuntu 9.04 Desktop

* Guest 2: Ubuntu 9.04 Mini

* Guest 3: Windows XP SP3

For each of them, I just configured the network device to use the Bridged Interface, and choosing a correct device. To be clear, you want to choose the one you currently have Internet connection. In my case, I have on the eth1, which happens to be my wireless card. However yours can differ!

* Environment Evaluation: I'd like to ping, ssh, share resources, build a mysql cluster, hadoop grid, test my grails app, write Android App... The dream I have always had is to have my own "cloud" of pre-defined VirtualBox with the profiles of what I usually need. I must say that I can finally rest my mind! I could finally make it work...

I have mixed up 2 different projects: my thesis boxes and a Software Engineering research developed in 2 different environments (that's why I love keeping my VirtualBox hard-disks in my backup storage for later reuse). Most of people who have tried integrating guests could only PING a machine, but not SSH, samba, etc. I got frustrated 3 or 4 months ago about the same situation and I had to settle. Anyway, the truth is that tonight I got it working without any voodoo command to turn off the firewall, to configure different tunnel... I just wanted it working... See how many people were having related problems here (http://www.savvyadmin.com/virtualbox-host-interface-networking-with-nat/)

Let's see each machine IP address

* Guest 2: Ubuntu 9.04 Mini: 192.168.0.100

* Guest 3: Windows XP SP3: 192.168.0.103. Note that the ping is from the Guest 2 to the Guest 3 machine!!! I had never accomplished this with previous versions of VirtualBox without reading the documentation for hours...

* Pinging the host from the Guest 2: this step I could reproduce using the voodoo steps described on other pages. Here's an example of the guest 2 pinging the host.

* SSH from among the machines: one HUGE step was that, at this time, we can definitely SSH from each of the machines. I just installed Ubuntu with openSSH server in each of them. That's ALL!!! I'm making an SSH connection from my machine to the Guest 2.

This is the SSH session between the Guest 2 and the Host.

The SSL connection between the hosts is done. In this screen, the user ssh from the Guest 3 (Windows) to the Guest 2 (Ubuntu).

Note that the window at far-most of the screenshot is the Host.

After testing all these features, I wanted to verify if the communication is done without no changes.

I was lucky to find out that the everything works outside-the-box, including Internet connection!!!!