How to Read/Write to a Apache hadoop file system (HDFS) file system using JAVA

Deploy a HDFS cluster.

Create HDFS client using JAVA.

Genarate Maven2 project.

mvn archetype:generate -DgroupId=org.wso2.carbon -DartifactId=HDFSClient
hadoop-core is the only dependency you want in maven2 based project.

set cluster config in the client

do file operation via JAVA API

find maven2 based sample in in OT svn.



How to install / uninstall OSGI bundles in WSO2 Carbon.

Download and extract WSO2 Carbon []

wso2carbon-core-3.2.0-SNAPSHOT/bin$ ./ -DosgiConsole

Check the Bundle status

osgi> ss

Framework is launched.

id    State       Bundle
0    ACTIVE      org.eclipse.osgi_3.5.0.v20090520
Fragments=1, 2, 3
1    RESOLVED    geronimo-jta_1.1_spec_1.1.0.wso2v1

Install Bundle

osgi> install file:/<file path>/<file name>

Bundle id is 134

133    ACTIVE      xmlbeans_2.3.0.wso2v1
134    INSTALLED   org.wso2.carbon.hdfs_0.1.0.SNAPSHOT

Check status of the Bundle


135    INSTALLED   hadoop-core_0.20.2.wso2v1

Start Bundle
osgi> start 135

135    ACTIVE      hadoop-core_0.20.2.wso2v1

Stop Bundle

osgi> stop 135

osgi> ss

135    RESOLVED    hadoop-core_0.20.2.wso2v1

HOW to Build WSO2 Carbon SNAPSHOT (trunk) in Amazon EC2 with EBS

Launch Amazon Admin Console.

Create a Debian 6.0 instance.

Use public image  [ debian-6.0-squeeze-64bit-minimal-pvgrub-20110222-05 (ami-1e39ca77) ]  to create an instance.

Login to the EC2 instance using created key

ssh -i   root@<ec_public_domain_name>

Create 30 GB EBS  ( full carbon build need 17 – 18 GB space to accommodated SVN src and binaries) and mount it into /mnt/carbon.

Create user named carbon and set home folder as /mnt/carbon/users/carbon

Change the user in to carbon and do all the build related activities.

Create Build environment in /mnt/carbon/opt/java.

Setup java / maven2 / Ant in /mnt/carbon/opt/java

Take  WSO2 Carbon trunk checkout using https url. You need  around 2.5 GB space only for SVN check out.

svn co  /mnt/carbon/src/carbon_trunk

Start build process from /mnt/carbon/src/carbon_trunk

mvn clean install

For a quick build switch off Tests

mvn clean install -Dmaven.test.skip=true

If you get issues in building try individual components following build order in root pom.xml

Current build order:


Full build need 6 - 7 GB space for .m2 to store all artifacts, another 2.5 - 3 GB for Carbon source and need more space for each product.
For a full WSO2 Carbon build system needs around 17 - 18 GB space.