How to Read/Write to a Apache hadoop file system (HDFS) file system using JAVA

Deploy a HDFS cluster.

Create HDFS client using JAVA.

Genarate Maven2 project.

mvn archetype:generate -DgroupId=org.wso2.carbon -DartifactId=HDFSClient
hadoop-core is the only dependency you want in maven2 based project.
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>0.20.2</version>
</dependency>

set cluster config in the client

do file operation via JAVA API

find maven2 based sample in in OT svn.

 

 


How to install / uninstall OSGI bundles in WSO2 Carbon.

Download and extract WSO2 Carbon [http://wso2.com/products/carbon/]

wso2carbon-core-3.2.0-SNAPSHOT/bin$ ./wso2server.sh -DosgiConsole

Check the Bundle status

osgi> ss

Framework is launched.

id    State       Bundle
0    ACTIVE      org.eclipse.osgi_3.5.0.v20090520
Fragments=1, 2, 3
1    RESOLVED    geronimo-jta_1.1_spec_1.1.0.wso2v1
Master=0

Install Bundle

osgi> install file:/<file path>/<file name>

Bundle id is 134

133    ACTIVE      xmlbeans_2.3.0.wso2v1
134    INSTALLED   org.wso2.carbon.hdfs_0.1.0.SNAPSHOT

Check status of the Bundle

osgi>ss

135    INSTALLED   hadoop-core_0.20.2.wso2v1

Start Bundle
osgi> start 135

135    ACTIVE      hadoop-core_0.20.2.wso2v1

Stop Bundle

osgi> stop 135

osgi> ss

135    RESOLVED    hadoop-core_0.20.2.wso2v1


HOW to Build WSO2 Carbon SNAPSHOT (trunk) in Amazon EC2 with EBS

Launch Amazon Admin Console. http://aws.amazon.com/console/

Create a Debian 6.0 instance.

Use public image  [ debian-6.0-squeeze-64bit-minimal-pvgrub-20110222-05 (ami-1e39ca77) ]  to create an instance.

Login to the EC2 instance using created key

ssh -i   root@<ec_public_domain_name>

Create 30 GB EBS  ( full carbon build need 17 – 18 GB space to accommodated SVN src and binaries) and mount it into /mnt/carbon.

Create user named carbon and set home folder as /mnt/carbon/users/carbon

Change the user in to carbon and do all the build related activities.

Create Build environment in /mnt/carbon/opt/java.

Setup java / maven2 / Ant in /mnt/carbon/opt/java

Take  WSO2 Carbon trunk checkout using https url. You need  around 2.5 GB space only for SVN check out.

svn co https://svn.wso2.org/repos/wso2/trunk/carbon/  /mnt/carbon/src/carbon_trunk

Start build process from /mnt/carbon/src/carbon_trunk

mvn clean install

For a quick build switch off Tests

mvn clean install -Dmaven.test.skip=true

If you get issues in building try individual components following build order in root pom.xml

Current build order:

<module>dependencies</module>
<module>orbit</module>
<module>service-stubs</module>
<module>core</module>
<module>components</module>
<module>features</module>
<module>samples/shopping-cart</module>
<module>products</module>

Full build need 6 - 7 GB space for .m2 to store all artifacts, another 2.5 - 3 GB for Carbon source and need more space for each product.
For a full WSO2 Carbon build system needs around 17 - 18 GB space.