1.
WEB SERVICE -
Open Netbeans, new Web Application -> JAVA EE 6
Create a new WebService by right-click on project folder
In the java file, add necessary operations thru : add operation
Build
-- Test -- Test WebService -- Deploy
Service is opened in browser. Copy the WSDL URL for when client is created.
As a new project , create
new Web Application -> JAVA EE 6
Create a new WebServiceClient and paste the wsdl url when required.
Go to WebServiceReferences , drag and drop the modules into the jsp file.
Basic syntax to create a form to be displayed on browser :
Basic example using "add method" for mainjsp file (Alter jsp file with necessary methods)
<%@page contentType="text/html" pageEncoding="UTF-8"%>
ARITHMETIC PAGE
<%-- start web service invocation --%>
<%
try {
com.cal.example.Calculate_Service service = new com.cal.example.Calculate_Service();
com.cal.example.Calculate port = service.getCalculatePort();
String no1=request.getParameter("num1");
String no2=request.getParameter("num2");
String subty=request.getParameter("submit");
if(subty.equals("ADD"))
{
int num1=Integer.parseInt(no1);
int num2=Integer.parseInt(no2);
int result= port.add(num1, num2);
out.println("ADD Result = "+result);
}
} catch (Exception ex) {
// TODO handle custom exceptions here
}
%>
<%-- end web service invocation --%>
Then right click on Client and RUN.
---------------------------------------------------------------------------------------------------------------------
2.
FOR ANY VM EXP., DO THE FOLLOWING TO ENSURE PARTIAL O/P :
CREATE VM : (For desktop iso)
Open VirtualBox and select a vm, go to :
Settings -> General -> Advanced .... Change .... Shared Clipboard : Bidirectional ,
Drag'n'Drop : Bidirectional .
Settings -> Storage -> Controller:IDE ... Choose desktop iso ( ending with .04)[16.04
preferably]
Settings -> Network -> Adapter1 ( Select BRIDGED ADAPTER ) .. click on Advanced ->
Promiscuous mode .. Select Allow All.
Save the settings and START.
After installing UBUNTU, go to Network Settings and set proxy.
Open System Settings -> Network -> Options -> IPv4 Settings :
Method : Manual
Change the Address, SubnetMask, Gateway, DNS server address as per
given values and save changes.
Open Terminal , type :
sudo gedit /etc/resolv.conf
if nameserver is not set to dns server value .. type the address beside the
else just close it.
Open Mozilla Firefox and check if Internet is working
(CHECKPOINT 1)
'nameserver',
TO INSTALL JAVA :
In Terminal , type :
sudo apt-get install default-jdk
After return of control,
type
Then, create a java file , say
:
java -version
to check if installation was errorfree.
: xyz.java
Go to Terminal :
javac xyz.java
java xyz
If it works ..... (CHECKPOINT 2)
---------------------------------------------------------------------------------------------------------------------------
TO INSTALL VM SERVER :
Choose server iso and continue with installation . (Change settings as in desktop iso)
Choose configure the network in Main Menu
Configure network manually
Enter IP Address, subnet mask and gateway as given.
Partitioning method
:
Guided - use entire disk
Setup with hostname, username , password, proxy and no automatic updates.
Install OpenSSHServer ... to select software or deselect software , press 'space' and then press 'Enter' to
move to the next step in menu. [DO NOT press enter before selecting necessary softwares]
Install GRUB Boot Loader
After server loads, PING ANOTHER VM by using :
ping
eg :
ping 10.6.4.155
This ensures connectivity.
-----------------------------------------------------------------------------------------------------------------
3.
REMOTE LOGIN :
WITHOUT PASSWORD :
After executing the above steps in server or desktop -
(For desktop , enter the following command to install sshserver and update changes :
sudo apt-get install openssh-server
sudo apt-get update
)
Enter the following commands in order :
ssh-keygen -t rsa
// press enter when asked for keyphrase and file location
{ Now, change current directory to that of ssh :
For example , if it shows
do ....
cd ~/.ssh
ls
id_rsa.pub)
cd vm1/.ssh
/home/vm1/ .ssh/id_rsa
or
cd /.ssh
while generating key,
or
cd /root/.ssh
or
}
//list the contents in directory ( authorized_keys
id_rsa
//if anything is not listed, theres some error and google on how to fix
chmod 700 id_rsa.pub
cp id_rsa.pub authorized_keys
ssh-copy-id @
{ for eg :
ssh-copy-id vm2@10.6.4.155 }
Now vm1 s terminal is logged into vm2
Enter
exit
to return to original vm .
WITH PASSWORD :
(For desktop , enter the following command to install sshserver and update changes :
sudo apt-get install openssh-server
sudo apt-get update
)
Enter the following command :
ssh @
{ for eg :
ssh vm2@10.6.4.155 }
Now vm1 s terminal is logged into vm2
Enter
exit
to return to original vm .
-----------------------------------------------------------------------------------------------------------------------
4 . FILE TRANSFER BETWEEN VM :
Execute the steps in order till installation of ssh server.
TO SEND FILE :
Type in the following command in the terminal :
scp /home/vm1/ vm2@ipaddress2:
Eg
scp /home/vm1/f1.txt vm2@10.6.4.155:/home/vm2/
:
TO RECEIVE FILE :
Type in the following command in the terminal :
{
If nano editor is not installed, then install it using this command
and update :
sudo apt-get install nano
sudo apt-get update
}
scp vm2@ipaddress2:
scp vm2@10.6.4.155:/home/vm2/f1.txt f2.txt
Eg :
Check for transfer by typing ls in terminal or go to the folder and view the file.
-----------------------------------------------------------------------------------------------------------------
5.
FOLDER TRANSFER BETWEEN VM :
First, create a directory using command :
mkdir
Eg
:
mkdir myDir
Inside the directory, create one or more files.
Then , type the following command in the terminal :
scp -r
Eg :
scp -r /home/vm1/myDir vm2@10.6.4.155:/home/vm2/
Check for transfer by typing ls in terminal of vm2 or go and check in the file system .
---------------------------------------------------------------------------------------------------------------------------
6.
EUCALYPTUS :
Create 2 VMs .... Configure network as DHCP .. dont use Ethernet.
Install server iso for both
Select Ubuntu Enterprise cloud
Set hostname , leave cloud controller address blank
(Press 'space'
Select the following :
then press
'enter')
Cloud controller,Walrus storage service,Cluster controller,Storage controller
Node controller
Partition disks :
{ FOR VM 2 }
select “Guided-use entries disk and set up LVM‟
Enter size as 10.5 GB
Enter username , password , clustername
Leave pool of IP addresses blank and then , Install
Create 3rd VM with desktop iso
GRUB Loader
to act as client (VM3)
Install qemu-kvm in VM1
Set a temporary password in VM2
Type the command in VM1
:
using
:
sudo passwd eucalyptus
{ FOR VM 1 }
sudo -u eucalyptus ssh-copy-id -i /var/lib/eucalyptus/.ssh/id_rsa.pub eucalyptus@
Remove temporary password in VM2
sudo passwd -d eucalyptus
using :
In VM3, go to Mozilla Firefox browser and type the following URL :
(if it doesnt load , type
Fehler! Hyperlink-Referenz ungültig.
Hyperlink-Referenz ungültig. /#login )
Username
:
admin
Password
:
DONT USE PROXY
admin
Give new username, password, email id.
Go to Credentials ->
Then
....
Download Credentials
{Download to Downloads}
cd Downloads
Transfer the file to VM1 using
:
scp euca2-admin-x509.zip vm1@ip_vm1:/home/vm1
In VM1 :
Type
:
mkdir -p ~/.euca
cd ~/.euca
chmod 0700 ~/.euca
chmod 0600 ~/.euca/*
sudo euca_conf –get-credentials mycreds.zip
unzip mycreds.zip
Check the contents using
In VM3 :
Type
:
sudo apt-get update
ls command.
:
Fehler!
sudo apt-get install euca2ools
Go to file named eucarc under the X.509 folder downloaded as certificate credentials
identify the
:
URL , ACCESS KEY , SECRET KEY
.
Now , type the following in the terminal
:
( its capital ' i '
and
before )
euca-create-volume –U -I -S --size 1 –z
euca-describe-volumes –U -I -S
The terminal output indicates private cloud setup and volume creation in client machine.
-----------------------------------------------------------------------------------------------------------------------
7.
OPEN NEBULA
Installation - Install 2 desktop iso’s or use a single vm and 2 terminals (use as root only for front end
node )
Frontend Installation : (VM1)
sudo –i
apt-get update
Install packages and dependencies:
apt-get install opennebula opennebula-sunstone nfs-kernel-server
To check if packages were installed: ls –l /dev/kvm
Open the file: gedit /etc/one/sunstone-server.conf
a. Change the line ‘:host: 127.0.0.1’ to ‘:host: 0.0.0.0’ (leave untouched if latter is already present)
Restart sunstone server: /etc/init.d/opennebula-sunstone restart
Generate keys: ssh-keygen –t rsa (click Enter for all subsequent queries)
Copy keys in the following manner:
a. cd /root/.ssh
b. chmod 600 id_rsa.pub
c. cp id_rsa.pub authorized_keys
Create a file in the same directory and put the following contents in it:
a. gedit config
b. Content:
Host* StrictHostKey Checking no
UserKnownHostsFile ./dev/null
Node Installation: (VM2)
Perform: apt-get update
Install packages and dependencies:
a. apt-get install opennebula-node nfs-common bridge-utils
Configure server interface (at node):
a. cd /etc/network/interfaces.d
Create a file and put the following contents in it
a. gedit eth0.config (delete any other file starting with ‘eth0.config’)
b. Content:
auto lo iface lo
inet loopback
auto br0
iface br0 inet static
address
network 192.168.1.4
netmask
broadcast
gateway
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
Restart networking: /etc/init.d/networking restart
Open Nebula Sunstone Log-In:
You can go to Open Nebula Sunstone’s Home Page on your browser using: ‘http://localhost:9869’
Username: oneadmin
Password: (Unique for each installation)
Found in terminal using:
1. su – oneadmin
2. cat .one/one_auth
Homepage should get displayed.
------------------------------------------------------------------------------------------------------------------------------------------
8.
CREATION OF VM TEMPLATE – OPEN NEBULA
1. Log-in to Open Nebula Sunstone Home Page.
2. On the left pane, click on Templates Tab and click on VMs
3. To add a VM Template, click on the green + button.
4. Enter the name, description and other attributes
5. After selecting all the options above, click the green Create button. You should see the main VM
Templates page again with your template updated.
------------------------------------------------------------------------------------------------------------------------------------------
9.
LIVE MIGRATION OF VM
Install OpenNebula Front-End VM and KVM node VM(Refer EX: 7 ) [ 2 VMs with Ubuntu 16.04
desktop image]
After installing OpenNebula front-end and kvm node , the following steps need to be performed in
the front-end VM:
1. Initially , list all the hosts,templates and vms using:
$ onehost list
$ onetemplate list
$ onevm list
2. CREATION OF HOSTS:
2.1. The hosts can be created via the command line
$ onehost create frontend –i kvm –v kvm –n dummy
(or) Through the open nebula web interface : localhost:9869 . Login with the username and
password as given in /var/lib/one/.one/one_auth file using the command
$ sudo gedit /var/lib/one/.one/one_auth
2.2 Navigate to Infrastructure -> hosts in the left menu pane in the web interface.
2.3. Click on the ‘+’ add option and specify the hostname and click on “create”.
2.4. Similarly create another host and both will be listed.
On clicking the enable button after creating the hosts, the status changes to init.
$onehost list can be used to list the hosts in the terminal.
3. CREATION OF TEMPLATE:
3.1. Under the virtual resources > template option is selected and in the similar way to hosts ,
templates are created.
$onetemplate list will specify the new template created.
4. CREATION OF VM :
4.1. The template created is instantiated to create a virtual machine by clicking on the “instantiate”
option in templates.
5. Deploy and migrate the VM- vm1 on host by specifying the vm id and host id.
$ onevm deploy
$ onevm migrate
Here, vm1 with id 0 is deployed on host 1 with id 0. host1 appears under the host column.
------------------------------------------------------------------------------------------------------------------------------------------
10 .
HADOOP INSTALLATION
Hadoop 3.1 Installation Steps
Install Java 8 and verify that it is working.
>> java -version
Step 1:Add User hduser with sudo privilege
>> adduser hduser
>> usermod -aG sudo hduser
Step 2: Install SSH Server and add the private key to the known lists
>> ssh-keygen -t rsa ‘’ -f ~/.ssh/id_rsa
Which generate public keys using rsa method.
>> cat ~/.ssh/id_rsa.pub
>> ~/.ssh/authorized_keys
>>chmod 0600 ~/.ssh/authorized_keys
Check whether you will be able to access to your localhost
through ssh ‘without a password’.
>>sudo apt-get install openssh-server
>>ssh localhost
Step 3: Installing Hadoop 3.1
3.1 Extract and move the hadoop to an installation directory)
>>wget www-us.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
or download manually from
www-us.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
>> tar xfz hadoop-3.1.1.tar.gz
extract zip file
>>mv hadoop-3.1.1 /usr/local/hadoop
Move extracted file into local disk (Installed)
3.2 Switch local user to hduser
>>su hduser
3.2 Set the hadoop environment variables
>>nano ~/.bashrc
Add the following lines to the bashrc file.
#HADOOP Variables
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
Save the bashrc file and exit by CTRL+O followed by CTRL+X
Refresh the bashrc file so that our environment variables can
be accessed.
>>. ~/.bashrc
Check the hadoop version
>>hadoop version
3.3 Change the Hadoop and Related Config
3.3.1 Change the JAVA_HOME variable in
$HADOOP_HOME/etc/hadoop/hadoop-env.sh file.
>>sudo nano $HADOOP_HOME/hadoop/hadoop-env.sh
set the JAVA_HOME=<>
3.2.2 Change the Hadoop Core config in
$sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml
tag.
Add the following property tag inside the configuration
fs.default.namehdfs://localhost:9000
3.2.3 Add the following properties to hdfs-site.xml
configuration
$sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml
dfs.replication1dfs.name.dirfile:///home/hduser/hadoop-store/hdfs/namenodedfs.data.dirfile:///home/hduser/hadoop-store/hdfs/datanode
3.2.4 Add the following properties to the mapred-site.xml
>>sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml
mapreduce.framework.nameyarnmapreduce.application.classpath$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*$HADOOP_MAPRED_HOM/shar
e/hadoop/mapreduce/lib/*
3.2.5Add the following properties to the yarn-site.xml
>>sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
yarn.nodemanager.aux-servicesmapreduce_shuffleyarn.nodemanager.env-whitelist
JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,
CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOM
E
4. Format and Prepare the HDFS
4.1 Make Sure the data directory mentioned in the hdfs-site.xml is exist and the current user has
permission to it.
>>sudo chown -R hduser:hduser /usr/local/hadoop
>>sudo chmod 777 /usr/local/hadoop/logs
(make sure current user has permission to it)
>>hdfs namenode -format
5. Start the hadoop dfs and yarn process
>>start-dfs.sh
>>start-yarn.sh
6. Check the hadoop deamons by running the following command
>>jps// java process
7. Access the Hadoop DFS and Hadoop YARN Web UI,
-- for NameNode UI
http://localhost:9870/
For -- YARN UI
http://localhost:8042/
WORD COUNT
Step 1: Start Hadoop start-dfs.sh
Start yarn start-yarn.sh
Step 2 : Access the Hadoop DFS and Hadoop Yarn Web UI http://localhost:9870/ -- NameNode UI
http://localhost:8042/ -- YARN UI
Step 3: Create a new folder in hadoop hadoop fs -mkdir /wordcountfolder
Step 4: Create a sample input text file for the word count.
gedit sampleinput.txt
Step 5: Copy the sample file to the hadoop filesystem hadoop fs -put sampleinput.txt
/wordcountfolder
Step 6: Create WordCount.java file gedit WordCount.java
Step 7: Copy the WordCount.java to hadoop filesystem
hadoop fs -put WordCount.java /wordcountfolder
Step 8: Run the WordCount.java file
hadoop com.sun.tools.javac.Main WordCount.java
Two new class files will be generated WordCount$IntSumReducer.class,
WordCount$TokenizerMapper.class
Step 9: Create a jar file using the class files generated
jar cvf WordCount.jar WordCount*.class
Step 10: Run the jar file with the sample input text
hadoop jar WordCount.jar WordCount /wordcountfolder/sampleinput.txt
/wordcountfolder/output.txt
Step 11: Open the output.txt in the hadoop web UI The part-r file contains the output. Download the
file and open it using gedit.
WordCount.java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordCount {
public static class TokenizerMapper
extends Mapper