Sunyit's Project BITS Documentations

The following is a collection of documents that I created solely for myself and colleagues in order to meet standards for implementing a Hadoop cloud service. That said there is a lot of information that is specific for the systems used and customized to only work for those who were apart the project. The objective for Project "Bits" can be found here in this link. All ip addresses have been marked with x's and urls generalized in order to protect the SunyIT network system. I continue to study the systems used here and release the documents in hope that others might take up the project and implement it at his or her's University/College.


Back-Bone of Bits Project
  • This is the server BitsGW which features a vpn connection across multiple colleges. Creating VM’s of BitsHP (hadoop machines) to have a scalable new projects. Also providing a LDAP connection service.
  • BitsGW has the following user: afassett, admin
  • BitsHP: Pxe server for machines behind it for Hadoop nodes. Which are booted dynamically without drives over the network.
  • Nodes: clients to the master, using dsl version of linux to size down on amount of memory the os uses to saving space.


Creating the BitsGW  (Building the Vpn Service)e
  1. Install vpn service
    • root@bitsGW:apt-get install openvpn
  2. using VPN connection using files from colleges.(there should be a key and conf files)
    • Grabbing the files from USB
      • root@bitsGW:mkdir -p /mnt/usb
      • root@bitsGW:grep SCSI /var/log/messages|less (find what the usb is called)
      • root@bitsGW:mount -t vfat -o rw, users /dev/sdb1 /mnt/usb
  3. Input files
    • root@bitsGW:mv client-sunyit.conf /etc/openvpn
    • root@bitsGW:mv lair /etc/openvpn
  4. Start Vpn with configurations
    • root@bitsGW:openvpn [client config file]
    • to simplify input this → /etc/openvpn/client-sunyit.conf
Resources:HOWTO: OpenVPN

Created a directory for “log” and inside it is logs for any errors on the file "backbone.log". I also made sure that the key file was chmod 600 or 700 either does not affect the process. Another problem encountered was using the permission script on openvpn you may need to use the following.

openvpn --script-securitory 2 --config client-sunyit.conf



Building the Ldap Service

Installing Ldap server using Version of Ldap 2.4.33

Commands
  1. Getting the file... wget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-.4.33.tgzwget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.33.tgz
  2. opening the file... gzip -dc openldap-2.4.33.tgz |tar x
  3. More Documents to read on SSL connections... http://www.openldap.org/doc/admin24/install.html
  4. Run configuration... Results: configure: error: BDB/HDB: BerkeleyDB not available

Download: wget download.oracle.com/otn/berkeley-db/db-5.3.21.tar.gz This will not work!!!
Must have a account and download it manually...

Berkely Installation

Used scp, downloaded the file from my windows machien to the linux server. Info on scp for help...

Other Research
Find a Google drive command line that looked interesting...may work but not going to go into it right now will just send it through a usb..


Network Research - Of Day 1
This is a record of all the network problems faced & how to debug them.

Resources


Network Diagnose

Testing on Linux
I tried with the following.../etc/network/interfaces, two files of interests 
/syslinux/syslinux.cfg &  
/syslinux.cfg.
  • auto lo
  • iface lo inet loopback
  • allow-hotplug eth0
  • iface eth0 inet static
  • address x.x.x.x
  • netmask x.x.x.x
  • network x.x.x.x
  • broadcast x.x.x.x
  • gateway x.x.x.x
  • #dns-* options are implemented by the resolvconf package if installed
  • #dns-nameservers x.x.x.x
  • #dns-search cs.sunyit.edu
Other commands tried...
  • /etc/resolv.conf
  • search cs.sunyit.edu
  • nameserver x.x.x.x
  • >route
  • >arp -a 
  • >sudo netstat -natp | grep sshd ← see if the service is running and what ip/ports listening on.
RESULT
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1180/sshd
tcpd 0 0 :::22 :::* LISTEN 11180sshd
Conclusion: all correct.

With a Windows machine I tried with a static IP and a dynamic IP, I wanted to see if this was my linux setup or just how I was doing it the process. In both case nothing worked and did not have any other settings.

Important commands
  • inside file /etc/rc.conf
  • /etc/init.d/networking stop
  • /etc/init.d/networking start
  • ifconfig_de2="inet x.x.x.x netmask x.x.x.x"

Network Resources of Day 2

With nick help learning new commands!
  • dhclient eth0
  • ifup eth0

This script lets me know whats on the network!
for i in {1..10}; do ping -c1 10.107.1.$i; done

Resource

http://serverfault.com/questions/73360/assigning-sequential-dynamic-ip-addresses-in-dhcpd

Building with real world (PXE Server)
File of importance - /boot/syslinux/syslinux.cfg
apt-get install syslinux

Linux Tutorial - Linux DHCP Server Configuration


Diskless Server
DisklessUbuntuHowto - Community Ubuntu Documentation
PXELINUX - Syslinux Wiki
apt - How to customize live Ubuntu CD? - Ask UbuntuHere is a way to modify the iso we get to clients..

Notes on Hadoop
Java SDK Issues
Problems with Source.list
Debian User Forums • View topic - Can't install anything with apt

Other Cloud Research

Comments

Popular posts from this blog

Ng-repeat: Order a List Vertically rather Horizontal

HTML5 Audio doesn't Autoplay for Chrome on Android

Updating Razor 2.0 to 3.0 with Asp.net MVC