Sunyit's Project BITS Documentations

The following is a collection of documents that I created solely for myself and colleagues in order to meet standards for implementing a Hadoop cloud service. That said there is a lot of information that is specific for the systems used and customized to only work for those who were apart the project. The objective for Project "Bits" can be found here in this link. All ip addresses have been marked with x's and urls generalized in order to protect the SunyIT network system. I continue to study the systems used here and release the documents in hope that others might take up the project and implement it at his or her's University/College.


Back-Bone of Bits Project
  • This is the server BitsGW which features a vpn connection across multiple colleges. Creating VM’s of BitsHP (hadoop machines) to have a scalable new projects. Also providing a LDAP connection service.
  • BitsGW has the following user: afassett, admin
  • BitsHP: Pxe server for machines behind it for Hadoop nodes. Which are booted dynamically without drives over the network.
  • Nodes: clients to the master, using dsl version of linux to size down on amount of memory the os uses to saving space.
Creating the BitsGW  (Building the Vpn Service)e
  1. Install vpn service
    • root@bitsGW:apt-get install openvpn
  2. using VPN connection using files from colleges.(there should be a key and conf files)
    • Grabbing the files from USB
      • root@bitsGW:mkdir -p /mnt/usb
      • root@bitsGW:grep SCSI /var/log/messages|less (find what the usb is called)
      • root@bitsGW:mount -t vfat -o rw, users /dev/sdb1 /mnt/usb
  3. Input files
    • root@bitsGW:mv client-sunyit.conf /etc/openvpn
    • root@bitsGW:mv lair /etc/openvpn
  4. Start Vpn with configurations
    • root@bitsGW:openvpn [client config file]
    • to simplify input this → /etc/openvpn/client-sunyit.conf
Resources:HOWTO: OpenVPN

Created a directory for “log” and inside it is logs for any errors on the file "backbone.log". I also made sure that the key file was chmod 600 or 700 either does not affect the process. Another problem encountered was using the permission script on openvpn you may need to use the following.

openvpn --script-securitory 2 --config client-sunyit.conf
Building the Ldap Service

Installing Ldap server using Version of Ldap 2.4.33

Commands
  1. Getting the file... wget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-.4.33.tgzwget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.33.tgz
  2. opening the file... gzip -dc openldap-2.4.33.tgz |tar x
  3. More Documents to read on SSL connections... http://www.openldap.org/doc/admin24/install.html
  4. Run configuration... Results: configure: error: BDB/HDB: BerkeleyDB not available

Download: wget download.oracle.com/otn/berkeley-db/db-5.3.21.tar.gz This will not work!!!
Must have a account and download it manually...

Berkely Installation

Used scp, downloaded the file from my windows machien to the linux server. Info on scp for help...

 Other Research
Find a Google drive command line that looked interesting...may work but not going to go into it right now will just send it through a usb..


Network Research - Of Day 1
This is a record of all the network problems faced & how to debug them.

 Resources
Network Diagnose

Testing on Linux
I tried with the following.../etc/network/interfaces, two files of interests 
/syslinux/syslinux.cfg &  
/syslinux.cfg.
  • auto lo
  • iface lo inet loopback
  • allow-hotplug eth0
  • iface eth0 inet static
  • address x.x.x.x
  • netmask x.x.x.x
  • network x.x.x.x
  • broadcast x.x.x.x
  • gateway x.x.x.x
  • #dns-* options are implemented by the resolvconf package if installed
  • #dns-nameservers x.x.x.x
  • #dns-search cs.sunyit.edu
Other commands tried...
  • /etc/resolv.conf
  • search cs.sunyit.edu
  • nameserver x.x.x.x
  • >route
  • >arp -a 
  • >sudo netstat -natp | grep sshd ← see if the service is running and what ip/ports listening on.
RESULT
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1180/sshd
tcpd 0 0 :::22 :::* LISTEN 11180sshd
Conclusion: all correct.

With a Windows machine I tried with a static IP and a dynamic IP, I wanted to see if this was my linux setup or just how I was doing it the process. In both case nothing worked and did not have any other settings.

Important commands
  • inside file /etc/rc.conf
  • /etc/init.d/networking stop
  • /etc/init.d/networking start
  • ifconfig_de2="inet x.x.x.x netmask x.x.x.x"

Network Resources of Day 2

With nick help learning new commands!
  • dhclient eth0
  • ifup eth0

This script lets me know whats on the network!
for i in {1..10}; do ping -c1 10.107.1.$i; done

Resource

http://serverfault.com/questions/73360/assigning-sequential-dynamic-ip-addresses-in-dhcpd

 Building with real world (PXE Server)
File of importance - /boot/syslinux/syslinux.cfg
apt-get install syslinux

Linux Tutorial - Linux DHCP Server Configuration


Diskless Server
DisklessUbuntuHowto - Community Ubuntu Documentation
PXELINUX - Syslinux Wiki
apt - How to customize live Ubuntu CD? - Ask Ubuntu - Here is a way to modify the iso we get to clients..

Notes on Hadoop
Java SDK Issues
Problems with Source.list
Debian User Forums • View topic - Can't install anything with apt

Other Cloud Research

Popular posts from this blog

Aspetcore: NLog with Postgresql

This Article is base ASP.Net Core 2 and the latest version of Nlog. 1. Install NLog and the Postgresql Package Using NuGet to install the package into your ASP.NET Core project: PM > Install-Package NLog.Web.AspNetCore PM > Install-Package Npgsql 3. Register NLog Add in your Startup.cs  public void Configure ( IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory ) { ... env.ConfigureNLog( "nlog.config" ); ... // make sure Chinese chars don't fk up Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); //add NLog to ASP.NET Core loggerFactory.AddNLog(); //add NLog.Web app.AddNLogWeb(); } 4. Add to Your Controller Add a logger object to your Controller. private readonly ILogger<YourController> _logger; Modify the constructor to use it public YourController ( ILogger<YourController> logger = null ) { if ( null != logger) { _logger = logger; ...
Read more

Mongodb Cheatsheet

This cheat sheet is based on the latest version of mongodb, which initial had setup and didn't encounter any problems. However that all changed when I was working remote and forgot I disabled the port no my network, rather than with the server. So I went pointing around where I didn't need to and ended up messing things up. The Best tutorial out there is of course on digital ocean for my setup on ubuntu 16.04 , this articles isn't so much about getting setup. I just wanted to include it because I found myself looking back at it to see if I missed a step. Bugs/Problems Now comes all the messed up things I found and had to deal with. firewall - Port seems to be open, but connection refused try using 0.0.0.0 in the bin_ip for the /etc/mongo.conf try sudo ufw disable and check to see if it works without the firewall (probably still wont but worth a shot), then just sudo ufw enable. Make sure port 27017 is open. Then check this  linux - Connection refused to MongoDB e...
Read more

Installing Windows on Acer chromebook 15 cb3-532

Much of my notes are just small problems I faced (what's new!) haha. Either way this was a really fun project to do for a friend and I got excited once I research all the information. You can find instruction videos but what helped me the most was the coolstar website. Sadly they don't have parameters in the url so I can link the machine I used in this article. -  https://coolstar.org/chromebook/windows-install.html Scared of messing with the bios? Don't be!    So you need to first remove the screw to put a legacy bios on the machine. Its easy to backup the chrome os one and with the same steps to change it. You can go back in and revert to the chrome version bios. Its that last part that had me worried and its very easy to do.  Problems getting into Developer Mode?   Follow the instructions to the letter, during the step to disable the OS verification, press Ctrl-D to boot into os. If you hear two beeps I would go back to set one. I got in a loo...
Read more