Posts

Showing posts with the label Hadoop

Introduction to Hadoop and MapReduce Notes

Image
Class Listing on Udacity What is   big data -  a subjective term but mostly large amount of data that is usually difficult to be processed on a small machine not  necessarily  large amounts of data. Challenges  with data are that data comes in really fast and from multiple places. The three V's  Volume, Variety, Velocity References When to use HBase and when to use Hive - Stack Overflow Apache Flume – Architecture of Flume NG | Cloudera Developer Blog CDH  -  distribution of Apache Hadoop and related projects. Hadoop Streaming Hadoop Storing format. Introducing Parquet: Efficient Columnar Storage for Apache Hadoop | Cloudera Developer Blog hadoop - Storage format in HDFS - Stack Overflow Terms NameNode    MapReduce   Shuffle and Sort   Apache Spoop    Apache Nutch    The Final Much of this information below is on a Google doc that was some what hidden in the course wiki b...
Read more

Sunyit's Project BITS Documentations

Image
The following is a collection of documents that I created solely for myself and colleagues in order to meet standards for implementing a Hadoop cloud service. That said there is a lot of information that is specific for the systems used and customized to only work for those who were apart the project. The objective for Project "Bits" can be found here in this link . All ip addresses have been marked with x's and urls generalized in order to protect the SunyIT network system. I continue to study the systems used here and release the documents in hope that others might take up the project and implement it at his or her's University/College. Back-Bone of Bits Project This is the server BitsGW which features a vpn connection across multiple colleges. Creating VM’s of BitsHP (hadoop machines) to have a scalable new projects. Also providing a LDAP connection service. BitsGW has the following user: afassett, admin BitsHP: Pxe server for machines b...
Read more

Popular posts from this blog

Aspetcore: NLog with Postgresql

This Article is base ASP.Net Core 2 and the latest version of Nlog. 1. Install NLog and the Postgresql Package Using NuGet to install the package into your ASP.NET Core project: PM > Install-Package NLog.Web.AspNetCore PM > Install-Package Npgsql 3. Register NLog Add in your Startup.cs  public void Configure ( IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory ) { ... env.ConfigureNLog( "nlog.config" ); ... // make sure Chinese chars don't fk up Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); //add NLog to ASP.NET Core loggerFactory.AddNLog(); //add NLog.Web app.AddNLogWeb(); } 4. Add to Your Controller Add a logger object to your Controller. private readonly ILogger<YourController> _logger; Modify the constructor to use it public YourController ( ILogger<YourController> logger = null ) { if ( null != logger) { _logger = logger; ...
Read more

Mongodb Cheatsheet

This cheat sheet is based on the latest version of mongodb, which initial had setup and didn't encounter any problems. However that all changed when I was working remote and forgot I disabled the port no my network, rather than with the server. So I went pointing around where I didn't need to and ended up messing things up. The Best tutorial out there is of course on digital ocean for my setup on ubuntu 16.04 , this articles isn't so much about getting setup. I just wanted to include it because I found myself looking back at it to see if I missed a step. Bugs/Problems Now comes all the messed up things I found and had to deal with. firewall - Port seems to be open, but connection refused try using 0.0.0.0 in the bin_ip for the /etc/mongo.conf try sudo ufw disable and check to see if it works without the firewall (probably still wont but worth a shot), then just sudo ufw enable. Make sure port 27017 is open. Then check this  linux - Connection refused to MongoDB e...
Read more

Installing Windows on Acer chromebook 15 cb3-532

Much of my notes are just small problems I faced (what's new!) haha. Either way this was a really fun project to do for a friend and I got excited once I research all the information. You can find instruction videos but what helped me the most was the coolstar website. Sadly they don't have parameters in the url so I can link the machine I used in this article. -  https://coolstar.org/chromebook/windows-install.html Scared of messing with the bios? Don't be!    So you need to first remove the screw to put a legacy bios on the machine. Its easy to backup the chrome os one and with the same steps to change it. You can go back in and revert to the chrome version bios. Its that last part that had me worried and its very easy to do.  Problems getting into Developer Mode?   Follow the instructions to the letter, during the step to disable the OS verification, press Ctrl-D to boot into os. If you hear two beeps I would go back to set one. I got in a loo...
Read more