Setting up Apache Airflow on AWS EC2 instance

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

AWS SNS error - Invalid parameter

So I was playing with AWS Lambdas, and wanted to setup SNS notification for that.

1
aws sns publish --topic-arn arn:aws:sns:eu-west-1:22834709971:my-awesome-sns-topic --subject 'Subject' --message 'Hello, hello!'

Game Of Life (in Scala)

I recently came across an interesting problem/game called Game Of Life. A visual version of this is available here. Also, here is a beginners introduction to Game Of Life, or simply Life, written by Alex Bellos at Guardian.

It is a cellular automation problem created by John Horton Conway, a British mathematician in 1970, not a game you play to win or lose. This problem has a few rules that we need to follow.

Rules

  1. A cell in the grid can be dead or alive.
  2. A live cell with less than 2 live neighbours will die.
  3. A live cell with 2 or 3 live neighbours will stay alive.
  4. A live cell with more than 3 live neighbours will die.
  5. A dead cell with exactly 3 neighbours will become alive.

These rules are pretty simple. Once we know the number of live neighbours of a given cell, we can determine it’s fate in a step. In this post I’m going to dicuss method to find neighbours of a given cell in a matrix.

In Game of Life, we take a seed grid and eventually get another grid as a output, by applying the rules to each cell.

A pattern to use RequireJS config in a separate file

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

SOLID design principle

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

Closures

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

Sample app for Play! 2.2, Spring-data-jpa and Backbonejs

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

Backbonejs in jsFiddle.net

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

jQuery :: Populate Select box dynamically (plugin version)

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]

Use Requirejs to load scripts from cdn server like cdnjs

Setting up Airflow on AWS Linux was not direct, because of outdated default packages. For example I had trouble using setuid in Upstart config, because AWS Linux AMI came with 0.6.5 version of Upstart.

AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)

Install gcc, python-devel, and python-setuptools

1
sudo yum install gcc-c++ python-devel python-setuptools

Upgrade pip

1
sudo pip install --upgrade pip

Install airflow using pip

1
sudo /usr/local/bin/pip install airflow[s3, hive, python]