Clojure does it neatly! https://clojure.org/guides/destructuring
This example is taken directly from the book.
|
|
Now I wanted to see how other languages support this.
Scala has powerful pattern matching abilities for destructuring a list.
|
|
Scala’s pattern matching is so powerful that you can go further with it. For example you can find the second last element, or third last element and what ever is inbetween them.
|
|
As you know Ruby has this ability as well!
|
|
|
|
|
|
Python also does it beautifully.
|
|
Javascript also support’s destructuring.
|
|
Are you kidding me? :smile:
If you want kernel version information, use uname(1).
|
|
|
|
Information about the distribution varies depending on the distribution. If your distribution supports Linux Standard Base then we can find that information using lsb_release
. It can also be found from *-release
files in /etc/
directory.
|
|
On Fedora you may need to install lsb_release
.
Ubuntu has two release files /etc/lsb-release
and /etc/os-release
|
|
Fedoro has /etc/fedora-release
, /etc/os-release
, /etc/redhat-release
, /etc/system-release
|
|
|
|
|
|
In order to install a package on HDInsights we need to use Script Actions
Script Action allows you to run a shell script on all nodes of an Azure cluster. You may run the script action when you create the cluster or while the cluster is running.
You may also persist a Script Action in the cluster, so that it is run on every new node that is added to the cluster.
|
|
This script will install and update the given packages in the parameters section.
For example, you may install a new version of Plotly
to use in your Jupyterhub
notebooks.
Your script action must be in a location accessible to the cluster. It could be anywhere, but let’s upload it to storage account.
Upload your Script Action to a Storage Account that your chosen HDInsights have access to.
Bash script URI https://
We can also add this script to HDInsights ARM template in the computeProfile
as scriptActions
.
|
|
__NB: You may create other Script Actions to do any kind of customisations on your HDInsights Cluster_
In this blog ‘m going to talk about installing the Hortonworks Sandbox manually on a Linux VM.
The first time docker is started, it sets up a base device with a default size of 10GB. All future images and containers would be a snapshot of this base device.
Base size is the maximum size that a container/image can grow to. By default docker limit containers to 10G. In Devicemapper, new container/images take zero size and grow up to the maximum size. Changing the base size will not actually change the physical usage of containers unless they grow larger than 10 GB.
You can view this information with the following command
|
|
Since Hortonworks image is greater than 10GB, we need to increase the storage base size. We can do this by changing the storage-driver to overlay and increasing the dm.bazesize.
For this we need to set storag-opt=dm.basesize
and change storage-diver
to overlay
.
Assuming we have Systemd
based setup
|
|
/etc/systemd/system/docker.service.d/docker.conf
|
|
|
|
Here the first line with empty ExecStart
is necessary to clear previous configurations.
Also note that we are increasing the dm.basesize
to 30G
IMPORTANT: Make sure there is enough space in /mnt/docker-data
After editing docker service definition for systemd we need to releoad systemd daemon and restart docker, for the new settings to take effect.
|
|
If you check sudo docker info
, Storage Driver should be overlay
and Docker Root Dir should be /mnt/docker-data
Now we have setup docker to work with Hortonworks Sandbox.
Download the docker image. It’s a huge one 12GB, use high bandwidth network :)
|
|
|
|
|
|
This is a heavy task. Go get your coffee now because it will take some time to load the docker
image.
This step would fail if you did not increase the base device size.
After it is loaded you should see sandbox-hdp
in your docker images
list.
Add the following to /etc/hosts
|
|
|
|
Again this will take some time
Go to http://sandbox.hortonworks.com:8888/
This Sandbox comes with a lot of components like Ambari, Ranger, Hive, Spark etc. installed and configured.
~/.ssh/config
file can be very useful to configure settings for your ssh connections.
For example you may configure default User
and IdentityFile
for a domain using.
|
|
I use trick for configuring multiple identities on Bitbucket server.
|
|
And you need to use personal.bitbucket.org
and work.bitbucket.org
when you want to ssh into them.
Bitbucket has wild card domain redirection, that’s why this works for Bitbucket.
For other domains, you may need to add an entry into /etc/hosts
.
If you have configured your SSH server to use a different port instead of 22
, lets say 2222
|
|
You can force Password Authentication by using
|
|
This is my favourite of all.
|
|
This can be very useful for accessing a port behind a firewall, may be for database.
If you are using AWS EMR or Hortworks Sanbdox, then you will need access to more than one port through ssh.
Using -L
command line option can be very lengthy, instead you could use LocalForward
directive in the ~/.ssh/config
|
|
Hortonworks Sandbox is a heavy container, with lots of components running on it. They recommend at least 8GB of RAM.
Open Docker preferences and select the Advanced tab. Increase the dedicated memory to at least 8GB of RAM or more.
Download the docker image. It’s a huge one 12GB, use high bandwidth network :)
|
|
|
|
|
|
This is a heavy task. Go get your coffee now because it will take some time to load the docker
image.
After it is loaded you should see sandbox-hdp
in your docker images
list.
Add the following to /etc/hosts
|
|
|
|
Again this will take some time
Go to http://sandbox.hortonworks.com:8888/
This Sandbox comes with a lot of components like Ambari, Ranger, Hive, Spark etc. installed and configured.
markdown
files with .md extension. I could obviously use my script to change extension of multiple files, but I thought, there should be a way to do this within emacs.
So I took a deep dive into Emacs documentation and the end result was this function:
|
|
The above function will ask the user for the new extension, and then rename the file, buffer and also set the visited file name to the filename with new extension.
And on my Spacemacs, I added this key-binding to that function. “o” is reserved for user defined functions in Spacemacs.
|
|
If you know a better way of doing this, please feel free to add a comment. Any other suggestions welcome as well.
setuid
in Upstart config, because AWS Linux AMI came with 0.6.5
version of Upstart.
AMI Version: amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2 (ami-c51e3eb6)
|
|
|
|
|
|
|
|
This will create a password less user airflow
|
|
|
|
You should be able to view Airflow ui at port 8080
Now let’s use upstart to manage Airflow process and respawning
This Amazon Linux AMI comes with Upstart 0.6.5
, which is very sad. So setuid
and setgid
doesnot work.
|
|
You should be able to view airflow-webserver
in initctl list
|
|
|
|
You can find the process id at /home/airflow/airflow/airflow-webserver.pid
|
|
|
|
This should keep Airflow Scheduler running in the background and respawn it in case of failures.
|
|
However got stuck on the following error - it just says Invalid Parameter
|
|
Turns out this has to do with my region settings.
I’d created SNS in eu-west-1
region (Ireland). But default region on my account was us-west-1
So, clearly that is a conflict.
Solution is easy, I just had to set region in ~/.aws/config
|
|
That’s it!
]]>It is a cellular automation problem created by John Horton Conway, a British mathematician in 1970, not a game you play to win or lose. This problem has a few rules that we need to follow.
These rules are pretty simple. Once we know the number of live neighbours of a given cell, we can determine it’s fate in a step. In this post I’m going to dicuss method to find neighbours of a given cell in a matrix.
In Game of Life, we take a seed grid and eventually get another grid as a output, by applying the rules to each cell.
Game of Life is played in an infinite two dimensional grid. But for the sake of simplicity let us take a 4x4 grid.
0 | 1 | 2 | 3 | |
---|---|---|---|---|
0 | Cell(1) | Cell(1) | Cell(1) | Cell(1) |
1 | Cell(1) | Cell(0) | Cell(1) | Cell(0) |
2 | Cell(0) | Cell(0) | Cell(0) | Cell(0) |
3 | Cell(1) | Cell(0) | Cell(1) | Cell(0) |
1 is alive and 0 is dead
A cell can have a maximum of 8 neighbours depending on it’s position in the grid, horizontally, vertically, or diagonally adjacent to it.
0 | 1 | 2 | 3 | |
---|---|---|---|---|
0 | Cell(1) | Cell(0) | Cell(1) | Cell(1) |
1 | Cell(1) | Cell(0) | Cell(1) | Cell(1) |
2 | Cell(1) | Cell(0) | Cell(1) | Cell(1) |
3 | Cell(0) | Cell(1) | Cell(0) | Cell(0) |
Let’s first represent the above seed grid in Scala.
|
|
Now we have a basic Representation of our models required in Game of Life.
Given the co-ordinates of a cell, we should be able to find list of it’s live neighbours.
|
|
Now this is the important bit.
A neighbour is any cell that is adjacent to a given cell, horizontally, vertically or diagonally. (1,1) is adjacent to (0,0), (0,1), (0,2), (1,0), (1,2), (2,0), (2,1), (2,2).
We will implement findNeighbours
functions.
|
|
Here is an algorithm, I found on StackOverflow, by Seb
http://stackoverflow.com/questions/652106/finding-neighbours-in-a-two-dimensional-array
|
|
Here is a direct implementation of this algorithm in Scala
.
|
|
You end up repeating the shims and paths in multiple locations. We can minimise this by using requirejs.config(config)
method.
requirejs.config() will help you define or override your dependencies configuration.
|
|
Here is an examlpe on how I use requirejs.config
method for minimising repeatation.
First let us define the reusable configuration file. Let’s name it rconfig.js
.
|
|
Here we have defined the paths for all the common dependencies.
Now let us use it in two different places.
|
|
|
|
The file-layout here is like the following.
Hope it helps!
]]>Checkout Cleancoders to know more.
Chances are, you may be already using SOLID principle, just without giving it a name.
S - Single Responsibility Principle (SRP)
O - Open/Closed Principle (OCP)
L - Liskov Substitution Principle (LSP)
I - Interface Segregation Principle (ISP)
D - Dependency Inversion Principle (DIP)
Thats right, SOLID stands for a bunch of other acronyms.
The single responsibility principle states that every class should
have a single responsibility, and that responsibility should be
entirely encapsulated by the class. All its services should be
narrowly aligned with that responsibility.
Fairly simple. A Student class should manipulate Student properties and not the the properties of School. Specification of the class remains mostly untouched unless there is a big change in requirement of your software.
It is good to keep in mind that, there can be some exceptions to this. For example a utility class may provide methods for managing both Students and Schools.
Software entities should be open for extension, but closed for modification.
The idea is that once a class is implemented completely, it should not be modified for including new features. Bugs and error correction must be done as and when required. Adding a new feature will require new specification and release of newer version.
http://en.wikipedia.org/wiki/Open/closed_principle
Objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program. See also design by contract.
That right there is the basic use of Dependency Injection, right? You define a parent interface and replace them by concrete implementation which are derivatives.
|
|
http://en.wikipedia.org/wiki/Liskov_substitution_principle
Many client-specific interfaces are better than one general-purpose interface.
Interfaces provide abstraction no implementation. When developing client interfaces, it should be kept at minimum and should only expose those methods which are essential for that client.
http://en.wikipedia.org/wiki/Interface_segregation_principle
|
|
##D - Dependency Inversion Principle (DIP)
One should “Depend upon Abstractions. Do not depend upon concretions.”
Dependency injection is one method of following this principle.
|
|
I first heard about Closures while developing something in Javascript. If you have used the popular javascript library jQuery, you have already used closures, knowingly or unknowingly.
Here is my attempt to explain Closures, through examples in few programming languages.
From Wikipedia:
In programming languages, a closure (also lexical closure or function
closure) is a function or reference to a function together with a
referencing environment—a table storing a reference to each of the
non-local variables (also called free variables or up values) of that
function. A closure—unlike a plain function pointer—allows a
function to access those non-local variables even when invoked outside
its immediate lexical scope.
What that means:
In closures procedure/method/function contexts become first-class. That means, with closures you can create functions that return functions, although that is only an outcome. An important point to understand here is, the closure methods refer to the context in which it was created, not to the one it was called.
To better understand closures one has to understand a variable’s scope, the best read about that would be understanding javascript varialble scope.
Closures store references to the outer function’s variables; they do not store the actual value. So if we change the value of reference in closure it changes value outside of its scope.
You may implement closures using Anonymous functions, but all anonymous functions need not be a closure, although many of them are.
Closures help us to write more expressive and concise code(once you get a hang of it!). We know objects have a state, using Closure we can give state to functions as well.
Now, let us take a look at examples of how to use closures in a few programming languages.
All of the following examples do the same thing: Create a closure to increment a number by another number.
This would be the easiest to understand code.
|
|
In the Javascript example above, we define a function incrementBy(x)
, which returns a function, that accepts parameter ‘y’ and returns sum of x and y. Here the value of ‘x’ will go into the closure of the returned function and will be accessible whenever the function is invoked.
Note that when calling incrementBy2(4)
our closure remembers the value 2 (i.e ‘x’) that was passed earlier when doing var incrementBy2 = incrementBy(2);
. And when invoking incrementBy2(4)
we are actually passing the value of y
as 4. Hence the statement return x+y
will transform to return 2+4
. Cool right!!?
|
|
The example in Scala is similar to the example in Javascript except for Scala’s awesome one liner syntax.
|
|
There are several ways of doing this is lisp. This is only one way of doing it. Here you are not getting a pointer to the closure function and will probably make it useless. See this link to see how to return functions is Clisp.
|
|
As you can see, the syntax of clojure and lisp are extremely similar.
|
|
The above example in Python is pretty similar to the one in Javascript and is easy to understand. Here we define the closure function increment(y)
and then return it. Just remember to take care of the tabs!
|
|
Ruby can do this is in two ways, using Proc and using lambda. The lambda functions in Ruby are similar to lambda’s in lisp.
So that was examples about closures. Hope that gives you at least some idea about Closures. If you know how do the same in any other languages, please feel free to share. Also checkout the references, they are good reads.
##References:
Thanks to Craig for sharing Go snippet
Reddit thread here: http://redd.it/223b7p
]]>It is available at https://github.com/WarFox/play-spring-data-jpa-backbonejs.
Also a demo app is hosted in Heroku at http://play-spring-data-jpa-backbone.herokuapp.com/
It started as a fork from https://github.com/typesafehub/play-spring-data-jpa and then created a separated repo to include backbonjs.
Feel free to fork and play with it.
]]>http://jsfiddle.net/deepumohanp/73xVu/
Check out https://developers.google.com/youtube/youtube_player_demo, for all availabel configuration and play with it.
For example you can hide the controllers in embedded video, by setting parameter controls
to 0
|
|
vq
.
|
|
modestbranding
to 1.
|
|
|
|
]]>
In Java, we can use the Calendar class to get day_of_month, day_of_week etc.
Last day of a month varies depending on the Month and on leap year we have extra day in February.
So to figure out the last day of any given month in a year, we write some code which is apparently simple.
Calendar object allows us to manipulate days, go forward or backward on the Calendar, add days, hours, minutes or seconds to any given time etc.
We will use these capabilities of Calendar class to get our last_of_month.
The idea here is to get the first day of next month and then reduce one day from it, which gives us the last day of the month relative to input date.
|
|
But, Java Calendar has a better way of doing this, by using getActualMaximum() method which is far more convenient.
|
|
A sample calendar util class which gives you few convenient methods is given below.
|
|