Extra Cookies

-- Taken on April 12, 2014 (flickr)

In last blog post, The job Client has been created and initialized.

This post will discuss on how does Client do to deploy the job to hadoop cluster.

Code snippets will be full of this post, to not confuse you, all comments added by me begin with //** instead of // or /* and the code can be cloned from Apache Git Repository, commit id is 2e01e27e5ba4ece19650484f646fac42596250ce.

The Resource Manager Proxy

1
2
3
4
5
6
7

//** org.apache.hadoop.yarn.applications.distributedshell.Client.java L330
public boolean run() throws IOException, YarnException {

  LOG.info("Running Client");
  yarnClient.start();

  ...

In last blog post, a hadoop distribution is built to run a YARN job.

1
2
3
4

$ bin/hadoop jar share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar \
    org.apache.hadoop.yarn.applications.distributedshell.Client -jar \
    share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar \
    -shell_command 'date' -shell_args "-u" -num_containers 2

The date -u command is executed in Hadoop cluster by above script, we might conclude that there exists a dispatcher named Client in "hadoop-yarn-applications-distributedshell-2.2.0.jar", responsible for deploying a jar to cluster with parameters, such as shell command and args, and notify the cluster to execute the shell command.

To see what's in the rabbit hole, let's step into the Client source code.

Code snippets will be full of this post, to not confuse you, all comments added by me begin with //** instead of // or /* and the code can be cloned from Apache Git Repository, commit id is 2e01e27e5ba4ece19650484f646fac42596250ce.

The Old MapReduce

The Hadoop 0.x MapReduce system composed of JobTracker and TaskTrackers.

The JobTracker is responsible for resource management, tracking resource usage and job life-cycle management, e.g. scheduling job tasks, tracking progress, providing fault-tolerance for tasks.

The TaskTracker is the per-node slave for JobTracker, takes orders from the JobTracker to launch or tear-down tasks, and provides task status information to the JobTracker periodically.

Story

We, programmers built Apps for people to use, sometimes, we could benefit from our users, too.

We could collect anonymous data from users by recording their behaviors on using our App, then analyzing those data, we could find the most favorable features of our App for us to plan for future development, we could uncover some hidden needs of users for us to add new features or create new Apps, we could cluster the users and use different marketing strategy on each users group, etc.

This post will be an example of how I do user clustering.

Imagine I have a music player app, which has 2 millions users.

Last week, a requirement came to me, that we needed to know how often some kind of actions users did in one of our products.

It's temporal, I didn't want to write a mapreduce program, instead, I figured out an hive SQL which is a little complicated (joins 3 tables and a long where clause).

It worked. I got the results, we knew how many times people did one kind of actions on their 1st day, 1st week, 2nd week and 3rd week.

But in the 4th week, I got 0!

It's weird.

I have written about how to use iOS and Android Libraries in Cocos2d-x, in mobile game industry, there is another cross platform framework named unity3d, which is more elegant, powerful, and, super expensive!

This post will show how to enable the App using unity3d framework to use native libraries of iOS and Android.

Unity3d Plugin Mechanism

Unity3d supports plugins well, compared with Cocos2D-X.

In order to use a plugin in unity3d, two things have to be done.

When I began to learn Objective C, I often heard of people talking about "sending message", I was curious, come on, it's just calling a method.

Then I recalled that I had that same feelings when functions become to methods as I began to touch C++.

To support polymorphism, C++ object model has a vtable inside every object which stores instance methods addresses, by which, C++ has a little dynamic characteristic.

But those dynamic are settled after compiling to machine code, unlike ruby or python, which has a virtual machine to dynamically interpreter the behaviors of a program, you might know, the duck typing.

Like C++, Objective C is also compiled to machine code, it supports polymiorphism, but also, to another level, it supports duck typing, you can send messages to any objects even the nil.

Been working on some libs on both iOS and Android platforms for a long time, recently needed to support Cocos2d-x framework, I have no interest to write a whole new lib based on C++, nightmare for maintenance, So I worked on to integrate both libs to Cocos2d-x framework.

iOS

Imagine my lib name is libHumbleAssistant.a, with a header file HumbleAssistant.h.

1
2
3

@interface HumbleAssistant : NSObject
+ (void)doSomething:(NSString *)order;
@end

Lucky me, Objective C belongs to C family, which could integrate with C and C++ seamlessly.