Extra Cookies

Start Blogging With Octopress

Posted on 2012-02-12

Old days

All files of Wordpress were in server side, whenever I want to make changes, I had to ssh to my remote machine.

I use a Linode VPS shared by some friends, sometimes, tragedies like "failed to start-up after reboot", "system needs to be re-installed" happened. I didn't want to lose any file, so, I relied on git, after changes were made directly on server, git pull on my Macbook Pro would get all things backed up locally.

To write posts, I used Wordpress editor at first, but it sucked, I like leaving one blank line between every paragraph, it couldn't.

I tried org-mode by using "org2blog" package, since I use Emacs a lot, it was an amazing writing experience. But when my blog post became long, and with lots of code snippets, problem occurred, Emacs ate up all CPU resources making system hung during org2blog-post-buffer occasionally.

Talk about Emacs

Posted on 2012-02-05

Using Emacs is like an adventure.

I found two very powerful extensions recently in Github. Yes, I’ve shared some powerful features/extensions last year by this post, but those two are not only powerful, but also handy, compared with align-regex, occur, anything, etc.

mark-multiple.el

I’ve used Eclipse, which has a good support for refactoring, but I want complain: Dialogs suck!

So, when I need to rename one variable or a function which occurs many times in a file, what am I supposed to do?

Hadoop RPC – Client (2)

Posted on 2011-12-29

What does org.apache.hadoop.ipc.Client do to help the real RPC client to make a remote method invocation on RPC server?

If you don’t know, or forget, please read my last post.

Firstly, create a Call instance, passing the param to it, which contains method name and parameters (class type and instances).

Then, get a client-server connection.

In the third place, send the Call instance to server using connection.sendParam method.

Hadoop RPC – Client (1)

Posted on 2011-12-27

In last post, I metioned that all RPC client calls will be dispatched to Invoker.invoke() method by the dynamic proxy, mainly work of method invoke are done by client.call.

//** org.apache.hadoop.ipc.RPC.java L218
public Object invoke(Object proxy, Method method, Object[] args)
  throws Throwable {
  final boolean logDebug = LOG.isDebugEnabled();
  long startTime = 0;
  if (logDebug) {
    startTime = System.currentTimeMillis();
  }
  //** Request comes to "invoke",
  //** then it calls "client.call",
  //** and get the result "value",
  //** then return it back by "value.get".
  //** The first parameter is an instance of "Invocation"
  ObjectWritable value = (ObjectWritable)
    client.call(new Invocation(method, args), remoteId);
  if (logDebug) {
    long callTime = System.currentTimeMillis() - startTime;
    LOG.debug("Call: " + method.getName() + " " + callTime);
  }
  return value.get();
}

There are two parameters in method client.call, the first one is an instance of Invocation, the second one is remoteId, which is easy to guess that it might be used to represent the connection between client and server.

Before step into client.call to see what are done there, it’s better to know what the Invocation is.

Invocation implements Writable interface, so, let’s look at Writable first.

Hadoop RPC – Dynamic Proxy

Posted on 2011-12-13

This will be the first article of a series of blogs I am about to post, which are all about what I learned from reading the Hadoop source code (cdh3u2).

Code snippets will be full of those posts, to not confuse you, all comments added by me begin with //** instead of // or /*.

An Easy Example of Java Dynamic Proxy

import java.lang.reflect.*;

public class ProxyTester {
    public interface Hello {
        void sayHello();
    }

    public class RealHello implements Hello {
        public void sayHello() {
            System.out.println("Hello, World");
        }
    }

    public class FakeHello implements InvocationHandler {
        private Object agent;
        public FakeHello(Object obj) {
            this.agent = obj;
        }
        //** Hijacking real method invocation
        public Object invoke(Object proxy, Method method, Object[] args)
            throws Throwable {
            System.out.println("Hello Hadoop first");
            //** Real method invocation happens here
            Object result = method.invoke(agent, args);
            return result;
        }
    }

    public void test() {
        RealHello realHello = new RealHello();
        FakeHello fakeHello = new FakeHello(realHello);
        //** "hello" will be a proxy,
        //** whose method invocation will be dispatched to "fakeHello".
        Hello hello = (Hello) Proxy.newProxyInstance(
            Hello.class.getClassLoader(), new Class[] {Hello.class}, fakeHello);
        hello.sayHello();
    }

    public static void main(String[] args) {
        ProxyTester proxyTester = new ProxyTester();
        proxyTester.test();
    }
}

The results:

Kerberos principal 主机名解析问题

Posted on 2011-10-26

在集群安装 Hadoop 的过程中，出现了这样的问题。

所有 Node 都起来了，工作正常，唯独 secondary namenode 在 doCheckpoint 的时候报错，而且是诡异的 403 http error。

// secondary namenode log
2011-10-24 17:09:12,255 INFO org.apache.hadoop.security.UserGroupInformation: Initiating re-login for hadoop/[email protected]
2011-10-24 17:09:22,917 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint: 2011-10-24 17:09:22,918 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.IOException: Server returned HTTP response code: 403 for URL: https://hz169-91.i.site.com:50475/getimage?getimage=1
...

于是怀疑 kerberos 认证问题，可是 secondary namenode 已经通过 Kerberos 验证了；

又怀疑 secondary namenode 向 namenode 请求服务被拒绝，可是 namenode 的 log 显示已经通过验证了。(hadoop/[email protected] 是 secondary namenode 的 kerberos principal，hadoop/[email protected] 是 namenode 的 kerberos principal)