Extra Cookies

Yet Another Programmer's Blog

Get Ranking and Reviews From AppStore

Ranking

App ranking informations can be get from iTunes RSS feeds provided by Apple, currently, at most Top 300 are listed in each feed.

For example, if I want the top 300 grossing apps in “Games” category, after input my requirements, I get the feed URL as follows:

https://itunes.apple.com/us/rss/topgrossingapplications/limit=300/genre=6014/xml

Using python, I chose feedparser as an RSS parser.

1
2
3
4
5
6

feed = feedparser.parse(
    'https://itunes.apple.com/us/rss/topgrossingapplications/limit=300/genre=6014/xml')
rank = 0
for entry in feed.entries:
    rank+=1
    print str(rank), entry['title']

Then, we get the ranking list:

1
2
3
4
5
6
7
8
9
10
11

1 Candy Crush Saga ® - King.com Limited
2 Clash of Clans - Supercell
3 MARVEL War of Heroes - Mobage, Inc.
4 Hay Day - Supercell

...

297 Overkill 2 - Craneballs Studios LLC
298 Earn to Die - Not Doppler
299 Extreme Road Trip 2 - Roofdog Games
300 Lords & Knights - Medieval Strategy MMO - XYRALITY GmbH

Reviews

In the iTunes RSS feeds, we can’t find how to get the App reviews, but actually, it has, someone made a little guess.

For example, one could get App’s most recent reviews from below feed, by specifying the App ID (535886823) and store country:

https://itunes.apple.com/us/rss/customerreviews/id=535886823/sortBy=mostRecent/xml

1
2
3
4
5
6
7
8
9
10

feed = feedparser.parse(
    'https://itunes.apple.com/us/rss/customerreviews/id=535886823/sortBy=mostRecent/xml')
rank = 0
for entry in feed.entries[1:]:
    rank += 1
    print '****** review %d ******' % rank
    print 'title: %s' % entry['title']
    print 'content: %s' % entry['content'][0]['value']
    print 'author: %s' % entry['author']
    print 'rating: %s' % entry['im_rating']

Pay attention, the first entry of this feed is the App description, not reviews.

Result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

****** review 1 ******
title: Ahhhhhh
content: Ahhhhhh
author: Andrew288195Az
rating: 1
****** review 2 ******
title: Perfect except
content: I still have to switch to safari to "Ctrl+F" a web page.

Seriously?
author: SarahTimmins
rating: 1
****** review 3 ******
title: Crashes too much
content: Title pretty much says it all.
author: RoboWarriorSr
rating: 2
****** review 4 ******
title: My penis is big
content: I have a penis of 1 feet
author: Jiackfabri
rating: 5

...

****** review 49 ******
title: Works great!
content: I was skeptical but I actually really like this browser. Good job Googlé.
author: eSquish
rating: 5
****** review 50 ******
title: Amazing
content: This app/browser is fantastic! It runs fast, and efficient it has shown me the awesomeness that is google and convinced me to buy a chromebook
author: Seandog247
rating: 5

We can only get 50 reviews, but the feeds updated more frequently than the ranking feeds.

There are four types of reviews list,

1
2
3
4
5
6
7
8

Most Recent
https://itunes.apple.com/us/rss/customerreviews/id=535886823/sortBy=mostRecent/xml

Most Helpful
https://itunes.apple.com/us/rss/customerreviews/id=535886823/sortBy=mostHelpful/xml

Most Favorable
Most Critical

I couldn’t figure out what the feeds URL are of last two types, if anyone knows, please leave a comment, and I’ll update this post.

Code

App Lifetime Measurement on iOS & Android

I was working on a mobile app analytics system recent months, how well the system works relies on the app usage data captured on the users’ devices.

This post is about technically how I did it in both iOS and Android operating systems.

Making Thread

When a user opens an app, an open session event needs to be captured, when a user tap the home button, app transitions to the background, the session needs to be marked as closed, when a user view an important page, or click a fancy button, counters need to be increased. Those needs must be handled in background threads since I don’t want my app UI freezing by some time-consuming “needs”.

In iOS, Grand Central Dispatch (GCD) is amazing, create a serial dispatch queue, and a background thread is alive.

1
2

dispatch_queue_t dispatchQueue =
       dispatch_queue_create("com.company.app", DISPATCH_QUEUE_SERIAL);

Every time a user activity need to be tracked, a task is added to this queue as follows, and all tasks will be executed sequentially.

1
2
3
4
5
6
7

- (void)open
{
    dispatch_async(self.dispatchQueue, ^{
        // task to mark session as open
        [self openSession];
    });
}

Android is not cool, I chose to use Handler.

A Handler allows you to send and process Message and Runnable objects associated with a thread’s MessageQueue.

In the first place, design a handler class SessionHandler inherits Handler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

public class SessionHandler extends Handler {
    public static final int MESSAGE_OPEN = 0;
    public static final int MESSAGE_CLOSE = 1;

    public SessionHandler(final Context context, Looper looper) {
        super(looper);
    }

    public void handleMessage(final Message msg) {
        super.handleMessage(msg);
        switch (msg.what) {
            case MESSAGE_OPEN:
                this.open();
                break;
            case MESSAGE_CLOSE:
                this.close();
                break;
            default:
                Log.i(SessionHandler.class.getName(), "Can't handle this message");
        }
    }

Then, create a HandlerThread.

1
2
3
4
5
6
7
8
9

private static final HandlerThread sessionHandlerThread =
        makeHandlerThread(SessionHandler.class.getSimpleName());

static HandlerThread makeHandlerThread(final String name) {
    final HandlerThread thread = new HandlerThread(name,
            android.os.Process.THREAD_PRIORITY_BACKGROUND);
    thread.start();
    return thread;
}

Third, create a Handler instance using that handler thread.

1
2

Handler sessionHandler =
    new SessionHandler(context, sessionHandlerThread.getLooper());

Finally, every time we need to track something, just send a message to the Handler instance.

1
2

sessionHandler.handleMessage(
    sessionHandler.obtainMessage(SessionHandler.MESSAGE_CLOSE));

Binding System Event

If these tracking methods are provided as an SDK, app developers may not want to add those tracking methods one by one in their code base. So, the interfaces of this kind of SDK should be minimum. There are things can be done automatically.

As in iOS, we could bind method calls to system events, for example, when app transitions to background or foreground.

1
2
3
4
5
6
7
8
9
10
11
12
13

NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserverForName:UIApplicationDidEnterBackgroundNotification
                    object:nil
                     queue:[NSOperationQueue mainQueue]
                usingBlock:^(NSNotification *notification) {
                    [self close];
                }];
[center addObserverForName:UIApplicationWillEnterForegroundNotification
                    object:nil
                     queue:[NSOperationQueue mainQueue]
                usingBlock:^(NSNotification *notification) {
                    [self resume];
                }];

I added below code to main method in SDK, app developers only need to call this main method in their code base, believe me, they will love your SDK.

In Android, sadly, I couldn’t find anything similar. It does provide system event bindings in Application, but only onCreate and onTerminate, useless for my needs. So I gave up, and looked for alternatives in Activity, maybe Application.ActivityLifecycleCallbacks could be helpful, but API level (14) is too high for me.

Sending Data

Process

The data captured are on the users’ devices, they should be transferred to server for analysis.

Although the data is anonymous, it’s about users’ behavior, encryption is needed to keep it safe during transfer.

The target devices often connect to cellular network, it’s much expensive than WiFi, compression is needed to keep the data size as small as possible.

For encryption, I implemented two ways, AES and RSA, RSA is more secure, but AES is easy to implement and use. Below are AES implementation in iOS and Android.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

+ (NSData *)encryptAES128Data:(NSData *)data withKey:(NSData *)key
{
    Byte *keyByte = (Byte *)[key bytes];
    if( [key length] != kCCKeySizeAES128){
        return nil;
    }

    Byte *valueByte = (Byte *)[data bytes];
    int valueLength = [data length];
    int bufferSize = valueLength + kCCBlockSizeAES128;
    void *buffer = malloc(bufferSize);
    size_t numBytesEncrypted = 0;
    CCCryptorStatus cryptStatus = CCCrypt(kCCEncrypt, kCCAlgorithmAES128,
                                          kCCOptionPKCS7Padding | kCCOptionECBMode,
                                          keyByte, kCCKeySizeAES128,
                                          NULL,
                                          valueByte, valueLength,
                                          buffer, bufferSize,
                                          &numBytesEncrypted);
    if (cryptStatus == kCCSuccess) {
        return [NSData dataWithBytesNoCopy:buffer length:numBytesEncrypted];
    }
    free(buffer);
    return nil;
}

The Java way is more elegant.

1
2
3
4
5
6
7
8
9
10
11
12

static final String AES128_ECB_KEY = "ampassword";
static byte[] encryptData(byte[] data) {
    try {
        SecretKeySpec secretKey =
                new SecretKeySpec(AES128_ECB_KEY.getBytes(), "AES");
        Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding");
        cipher.init(Cipher.ENCRYPT_MODE, secretKey);
        return cipher.doFinal(data);
    } catch (Exception e) {
        return null;
    }
}

For compression, gzip is enough.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

- (NSData *)gzipDeflateData:(NSData *)data
{
	if ([data length] == 0) return data;
	z_stream strm;
	strm.zalloc = Z_NULL;
	strm.zfree = Z_NULL;
	strm.opaque = Z_NULL;
	strm.total_out = 0;
	strm.next_in=(Bytef *)[data bytes];
	strm.avail_in = [data length];

	if (deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED, (15+16), 8, Z_DEFAULT_STRATEGY) != Z_OK) return nil;
	NSMutableData *compressed = [NSMutableData dataWithLength:16384];  // 16K chunks for expansion
	do {
		if (strm.total_out >= [compressed length])
			[compressed increaseLengthBy: 16384];
		strm.next_out = [compressed mutableBytes] + strm.total_out;
		strm.avail_out = [compressed length] - strm.total_out;
		deflate(&strm, Z_FINISH);
	} while (strm.avail_out == 0);
	deflateEnd(&strm);
	[compressed setLength: strm.total_out];
	return [NSData dataWithData:compressed];
}

Still, I like the Java way.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

static byte[] gzipDeflatedData(String jsonStr) {
    GZIPOutputStream gzipOutputStream = null;
    try {
        byte[] data = jsonStr.getBytes("UTF-8");
        final ByteArrayOutputStream byteArrayOutputStream =
                new ByteArrayOutputStream(data.length);
        gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream);
        gzipOutputStream.write(data);
        gzipOutputStream.finish();
        gzipOutputStream.flush();
        return byteArrayOutputStream.toByteArray();
    } catch (final UnsupportedEncodingException e) {
        return null;
    } catch (final IOException e) {
        return null;
    } finally {
        if (null != gzipOutputStream) {
            try {
                gzipOutputStream.close();
            } catch (final Exception e) {
            }
        }
    }
}

Transfer

In iOS, I wrapped sendSynchronousRequest of NSURLConnection to a dispatch queue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

NSMutableURLRequest *request =
    [NSMutableURLRequest requestWithURL:serverUrl
                            cachePolicy:NSURLRequestReloadIgnoringCacheData timeoutInterval:60.0];
[request setHTTPMethod:@"POST"];
[request setValue:@"application/x-gzip" forHTTPHeaderField:@"Content-Type"];
[request setValue:@"gzip" forHTTPHeaderField:@"Content-Encoding"];
[request setValue:[NSString stringWithFormat:@"%d", [jsonData length]] forHTTPHeaderField:@"Content-Length"];
[request setHTTPBody:dataEncrypted];

dispatch_group_async(self.dispatchGroup, self.anotherDispatchQueue, ^{
    NSURLResponse *response = nil;
    NSError *responseError = nil;
    NSData *responseData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&responseError];
    NSInteger responseStatusCode = [(NSHTTPURLResponse *)response statusCode];
    if (responseError) {
        // handle failure
    } else {
        if (responseStatusCode == 200) {
            // do things
        } else {
            // handler failure
        }
    }
});

Why I use dispatch group is because I don’t want the uploading task block the tracking tasks, so the task is dispatched to another queue, in many circumstances, we have to wait all the tasks in tracking thread and uploading thread are finished, then do somethings.

For example, if I want to send all the data captured every time app transitions to background, five seconds is not enough sometimes, app will crash and it’s a bad user experience. We can add a long-running task by using beginBackgroundTaskWithExpirationHandler, when all the tasks in this dispatch group are finished, notify the system our long-running task is finished.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

[center addObserverForName:UIApplicationDidEnterBackgroundNotification
                    object:nil
                     queue:[NSOperationQueue mainQueue]
                usingBlock:^(NSNotification *notification) {
        UIApplication *application = [UIApplication sharedApplication];
        __block UIBackgroundTaskIdentifier taskId = [application beginBackgroundTaskWithExpirationHandler:^{
                Dispatch_async(dispatch_get_main_queue(), ^{
                        // If task is overtime, end it
                        if (taskId != UIBackgroundTaskInvalid) {
                            // Failed to finish background task, cleanup
                            [application endBackgroundTask:taskId];
                            taskId = UIBackgroundTaskInvalid;
                        }
                    });
            }];

        [self close];
        [self upload];

        dispatch_group_notify(self.dispatchGroup, dispatch_get_main_queue(), ^{
                // Finished executing background task
                if (taskId != UIBackgroundTaskInvalid) {
                    [application endBackgroundTask:taskId];
                    taskId = UIBackgroundTaskInvalid;
                }
            });
    }];

In Android, as I did before, create another handler UploadHandler in SessionHandler for the uploading tasks, HttpClient can be used to send the data to server via HTTP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

final HttpClient httpClient = new DefaultHttpClient();
final HttpPost httpPost = new HttpPost(url);
httpPost.addHeader("Content-Type", "application/x-gzip");
httpPost.addHeader("Content-Encoding", "gzip");
httpPost.setEntity(new ByteArrayEntity(dataEncrypted));
try {
    final HttpResponse response = httpClient.execute(httpPost);
    final int statusCode = response.getStatusLine().getStatusCode();
    if (statusCode == 200) {
        // do things
    } else {
        // handle failure
    }
} catch (Exception e) {
    Log.w(TAG, "Error occured during data sending, abort reason: " +
            e.getMessage());
    // handle failure
}

User Identification

Each iOS device has a Unique Device Identifier (UDID), it could be used to identify the app user, but Apple Has Started Rejecting Apps That Access UDIDs.

In iOS 6, Apple provides ADID, we could use it to identify people, but app user can switch it off if they don’t want to be tracked.

I use OpenUDID and ADID to identify app user, what OpenUDID does is generating a UUID and caching it for later look up.

In Android, it has IMEI, ANDROID_ID, and mac address, none of them is reliable, so I hash those three with device model use SHA to identify the app user.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

// Android UUID known to be duplicated across many devices due to manufacturer bugs
public static final String INVALID_ANDROID_ID = "9774d56d682e549c";

public static String getUDID(Context context) {
    String imei = "";
    if (context.getPackageManager().checkPermission(Manifest.permission.READ_PHONE_STATE,
            context.getPackageName()) == PackageManager.PERMISSION_GRANTED) {
        TelephonyManager tm = (TelephonyManager)context.getSystemService(Context.TELEPHONY_SERVICE);
        imei = tm.getDeviceId();
    }
    String androidId = Settings.Secure.getString(context.getContentResolver(),
            android.provider.Settings.Secure.ANDROID_ID);
    if (androidId == null || androidId.toLowerCase().equals(INVALID_ANDROID_ID)) {
        androidId = "";
    }
    String macAddr = (getMacAddr(context) == null) ? "" : getMacAddr(context);
    return SHA(imei + androidId + macAddr + Build.MODEL);
}

public static String getMacAddr(Context context) {
    String macAddr = null;
    if (context.getPackageManager().checkPermission(Manifest.permission.ACCESS_WIFI_STATE,
            context.getPackageName()) == PackageManager.PERMISSION_GRANTED) {
        final WifiManager manager = (WifiManager)context.getSystemService(Context.WIFI_SERVICE);
        final WifiInfo info = manager.getConnectionInfo();
        if (null != info) {
            macAddr = info.getMacAddress();
        }
    }
    else {
        // no permission
    }
    return macAddr;
}

static String SHA(final String toConvert) {
    try {
        final MessageDigest md = MessageDigest.getInstance("SHA");
        final byte[] digest = md.digest(toConvert.getBytes("UTF-8"));
        final BigInteger hashedNumber = new BigInteger(1, digest);
        return hashedNumber.toString(16);
    } catch (final NoSuchAlgorithmException e) {
        throw new RuntimeException(e);
    } catch (final UnsupportedEncodingException e) {
        throw new RuntimeException(e);
    }
}

Sharing My GTD Notes

Last week, I read the old book Get Things Done again, it did make me think. I read it first two years ago, at that time, I didn’t feel the benefits since I could manage all stuff in my mind. But now, I can’t, I do forget things, I do procrastinate things. During my reread, I kept asking me, why I just missed this treasure, so silly I was.

Below are my reading notes.

Stuff: anything you have allowed into your psychological or physical world that doesn’t belong where it is, but for which you haven’t yet determined the desired outcome and the next action step.

We need to transform all the “stuff” we’re trying to organize into actionable stuff we need to do.

Five-stage methods:

  • Collect things that command our attention
  • Process what they mean and what to do about them
  • Organize the results
  • Review as options for what we choose to
  • Do

Collect

In Box requirements:

  • Every open loop must be in your collection system and out of your mind
  • You must have as few collection buckets as you can get by with
  • You must empty their regularly

Process - Organize

Process guidelines:

  • Process the top item first
  • Process one item at a time
  • Never put anything back into “In Box”

gtd_chart

Review

Review your lists as often as you need to, to get them off your mind.

Review Flow: Calendar - Next Actions - Projects - Waiting for - Someday/Maybe

Weekly Review:

  • Gather and process all your “stuff”
  • Review your system
  • Update your lists
  • Get clean, clear, current, and complete

Do

Model for choosing actions:

  • Context (I prefer Context over Project)
  • Time available
  • Energy available
  • Priority

Model for evaluating daily work

  • Doing predefined work
  • Doing work as it shows up
  • Defining your work

Model for reviewing your own work

  • Life
  • Three-to five-year vision
  • One-to two-year goals
  • Areas of responsibility
  • Current projects
  • Current actions

Project Planning

Planning Steps:

  • Defining purpose and principles
  • Outcome visioning
  • Brainstorming
  • Organizing
  • Identifying next actions

If you’re waiting to have a good idea before you have any ideas, you won’t have many ideas.

Benefits of asking “why?”:

  • It defines success
  • It creates decision-making criteria
  • It aligns resources
  • It motivates
  • It clarifies focus
  • It expands options

Developing a vision:

  • View the project from beyond the completion date
  • Envision “wild success”
  • Capture features, aspects, qualities you imagine in place

Brainstorming Keys:

  • Don’t judge, challenge, evaluate, or criticize
  • Go for quantity, not quality
  • Put analysis and organization in the background

Organizing steps:

  • Identify the significance pieces
  • Sort by (one or more): components, sequences, priorities, details to the required degree

End of 2012

Luckily, our planet survived from the “2012”.

Like last year, I feel 2012 is much shorter than previous years.

During this year, I worked on things that I had to but thought stupid, things that I did and deeply enjoyed, and things that I had waited long and finally did it.

I almost always work alone, I could freely choose languages and tools to work on, e.g. use Java for web crawlers, python to run hadoop streaming map-reduce, Objective C for iOS programs, and even use Intellij IDEA to write Android programs. I really did enjoy it!

Last year, I said I like agile. If I’m working on NASA, I might probably bear the waterfall shit, but, now I found out that it’s even worse when you do waterfall, but don’t believe you are doing waterfall.

I also met a guy, maybe you are familiar too. Some day he comes, and ask for why you didn’t do some x in y, you answered, it is because of z, and let p in will make it …, like always, he didn’t let you finish, said he get it. The truth is, you had no idea of what the fuck you were talking about. Then the question is, why “people” (might including me) asks questions that he don’t know what’s right or wrong and pretend he knows, I believe it’s a worldwide “hard” question.

Last year, I wrote 22 blogs, and this year is 17, I still can’t get rid of the procrastination syndrome, often had some ideas to write, but thought maybe wait two or three days, and then never. Writing blog posts is not only about sharing knowledge, but also organizing thoughts, refactoring knowledge, and learning more, which is the most important part.

I never thought that the imperfection of a product turns to be a great benefit. Kindle, and its super slow touch response, letting me concentrate on one only thing, READ.

Books I’ve read this year:

Books I am reading:

Hope next year, not only read more books, but also write more book reviews.

The two programming languages I’ve learned this year are: Ruby & Scheme. What about next year? Maybe go and erlang.

At the end of this year, the communist party of China issued new rules on internet control, worse than ever. I don’t like the government, the super rich princelings, the polluted air, the house price …, I thought about leave, but I don’t think I will, at the end of this year, I am seriously considering.