Lessons learnt from SG 4D server glitch

Published on September 19, 2011

Last week, I upgraded Google’s App Engine to version 1.5.2.

I was careless and did not perform enough tests, and it turned out there is a bug with App Engine SDK 1.5.2 (I believe same with 1.5.0). There is a memcache error due to missing library.

If you encounter a “javax/cache/CacheException”, then you are probably having the same problem. The fix is simply upgrading to the latest version at 1.5.4.

Over that 2 days where the bug infested the SG 4D app, I observed a few interesting behaviors. Here’s what I leant.

Lesson 1: Ads revenue increased 10X

As the app could not display the latest results, users has nothing better to do, so they click the ads.. That resulted in 10X more clicks than usual!

An evil thought: Introduce glitches to drive ads revenue (just kidding!)

Lesson 2: In-app-purchase increased 5X

The in-app-purchase to unlock the premium features increased too! Users thought that the app has “expired”, hence they can’t see the results, hence the purchase.

(this hypothesis is true, and was validated by a couple of emails from users)

Lesson 3: Other 4D apps rise quickly

The effect is immediate. On the first day that my app did not work, users downloaded alternatives. There are 2 other 4D apps, and they were both propelled to top 50!

Surprisingly, my downloads remain the same. That means my downloads are constantly contributed by new users, and is unaffected by the bug.

Lesson 4: Test, test, test!

Remember to test during software upgrade, even when it comes from Google!