Sankasaurus

For those of you like me that have totally missed the Publishing page, here is how to create a link to your Android application in the Android Market:

market://details?id=<packagename>

OR

http://market.android.com/details?id=<packagename>

So for Remembory, I have this:

market://details?id=com.gbott.remembory

The “old” way was to link to the search page by using this:

market://search?q=pname:<package>

… but the details page method saves the user a click, or a tough decision when there are two applications both called Remembory.

For doing anything serious on Rackspace Cloud, you need to be able to use machines larger than 2Gb of RAM. Problem is, machines larger than 2Gb of RAM cannot be imaged (or backed up) – until now. About a month ago, they announced the ability to snapshot a machine into Cloud Files. Today I decided to take it for a spin and hit Snag number one: there was no way to do an image on my new 16Gb machine. After talking talking to one person at Rackspace Cloud, I was none the wiser – Snag two was the lack of training for their support staff. After asking for the supervisor, he created a ticket for the approval for the terms (letting me know I would be charged for the storage in Cloud Files) and off I went. Until I hit Snag 3 – the “Images” tab for the server details still showed no way of performing an image. I was told to use the “My Server Images” and HORAY I could make an image of my 16Gb machine.

Finally the feature that prevented my company from using Rackspace Cloud, and instead using AWS, was fixed! Congratulations to the Rackspace Cloud team.

Another thing I learnt today is that there are two data centers for Rackspace Cloud machines – DFW and ORD. The first server you provision gets assigned to one of the two data centers, and which ever one it gets put into is the data center that all of your other servers will be put into as well. So if the first server gets put into DFW, then all of the other ones you create will be. That is until you delete all of your servers. Then once again, the first server can get put into any data center.

I am hoping that in the future, we will have a choice about which data center a server is provisioned in, particularly since it will help for geographical distribution.

I have added another open source project to my list – a JSON SerDe for Hive. You can check it out here:

http://code.google.com/p/hive-json-serde/

This SerDe (serializer/deserializer) will let you read JSON files as input for Hive tables. In the future, it will also support writing JSON data, but that is for another day.

Please let me know if you have any comments or questions about it.

My good mate* Joel Spolsky wrote a nice piece about backups (or rather, restoration), and I wanted to echo his remarks and how they relate to using AWS EC2.

If you are using EC2, you will quickly find that if an instance is terminated, any data on that instance is gone – lost forever. At first, this seems like a terrible idea, but in fact, it encourages you to get into best practices, and discover the awesome benefits of EBS.

We have many instances running of different types. We have built a “custom” Debian AMI for each of the instance types we use (web, database, management, etc). If you were to launch an instance with one of these AMIs, you would not have a fully working system. That is because these AMIs have sym-links for important and/or dynamic data. For example, on the web AMI we have created, /etc/apache2, /etc/php5/ and /var/www are all sym-links. To where? A directory that an EBS volume is mounted to. That’s right, all of the web configuration and website code only lives in an EBS volume. It is simple enough to write a little script that creates a nightly Snapshots of each EBS volume.

Now for the power of this setup. Every time you want to bring up another instance of the same type (say, for horizontally scaling), you are in fact doing a restoration from backup. Take a Snapshot (your backup), create an EBS volume, attach it to the new instance, and make it live! This doesn’t just work for scaling, it works for bringing up staging servers that are mirrors of production or running experiments without affecting production.

We can even take it a step further! Those AMIs and Snapshots are all stored in S3 – data available to the whole Region. An instance and EBS volume exist in only 1 of the Availability Zones within that Region. You can use your backups to restore into a new Availability Zone which you can use to create a high-availability solution.

Happy scaling!

* I don’t know Joel personally – we have never met – but I do follow his work, like his company and LOVE Fogbugz!

I would love to not have to install the real Java and Tomcat manually on Debian, but I have little choice in the matter. Take a look at this:


$ apt-get install tomcat5.5
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
ant ant-gcj ant-optional ant-optional-gcj antlr build-essential debhelper
default-jdk default-jre default-jre-headless defoma dpkg-dev ecj ecj-gcj fastjar
file fontconfig fontconfig-config g++ g++-4.3 gappletviewer-4.3 gcj-4.3
gcj-4.3-base gettext gettext-base gij-4.3 gjdoc hicolor-icon-theme html2text
intltool-debian java-common java-gcj-compat java-gcj-compat-dev
java-gcj-compat-headless jsvc libantlr-java libantlr-java-gcj libasound2
libatk1.0-0 libatk1.0-data libbcel-java libcairo2 libcommons-beanutils-java
libcommons-collections-java libcommons-collections3-java libcommons-daemon-java
libcommons-dbcp-java libcommons-digester-java libcommons-el-java
libcommons-launcher-java libcommons-logging-java libcommons-modeler-java
libcommons-pool-java libcompress-raw-zlib-perl libcompress-zlib-perl libcups2
libdatrie0 libdb4.5 libdigest-hmac-perl libdigest-sha1-perl libdirectfb-1.0-0
libecj-java libecj-java-gcj libexpat1 libfile-remove-perl libfontconfig1
libfontenc1 libfreetype6 libgcj-bc libgcj-common libgcj9-0 libgcj9-0-awt
libgcj9-dev libgcj9-jar libgcj9-src libglib2.0-0 libglib2.0-data libgtk2.0-0
libgtk2.0-bin libgtk2.0-common libice6 libio-compress-base-perl
libio-compress-zlib-perl libio-stringy-perl libjaxp1.3-java libjaxp1.3-java-gcj
libjpeg62 liblog4j1.2-java liblog4j1.2-java-gcj libmagic1 libmail-box-perl
libmail-sendmail-perl libmailtools-perl libmime-types-perl libmx4j-java
libobject-realize-later-perl libpango1.0-0 libpango1.0-common libpixman-1-0
libpng12-0 libregexp-java libservlet2.3-java libservlet2.4-java libsm6 libsqlite3-0
libstdc++6-4.3-dev libsys-hostname-long-perl libthai-data libthai0 libtiff4
libtimedate-perl libtomcat5.5-java libts-0.0-0 liburi-perl libuser-identity-perl
libxcb-render-util0 libxcb-render0 libxcomposite1 libxcursor1 libxdamage1
libxerces2-java libxerces2-java-gcj libxfixes3 libxfont1 libxft2 libxi6
libxinerama1 libxrandr2 libxrender1 libxtst6 make mime-support patch po-debconf
python python-central python-minimal python2.5 python2.5-minimal ttf-dejavu
ttf-dejavu-core ttf-dejavu-extra x-ttcidfont-conf xfonts-encodings xfonts-utils
...
0 upgraded, 146 newly installed, 0 to remove and 0 not upgraded.
Need to get 101MB of archives.
After this operation, 288MB of additional disk space will be used.
Do you want to continue [Y/n]?


WTF? I understand that Tomcat needs some kind of Java, but this is ridiculous. It is installing ant, fonts, compilers and worst of all, the most evil Java ever.

Ubuntu has the sense to make Sun Java available, but even if you do have Sun Java installed, the above is true on Ubuntu.

For shame!

I’ll stick to downloading from java.sun.com and tomcat.apache.org.

For some strange reason, Chrome OS is getting a lot of press. Is it a slow news day?

They say that it is direct competition to Microsoft, that it makes Linux less relevant… are they serious? Chrome OS is a non-announcement. There is a project that has existed for over 2 years called “cl33n“. From the creator:

This “OS” is due to the released mid 2010. Is that how slowly things move inside Google? Why would it take them 12 months to create nothing more than cl33n?

What I am trying to say, is that Chrome OS is nothing new. Cl33n is not alone in this space either – other project like Webconverger share my view.

While on the subject of Google’s non-annoucements, did you hear that Gmail, Doc, etc are out of beta. Big news huh? So what is their excuse now for daily “Server error” dialogs?

I can’t believe it, but I won! I have been trying to set up Nagios on a RHEL5 machine running SELinux and have been loosing the fight for the last 3 days. But today, I win! This is such a win, it is worth sharing.

Now that I have won though, I believe this is not Nagios specific at all, and if I had bothered to learn about SELinux, this may have been obvious. Anyway, the error Nagios was giving me was:

As you may have already guess, the solution has nothing to do with the location or permissions of the file, the file was not missing, Nagios was running, and Nagios was checking external commands. The final line of the message is great though, and I can only hope we start to see more old English in error messages.

The problem of course, was that SELinux was enabled and stopping this blatant security violation. You can check to see if SELinux is on by running:

If you got “Permissive” or “Disabled”, then this post is not for you. To see SELinux’s side of things, check out /var/log/messages:

As you can see, SELinux is trying to give you a hint with that sealert bit, so you should take it.

That raw audit message is GOLD! There is some other information in there, but nothing about what the next step should be to create a policy and make it permanent. Using chron I have heard is a temporary fix. The solution is copying that raw audit message into an empty file and running audit2allow to create a policy:

This creates a file call NagiosPing.pp which contains the SELinux policy needed to make these errors go away. The only thing left to do is to install this policy:

If your setup was like mine, SELinux was actually preventing 3 different actions, needing 3 different policies. HA! That is easy now – just repeat the steps until Nagios is doing your bidding.

I have recently been updating and freshening up my resume, and thought I would do something different… something to stand out. Having interviewed many people, from software engineers to product managers and CTOs, I have had the opportunity to see many resumes. Most of the resumes I see are quite bland and boring – as if written for a machine. They have no personality, no passion and no creativity. I wanted my resume to have personality, passion and creativity.

Since I am a well rounded engineer, I thought I would be able to cover more of my skills in a skills-based resume, instead of a chronological list of my work experience. I went through and split my skills into areas (Coding, Data, Operations, Communication, etc) and listed a few points under each. At the end, I added my work experience, but only with a short paragraph of what I did there.

While I thought this style made a good resume, recruiters hate it, and so do people trying to hire.

After getting some advice, I wanted to share it with the rest of the world. Here are the things people want to see in software engineering/technical resume:

• Technical Skills: After your object at the top (if you want to display passion), make a clear section describing your technical skills – languages, databases, environments.
• Summary: Recruiters give your resumes only seconds when determining its fit for a role, so help them out by having a small summary easily telling the reader how many years of experience you have, your last role, and perhaps your education. From this, they can deduce.
• Accomplishments: When you do list the companies you have worked for, make sure to include some of your notable accomplishments and things you are proud of. If possible, try and quantify them (revenue generated, queries per second, amount of data).
• Your Role: Let the reader know what your role and responsibilities were at each position you have held

Resume writing is a complex task, and something a lot of people dread. I have definitely not covered everything, but I did want to make clear a skills resume is not a good idea. Also be aware that many recruiting companies load resumes through a computer before even looking at them. Try to have keywords a search engine could pick up on (like SEO for resumes).

Perhaps resumes written for machines is a good thing after all?

I have successfully unlocked my T-Mobile G1 for free? How you ask? After 90-days, T-Mobile allows you to unlock almost any phone you have purchased from them with the T-Mobile branding, and all for FREE. The process can take about 2 weeks (as it did for me), but it worked. Here is how:

1. Call T-Mobile on 1-877-453-1304 from another phone (not your G1). I got that number from their Contact page
2. Its the standard voice prompt thing, so say English, enter your G1’s phone number, then when they ask what you want help with, say the magic word “Agent”
3. Then they ask what you would like to talk about, and say “SIM Unlock Request”
4. My experience with T-Mobile has been pretty good, within a minute I was talking to the real person. At this point, you verify your identity, and explain to them you want to unlock you phone because you are going overseas and want to use another SIM card.
5. They will ask you for you phone’s IMEI number. This is a sacred number, so be careful who you give it to, as you can report the phone stolen, give them the IMEI number, and have the phone permanently disabled. You will find the number:
   1. on the side of the box your phone came in,
   2. on the G1 itself under Settings -> About phone -> Status, or
   3. by dialing *#06#
6. Give them your email address, and within 14-days, you will get an email from T-Mobile with your unlock code.
7. To unlock the phone, power it off, insert a non-t-mobile SIM card, and power it back on
8. At the prompt, enter the unlock code from the email and you’re done!

The reason for writing this post is I had no idea this could be done for free with any T-Mobile phone after 90-days has past. There is no need to pay $25 to any scam site, just do it legit, for free, and without issue.

Hi have been playing around with Google App Engine for a while now, and wanted to learn Django on a new application I am building. The two articles I started with are:

• Using the Google App Engine Helper for Django
• Using Django 1.0 on App Engine with Zipimport

Both Django and GAE are being developed as I write this, so although these instructions are kind of recent, they are already out of date, or rely on you having knowledge of Django. Since there are a lot of others with no Python or Django experince wanting to learn, I thought I would make a tutorial that works as of today, but who knows a month from now or even tomorrow.

Note: This tutorial is written for Linux. Mac/Windows users will have to translate

Create django.zip

Django comes as a .tar.gz file, but we want a zip file to take advantage of the Zipimport library, so some conversion is needed.

1. Download the latest Django and untar the file (in this case, it is 1.0.2)
2. cd Django-1.0.2-final
3. Create the zip file, but without the admin section since App Enging supplies it’s own
   zip -r ~/django.zip django -x ‘django/contrib/admin*’

Get the Helper

The App Engine Helper or Django is an open source bootstrapper for getting Django started on App Engine. Downloading it from the website will currently give you quite an old version (r52) which will not work in this tutorial. Instead, use subversion to get the latest (r74).

1. cd ~
2. svn export http://google-app-engine-django.googlecode.com/svn/trunk/ gae-django-tutorial
3. cd gae-django-tutorial
4. mv ~/django.zip .

Setup App Engine

As you would with any GAE application, edit the app.yaml file to refer to your application ID. The Helper also needs to know where your Google App Engine SDK is, since it is going to change how you start the development server, so create a link to it:

1. cd gae-django-tutorial
2. ln -s /path/to/google_appengine .google_appengine

Start the development server

Django has a different way of running the development server. Instead of using dev_appserver.py to start the dev server, do the following:

1. python manage.py runserver

If everything is running correctly, you should see something like:

INFO:root:Server: appengine.google.com
INFO:root:Checking for updates to the SDK.
INFO:root:The SDK is up to date.
INFO:root:Running application google-app-engine-django on port 8000: http://localhost:8000

However, if you are like me and saw an error like the following:

ImportError: No module named antlr3

Then you will need to install the antlr3 python module. Luckily this is easy.

1. Go to http://www.antlr.org/download/Python and download the latest zip
2. unzip antlr_python_runtime-3.1.zip
3. cd antlr_python_runtime-3.1
4. sudo python setup.py install

Lets see the site! When you go to http://localhost:8000/ you should see a page saying “It worked! Congratulations on your first Django-powered page.”
Pretty (un)impressive huh?

Ok, now lets start doing something. Kill the server by pressing Ctrl-C. The Django tutorial is the next stop, which involves creating the Polls Django app. You can read through there to get a full understanding. For simplicity, I am only what I did to get it working.

1. python manage.py startapp polls
2. cd polls
3. Edit models.py

   from appengine_django.models import BaseModel
   from google.appengine.ext import db
   
   class Poll(BaseModel):
     question = db.StringProperty()
     pub_date = db.DateTimeProperty('date published')
   
   class Choice(BaseModel):
     poll = db.ReferenceProperty(Poll)
     choice = db.StringProperty()
     votes = db.IntegerProperty()

4. Edit views.py (notice that order_by() from the Django tutorial is just order() here)

   from django.template import Context, loader
   from polls.models import Poll
   from django.http import HttpResponse
   
   def index(request):
     latest_poll_list = Poll.objects.all().order('-pub_date')[:5]
     t = loader.get_template('polls/index.html')
     c = Context({
       'latest_poll_list': latest_poll_list,
     })
     return HttpResponse(t.render(c))

5. cd ..
6. Edit urls.py and add to urlpatterns
   (r’^polls/’, ‘polls.views.index’),
7. mkdir -p templates/polls
8. cd templates/polls
9. Create index.html

   {% if latest_poll_list %}
   <ul>
   {% for poll in latest_poll_list %}
     <li>{{ poll.question }}</li>
   {% endfor %}
   </ul>
   {% else %}
     <p>No polls are available.</p>
   {% endif %}

10. cd ../..
11. Edit settings.py and add to INSTALLED_APPS
    ‘polls’,
12. Now you can run the dev server again:
    python manage.py runserver
13. And go to:

    http://localhost:8000/polls/

14. You should see the message “No polls are available.”

Adding some data

You can go through the rest of the Django Tutorial to fill out the rest of the views. One last thing to know is the admin site. To view it, go to:

1. http://localhost:8000/_ah/admin

Notice that the Datastore Viewer is empty – it doesn’t even know about our Poll or Choice Models. Don’t panic. Go back to your terminal, and terminate the dev server. Now run:

1. python manage.py shell

This brings up a special python shell with the Django environment set up for you. Now you can run:
>>> from polls.models import Poll, Choice
>>> import datetime
>>> p = Poll(question=”What’s up?”, pub_date=datetime.datetime.now())
>>> p.save()

You have just created a new Poll. To end the shell, press Ctrl-D.

If you start the dev server again (or if you never stopped it in the first place), you should be able to go to:
http://localhost:8000/polls/
to see you new Poll, and go to the admin site:
http://localhost:8000/_ah/admin/datastore
to see and create new Polls

Easy huh?