Category: Software Engineering

Coding, Python, web development, architecture, and deployment

  • First Experience Scaling Web Apps on EC2

    Last week I had a bit of a crisis situation with my apps.  After several mornings of waking up to find the server not responsive and 700+ emails waiting in my inbox something had to be done.

    When starting out I leaned towards keeping the costs as low as possible.  That meant putting several virtual hosts (apache) and MySQL on the same server.  But as the traffic for the site grew I started getting more and more crash situations.  It took a while to diagnose but evenutally I figured out that HTTP requests would come in and eventually use up all the memory on the server at which point MySQL would crash and restart.  In the time it took MySQL to restart I would end up with several hundred error emails from failed requests.  Fun.

    After some struggling I decided to launch another instance to host my most taxing web application.  Thus isolating the memory requirements of apache from MySQL.

    Setting the new server up behind a load balancer will make future scaling much easier.

    Doing something similar without the flexibility of Amazon’s AWS would have been a much bigger nightmare.

    Luckily now that the apps are making decent money paying for the extra server usage is easy to justify.

  • Writing another (short) book

    I have a book available in for the kindle that I wrote several years ago. It’s been doing ok for sales considering that I have never really done anything to promote it.

    This past week I have been following someone who is selling a course on kindle publishing and I learned a few things from watching his examples. It’s got me a bit charged up to get another short book written up and published on kindle.

    Over the last couple days I hammered out the first 4500 words of my goal of 10,000 words about brewing beer. Hopefully I can pull this together quickly and get it up for sale by the end of September.

    After this one I already have an idea of what I want to write next.

    It’s strange that I never considered myself a good student in English class at school and yet I’ve managed to keep this blog going for nearly a decade, have one book published on Amazon.com, a second one now 50% completed and a third book in the research stage.

  • Using tmux and Vim

    Working on the same projects from multiple computers can be difficult.  There are lots of solutions for this problem such as using Dropbox to sync files between computers. but that leaves your workspace to start from scratch ie when working from the office last I had these 10 files open, and a couple of things running, then when I get home if the files sync over I have to reopen those files locally and configure that computer for anything else that needs to run to continue debugging.

    Enter Tmux and Vim.

    Lately I’ve been working 3 days at home, 2 days at the office each week.  As a result I have re-evaluated my workflow to find something a bit more efficient.  Vim in the terminal is brilliant once you’ve got it set up and have mastered it.  It has been my main editor for about 3 years now.  There are a ton of great resources out there to learn Vim for those of you still trying to find the way.

    Tmux is new to me though.  It’s basically a better version of screen.  If your not familar it basically provides a way to do 2 main things.   First it allows you to create a persistent session that you can connect to and leave running.  Secondly in this session you can open up and layout many different consoles into tabs or windows. It’s sort of a command line version of VNC.

    I have tmux installed on my Linux servers on AWS so that I can work efficiently on those machines, and I have it installed on my workstation at the office.  I can log into the environment with SSH and continue right where I left off even if I’m switching between my laptop and a friends computer.

    With Vim I have the ability to keep my settings and plugins saved and configured the way I like them with a configurations version controlled on github.  Getting a new Linux or Mac machine up and running takes very little time to do with this setup.

    I’ll give my best tips and tricks for Vim in another post.

  • Write a lot of Code

    I write a lot of code.  I love it.  The challenge and creativity of solving problems along with the hope that the right program could save either myself or many other people their valuable time or provide them something better to do with their time is a great reward.

    There are about 5 software projects that I’m in the process of working on.  All of them have me excited.

    One interesting thing I’ve noticed is just how much more difficult it is to code when working on a project in a team.  On my own personal projects I have no problems with adding or removing 500 lines of code in a day because I have a pretty good understanding of how everything works.  With a team project there are more moving pieces and therefore every change requires second guessing, searching for usages and being vigilant to validate arguments and document more thoroughly.

    Skipping the boiler plate code and coding as much as possible as quickly as possible gets me to the real challenges quicker, forces me to learn things at a more rapid pace, and arrive at a functional (but perhaps more buggy) product sooner.

    Learning new things and always trying to practice to gain mastery in my craft is a never ending journey.  Putting fingers to keyboard to solve problems is an addiction I can’t shake.

    Tomorrows Goal: write more code.

  • The singularity

    In the last few days I read two books. I don’t read very many fiction books but these got me thinking.

    The singularity is a term taken from physics. In the world of physics a singularity is an object or event in space where the laws of normal physics break and thing behave in wild and unpredictable ways. The term has been taken over to the world of technology to be the point in time when computers become intelligent enough to improve themselves. Once there is a computer AI as smart as a human it will not take long for it to start coding on itself and eventually becoming 2x, 5x, 100x smarter than anyone on the planet.

    When this happens (not if) the world will change dramatically very quickly. But whether or not it will be good or bad is unknown. Any futurist cannot be certain of anything afterwards.

    Will people have jobs if a computer is capable of everything we can do? With mass unemployment as factories become completely automated, followed by AIs redesigning the products and taking over business management roles will it result in a wealth of free time or crises as people have no money.

    Will removing people from the supply chain bring the prices of goods to zero? Probably not since there will be at least a few thing that will remain scarce such as property, and time. How that will be handled if no one makes any money is anyone’s guess.

    Will an army of automated robots be able to re-forest the Sahara desert and save the planet from climate change?

    It is a fascinating scenario to think about because literally every industry will change. There are so many variables and uncertainties that a thorough discussion would probably fill a 1000 page book.

    The role of science fiction is sometimes to explore these ideas and present them to a wider audience so that culturally we can be somewhat more prepared for what the future may bring. The singularity is something far more likely to happen in the next 30 years than the invention of faster than light travel or transporter pads and yet it has been off the radar for most people.

  • iTunes Download Stats

    Recently I’ve been building out my iPhone App server to provide a business dashboard with all the relevant services and numbers that I care about available at a glance. It avoids me having to sign in and out of many different sites to get the information and makes it easier to push things together – for example charting both Admob and iAd data on the same graph.

    Thankfully the web is becoming more programmable every week and these things are becoming easier to put together quickly.

    This is a chart I built last night to display the downloads and updates across all my apps for the past 31 days:

    You can see the jumps in downloads that correspond to when I released updates to iTunes.

    With these sorts of things I’m finding that there is a tipping point. If the custom page I have created is only 90% as good as going to the original source then I’ll just opt to login there but once it becomes as good as or better than that you’ll quickly forget about the 10 different logins you needed to get all those numbers.

    Being in charge of it is even better. I use iAd and Admob for advertising and can pull those numbers in and compare them appropriately. On the same page I display data from Apple, Google, Linkshare, as well as numbers I collect myself such as traffic, link clicks and ad impressions. I only have to login to iTunes Connect to release new Apps.

    I will continue open sourcing the components for this system over the next few weeks.

  • django-linkshare

    One of the ways to make money on the itunes store is to sign up for the itunes affiliate program. Linkshare runs the program in the USA and they will give you a 5% commission on all sales that you refer. It works through cookie based tracking that is valid for 72 hours… Meaning that if you follow one of my links (even to a free download) and then buy something 2 days later then I get credited for the referral and make a few cents.

    Linkshare links currently make up about 15% of the revenue in my app business. It’s an extra little bit of money that takes very little effort to add to the bottom line.

    I’m building up my back end platform for reporting on all the various numbers I get for the business and putting them in one place. This past friday I got an email about the new web services REST api for Linkshare which I can use to generate various reports. It took just a couple hours to put together a django app that can download the month to date numbers and store them in the database for reporting.

    There might be a handful of people out there interested in using this sort of thing so I put the code up on github.

  • Fabric For Development

    Fabric is a pretty awesome tool for deploying projects. But it turns out that it’s also pretty awesome for lots of other stuff.

    At PyCon 2012 there was a talk given by Ricardo Kirkner (which you can watch here) that inspired me to play around with fabric in some new ways.

    It’s possible to use fabric as a wrapper around the standard django ./manage.py script, to help setting up virtualenvs and install packages. Using fabric to script around these things means that there are fewer tools that new developers will need to get set up and know how to use. Scripts that normally might have been loose bash files can now be collected, organized and documented.

    I’m currently working on a large django project with 4 other developers who are new to python and django. Getting everyone’s development environment working was a big pain since there were multiple platforms (Mac and Linux) and different configurations for base packages. If I had thought of this sooner I might have been able to create a reliable fabric script so that “easy_install fabric; fab init_project” got them from zero to running django app.

    There’s also several oneliners that I run fairly regularly which can be saved in fabfile.py and called much easier. For example:

    def clean():
        local('find . -name "*\.pyc" -exec rm -r {} \;')
    

    Now

    $ fab clean

    will clear out any .pyc files in the project.

    It’s also possible to manage the virtualenv environment through fabric:

    VIRTUALENV = '.virtualenv/'
    def setup_virtualenv():
        created = False
        virtual_env = os.environ.get('VIRTUAL_ENV', None)
        if virtual_env is None:
            if not os.path.exists(VIRTUALENV):
                _create_virtualenv()
                created = True
            virtual_env = VIRTUALENV
        env.virtualenv = os.path.abspath(virtual_env)
        _activate_virtualenv()
        return created
    
    def _activate_virtualenv():
        activate_this = os.path.abspath("%s/bin/activate_this.py" % env.virtualenv)
        execfile(activate_this, dict(__file__=activate_this))
    
    def _create_virtualenv(clear=False):
        if not os.path.exists(VIRTUALENV) or clear:
            args = '--no-site-packages --distribute --clear'
            local("%s /usr/local/bin/virtualenv %s %s" % (sys.executable, args, VIRTUALENV), capture=False)
    
    def virtualenv_local(command, capture=True):
        prefix = ''
        virtual_env = env.get('virtualenv', None)
        if virtual_env:
            prefix = ". %s/bin/activate && " % virtual_env
        command = prefix + command
        return local(command, capture=capture)
    
    def manage(command, *args):
        virtualenv_local("python manage.py {0} {1}".format(command, ' '.join(args)), capture=False,)
    
    def runserver(*args):
        manage('runserver', *args)
    

    These functions let you create the virtualenv and run commands in the virtualenv (without having manually activated it). virtualenv_local wraps the call to fabric’s “local” function and sources the activate script before launching the command specified. the manage function provides a way to call django’s manage.py script, and the runserver function gives a way to use fabric to launch the server (in the virtualenv). So now the fab command can be used to consolidate both virtualenv tools and manage.py script into one document-able fabfile.py with a consistent command-line interface.

  • Python is like a Secret Weapon

    I am continually in awe of the power of Python code. The readability of the language combined with the lack of magic, and the massive number of easily obtainable libraries out there make getting things done insanely fast. It is the only language where I consistently experience writing code that works the first time.

    Last night I added a geo-location aware link redirector with click tracking to my django web app. It took about 1 hour to code while I was listening to a panel discussion. Within that hour I had written, tested, added additional models, migrated the production database and deployed the new application.

    At my day job, we have decided to migrate our web app from grails to django. The benefits have been numerous. The Groovy to Python conversion has resulted in significantly less lines of code, unit test runtime has dropped from 15 minutes to 3 seconds. The delay in starting the dev server dropped from 10 seconds to instantaneous. The functional style of code makes it far easier to avoid copy/pasting logic between subclasses. Python’s flexible import statement allows us to structure the code so that it makes more sense.

    If you’re not using Python. You should be.

  • Peculiar Puzzle – Missing GET Parameters

    over the last week I have been seeing an odd error usually just once or twice per day out of the 10,000+ requests per day that hit my django web app backend for the iPhone apps.

    It appears as though the GET parameters get dropped and I can’t explain why. Hoping someone out there has some suggestions.

    The code in my iPhone apps check in to get data from the webserver. This is new content for the app. For each request it needs to have a parameter to determine which app is requesting the data – since there are a few using the same server.

    The app code calls a statically defined url string like “http://appserver.com/app_data?app_id=aiuwbev”

    However, about one in every 10,000 requests that come though create an exception because that app_id is not present.

    The other bits seem to be on the mark. They don’t seem to be coming from a web browser manually.

    So the most likely culprit is either that some small number of users are doing something sneaky, or there is a bug somewhere.

    I’m curious if anyone has seen this issue before, or knows what might be causing it.

    Edit:
    Solved – the problem was due to the string encoding on iOS side of things. Needed to enforce a utf8 encoding for all urls.