Category: Software Engineering

For those of us who can’t take the trip to Montreal this year for Pycon all the talks are on the Pycon 2015 youtube channel.

To become the best of the best you have to learn from the best and all the most active and knowledgeable Python developers are at Pycon giving amazing talks that will teach you something you didn’t know before.

This talk from David Beazley about concurrency and socket programming is quite informative. Watching him livecode the entire presentation is extremely impressive – it takes a tremendous amount of knowledge and experience to deliver a talk like this:

There are so many goodies in the youtube channel. I encourage every python programmer to watch and learn as much as they can.

I saw this said in a book I recently finished reading about company planning and strategy.  For sure when I read it first it caused me to pause.  Could there really be that much difference between good and great?

The more I thought about it, the more I came to agree that it is a pretty good estimate of the difference in costs to a business.  One great employee will do things better, with fewer errors and without the communication overhead of having to distribute the same work among three people.

In terms of salary costs that can add up very quickly.  The great employee may demand a 20-30% premium over the good employees but by needing 3 people your total costs skyrocket up 250%

It’s something to keep in mind if you ever are interviewing or hiring.  It’s usually better to wait for a truly great employee than to settle for someone that you don’t feel is going to bring a tremendous amount of value to your team.

Stack your company/team with A players and your chances of achieving your big audacious goals multiplies.  Introducing even a few even a few mediocre people can derail a project and create a stressful work environment for everyone.

Why would anyone ever suggest that it’s better to focus on quantity over quality?  Clearly pushing out a pile of sub-par work is not a good idea when you could take your time to think though things and produce a master piece the first time.

The fact is that you can’t easily produce top quality work without the experience you gain from doing something many times over.  Each iteration is a chance to learn from your mistakes and iterate on better approaches.  You can’t gain this experience without sometime dedicated to producing a quantity of work.

Malcolm Gladwell might consider this to be part of the 10,000 hours it takes to really master something.  A Aikido dojo might just consider it practice, and a student would consider it to be homework.  This is what it takes to get really good at something.

When you learn your multiplication tables you take the time to go over them again and again until your recall speed is high.  If you got to the test and had to multiply every 1-digit number by counting on your hands you would never finish the test before the time was up.  Getting good at anything takes sheer quantity to train your brain to be quicker.

As adults we tend to stop focusing on this as a way to learn new things.  We’ll just wing it and let our previous 30 years of life experience just coast us through a new task.

Even by the time you get to University the amount of practice you are expected to do is greatly reduced.  Professors will walk through one sample problem in the classroom and give one assignment question for each broad topic.  It’s hardly enough time to grasp some concepts, and certainly not enough practice to burn that knowledge deep into your cerebellum.

A journey to undertake a significant amount of practice like this takes small steps of progress every day.  By adding to your daily habits with the action of doing something to add a bit more practice can make a massive difference over the course of a few months or years.

I have personally challenged myself to write some Python code every day.  After a streak of 150+ days in a row now this is very clearly paying off.  The more I do, the deeper into the language nuances I get and my speed and recall has gotten better.  It means I’m more confident in the code I write every day and I can leverage the language in ways that make the programs I write better.

The focus is on the process of doing these things in volume, with a eye towards getting better each time.

Daily habits like these have a massive effect on you over time.  It’s like compound interest.  After a certain amount of practice you will be better than 99.9% people and at that point you are at the high end of a long tail distribution and get to enjoy the benefits of being among the best in the world at something.

For a decade I have pushed back on switching to an IDE for coding Python.  My preference has always been to use the minimal editor possible to help force a level of consciousness in my code and avoid the complacency that can come from having tools that are just too powerful.  Auto-complete is awesome, but if you lean on it too much it can prevent you from actually caring enough to retain information.

When the answer to a question is easy to find – your brain opts to avoid the effort required to push that knowledge into longterm memories in favour of just asking the question again.  Why bother remembering APIs when the IDE will just finish the typing for you.

Perhaps some of my distaste for IDEs came from having to use Eclipse to write Java.  A slow clunky editor that even after years of programming left me feeling inadequate at writing Java code.  It pushed me back to Vim for a long time in an effort to actually feel attached to my craft again.

Over the last couple weeks I have been exploring using PyCharm to write python code.  This was driven by being required to dig into some incredibly convoluted code written by other developers where a simple text editor just didn’t provide enough help with understanding how things worked together.

This experience has opened my eyes.  A good IDE really can connect a lot of dots.  I still think there is incredible value in learning a language, frameworks, and libraries as well as possible by actually typing them out in a dumb editor.  It takes a while to develop the knowledge to really be considered an expert.

Once you have the expertise an IDE can provide a lot of leverage.  I will probably never give up the simple text editor (vim is always a requirement for remote development) but will continue to lean on PyCharm for when development speed is critical.  Especially with a dynamic language like Python the insight from an IDE can sometimes be a lifesaver.

One of my new years resolutions was to give my side business a chance to start making real money.  To get actionable on that goal I started building a software tool (which is currently undergoing private beta testing).  The next part of the strategy is to produce several informational products and grow an email marketing list.

Over the next while I will be writing a lot.  I’m expecting to need to produce about 30,000 words of content. To hit that I have a set the daily goal of writing 1000 words every day.

Much like my daily commitment to write and push code to github (124 day streak!) I’m going to get a streak going for writing.

It’s going to be an ambitious month.

Over the last couple weeks I have been getting acquainted with using Ansible to provision and deploy servers. One nice thing with using Ansible over a tool like fabric for deployment is that everything from provisioning, initializing, updating, management, and application deployment can be done with a minimal amount of code.

There are tips I learned as I got started with Ansible:

1. install ansible with pip into a virtualenv.  This seemed to fix issues I ran into when having it installed globally on my computer.
2. create a new directory to work with – it will have all the templates for nginx, supervisord, roles, playbooks, and server lists in it
3. create an inventory file that lists all the servers you will manage
4. split things into multiple playbooks – I have a webserver, database server and deploy playbook. having the deploy playbook separate keeps it simple and it will run quickly.
5. the roles found on ansible-galaxy are almost never exactly what you want, even if they are you’ll likely have to read through the templates to find out what variables can be used to customize them.  In almost all cases it’s good to clone the role into your working directory’s roles subfolder so they are easily accessible and editable.
6. the idempotent property of ansible playbooks make it easy to procedurally build up your playbook.
7. it’s often easier to have an open ssh connection to manually see what’s changing on the server, make sure templates are rendering correctly etc.

my working directory looks like this for a simple single server Flask webapp.

For my deploy playbook things couldn’t be simplier:

---
- hosts: webservers
  vars:
    app_name: awesomeapp
    repo_url: https://github.com/mfwarren/secretproject.git
    repo_remote: origin
    repo_version: master
    webapps_dir: /srv/webapps
    virtualenv_root: /srv/webapps/pyenv/versions/latest_v3
  tasks:
    - name: git pull project
      git: repo={{repo_url}} dest={{webapps_dir}}/{{app_name}} version=master
      notify:
        - restart app
    - pip: requirements={{webapps_dir}}/{{app_name}}/requirements.txt virtualenv=latest_v3 executable={{virtualenv_root}}/bin/pip3
    - shell: "{{virtualenv_root}}/bin/python3 manage.py deploy"
      args:
        chdir: "{{webapps_dir}}/{{app_name}}"
  handlers:
    - name: restart app
      supervisorctl: name={{app_name}} state=restarted

the tasks are very easy to follow, git pull the latest changes, install requirements that might have changed, run the ‘deploy’ command on the app which does DB migrations, and finally kick supervisord to restart the app.

On the command-line my process for executing this deploy looks like:

ansible-playbook deploy.yml -i ansible_hosts

Ansible is a tool for power users. It would be quite difficult to use without experience setting up the servers manually. Unlike Heroku or another PaaS, it will take several hours to learn and configure your first server through Ansible. However that onetime cost should pay for itself with the additional control you have, and server cost savings.

I certainly would love to see tools created to analyze your project and create a set of playbooks and templates which would create a production quality server setup, similar to how Heroku’s Buildpacks can figure out how to get a Django/Rails/Node.js app running.