Category: Software Engineering

There’s a quote in Computer Science – “There are only two hard things in Computer Science: cache invalidation and naming things.”  Well, naming things is hard for everyone.

Everyone brings to the table a history of accumulated connotations and associations such that the meaning of everything is different to each and every one of us.  Finding the right word for something new is ever more challenging in some ways. We intuitively gain a sense of meaning of words even if we have never heard them before in many cases.. this requires deliberate thinking.

In software, naming variables well can be the difference between sensible readable self-documenting code or unintelligible write-once junk.

Likewise naming things in the real-world is also difficult, the right name really does make an impact.  The right company name will make a difference, the wrong name will make everything harder.

Open Source software has been on my mind a lot the last few weeks with regards to a problem that exists within the community.

The world is built on the shoulders of Open Source software. If you look at any tech startup out there,  they are more public code than proprietary. We leverage open source as a starting point for new ventures and new innovation.  Even large companies – the Googles and Facebooks of the world would not run without that foundation of open source code.

The reliance on Open Source inflates the risks. Bugs in open source come at huge costs due to the amount of shared code by everybody. The direct costs of a bug or security vulnerability could be massive.  For example, the estimated cost to fix the Heartbleed vulnerability in OpenSSL was $500,000,000. There are indirect costs as well, relying on goodwill of volunteers means you can’t have the best people work on the hardest problems.

Yet, despite the critical nature of this code the vast majority of work on open source goes unpaid.  Most projects are started out of personal interest – an often fleeting commitment. There is little accountability and in fact as has happened time and time again these developers burnout and abandon popular projects.  Were you to do ‘due diligence’ on any of these projects you would find that 30% of the top 133 projects on Github are maintained by a single person, that the next 34% have only 2 contributors.  These are at risk of being abandoned quietly or at short notice.  Key pieces of supporting infrastructure such as package management services need to be up, but are maintained by people who need to book vacation days from their paying jobs to do so.

Github was launched in 2008 and with it, a new wave of open source developers started to dip their toes into it.  These open source boomers are becoming more senior, they are realising the value of the code they write, and they are starting families that vie for their time and attention. For many of these developers burnout is real and they increasingly need to justify their continued involvement.  If we ignore this problem the code we all rely on for a functioning internet could be abandoned.

It is a growing problem for which something must be done.

Loss aversion is the preference we have to avoid losing $5 compared to gaining $5.  It’s logically the same value in this example but yet some studies have shown that we are twice as likely to favor avoiding the loss.  This effect plays out multiple different ways in our lives.  We tend to hold on to losing stocks to avoid taking the loss, we price anchor our houses and have a hard time accepting lower bids, we stay in bad relationships, we stick to secure jobs instead of following our dreams.

We should be conscious that we are ourselves under this bias with our own choices.  The truth is that perceived losses FEEL worse than reality. Actual losses are usually less dire than we imagine. Worst case scenarios are usually not as bad as we intuit.

The flip side of that coin is that potential gains are usually undervalued.  The prospect of making an extra $100 per month has LESS perceived value than its actual impact on your finances would be. The upside to a successful entrepreneurial venture are under appreciated by all but the most motivated entrepreneurs.

We are not great at logically assessing risk. Hence cold hard math can be a competitive advantage since it has a way to shine a light on our biases and accept facts.  When you find yourself contemplating taking a loss assume that your ability to come to the optimal solution is compromised and therefore the exercise of doing a more logical analysis is often beneficial.

One of my ambitious goals for the year is to take on 12 programming projects.  The projects I’ll undertake for this challenge have a couple of requirements:

1. It should be something that I can complete in 1 month of calendar time
2. it should take less than 40 hours of effort
3. it should ideally challenge me to learn something new
4. it should maximize impact for someone other than myself

Trying to think of project ideas that can have a big impact with less than 40 hours of effort is not easy.  Luckily Kenneth Reitz serves as a model for how to accomplish this.  He is best known for authoring the awesome requests library, which is ubiquitous. Over the last while he’s been on a tear.

• typy.io – a service for sharing text snippets
• pipenv – a wrapper that combines pip and virtual envs
• maya – datetimes for humans
• saythanks.io – a way to send thank you to open source developers

These are potentially high impact projects compared to the effort required to create them. It’s something that I will try to replicate this year with my own projects and an approach to open-source development that I wish more people took.  Flooding the community with high impact contributions enriches us all.

Impact can be a trade off.  A large impact can come from a small improvement for many people or a large benefit for a small number of people. And of course these are all relative to what leverage you have to help.  The audience that Google can have an impact on is vastly different than the number of people I can reach. So I’m trying to be realistic about what kind of impact a 40 hour project can have.

Optimizing for impact seems to be a great goal.  It re-frames the importance of a project; If I could do something in 40 hours that would double someone else’s business it might be a worthy project to consider.  If I could contribute a wrapper for an API that is used by 1000s of developers it could have a wider impact.

Scratching your own itch is the common motivation for open-source development, but in a sense it is inward focused and self-serving. In a world where we increasingly don’t talk to our neighbors or contribute to our communities doing things to help others sounds radical.

Be radical, make an impact.

For software developers there is an unhealthy prevailing belief that being a great programmer is some innate skill that others have. Brilliance with developing code is difficult to train for because it either requires some gift you don’t have or years of on the job experience.  There is a large amount of impostor syndrome within the community which is not healthy or productive.

Of course people who are top developers know that it takes a lot of hard work to understand core concepts. It helps to have a mentor and a solid education and access to training.

There is a tactic to getting better which more programmers should be using.  Deliberate practice is the most critical aspect to improving any craft and programming software is not an exception.  Like playing piano or painting or ceramics there is creativity and technical skill which can be improved on with deliberate practice.

If you want to get better at your craft it is not good enough to simply work on job tasks. For work you typically do something once and then it’s done, there’s few opportunities for repetition and critical evaluation.  If you were learning to play piano and you had a sheet of music the equivalent to developer workflow would be to play through the song once, stopping to go back and fix your mistakes then when you finished you’d put the song away move on to a new song.  Practising piano requires playing the same song hundreds of times, you start by playing and focusing on not making mistakes, when that is accomplished there is still practice at making the song sound good with appropriate pedal usage, tempo and dynamic, and finally when that is good enough you can continue to practice the same song and add your own touches – arpeggios, slurs, delays etc.

How many times have you implemented a deck of cards?  Can you write one top to bottom without looking up examples on stack overflow, or querying the documentation or searching through code completion lists? Could you write a deck of cards in a procedural, functional and object oriented styles?  Could you meta program a deck of cards? Could you make a deck that is thread safe? distributed? Web scale? Obfuscated?

Practice. Do it often, and do it deliberately.

Writing code everyday has been an interesting challenge.

In 2015 I started to work towards a long streak on GitHub which eventually capped out at 250 days. The questions I wanted to answer was:

• Can I apply ‘deliberate practice‘ to programming and get better?
• Can ‘free coding’ (like free writing) be effective way to push through writers block?
• How important is memorising to your coding performance?
• If syntax and API unknowns don’t present bottlenecks to your flow how fast can I translate an idea into code – can it be limited by typing speed?

I started a repository for my daily coding.  It had a simple script to generate a blank file everyday for me to code in and I would try to code something.  Sometimes it would be to explore a new python module, or fiddle with syntax, or challenge myself with a rosetta code example or replicate a previous day’s code from memory.  I wrote dozens of Flask apps, memorised the methods of lots of APIs, and gained a level of confidence with writing Python that I don’t have with any other language.

At the end of the streak I had a repository with hundreds of small scripts.  Only a handful of them were multi-day efforts or had any real value.  The variety of this collection proved to be useful on it’s own too – several times I have referred back to these examples to help with my actual work and to copy/paste snippets from.  Some of them started me down a path of exploration – like calculating the return on investment for solar panels.

Part of what enabled me to maintain this streak as long as I did was a simple script I wrote to check GitHub for daily activity and email me if I hadn’t yet committed any code.  This simple hack was enough of a reminder to keep me focused even when I was otherwise distracted.

This past week I turned that script into a web service anyone can use.  CodeStreak.io will watch your public github activity and email or SMS you if you haven’t yet pushed any code for the day.  This is the first project of 2017 that I plan on building to expand on my previous streak.

In 2017 I want to build 12 projects.  Each should be roughly 10-20 hours of effort and result in something that provides value for other people.  CodeStreak.io is an example of the kind of project that I want to undertake this coming year, but it is also a tool to help ensure that the momentum is sustained for 12 months.  Blocking out 4 hour chunks of time is a helpful way to really focus and be productive, but 4 hours once per week has been (for me) too sparse to maintain interest in something long enough to finish it.  A little bit everyday keeps a project on your mind.  Attempting to maintain a streak will be a tool to power through the bits that are otherwise uninteresting or difficult.  CodeStreak.io is a foundational tool necessary to accomplish my 2017 goal.

The questions I want to explore with this new goal are:

1. Without a concern for generating revenue can I just write cool things and get them out there?
2. Can I get deeper into something new and create something useful out of it with less than 20 hours of effort?
3. Can you get good at seeing a project from start to finish – what skills or traits will improve the odds?

Hopefully, I’ll have some answers at the end of 2017.

One of the welcome additions to Amazon’s AWS offerings is a simplified server provisioning service to compete directly with Digital Ocean called Lightsail.  Lightsail provides a nicer web UI for launching instances, many quick launch options for common apps like WordPress or GitLab and simplified billing (yay!).  With Lightsail you don’t need to pre-pay for Reserved Instances to get a good price on an EC2 server.

Dokku is mini heroku you can run on your own servers.  It uses the same buildpacks that Heroku does to enable git push deployments. By building on top of Docker a collection of available dokku plugins make it easy to start up databases, caching or tie in other services.  In this tutorial I add Postgresql and get an SSL cert using Let’s Encrypt

Together, Lightsail and Dokku create an easy way to manage your application deployment on an inexpensive server.

Get started on Lightsail by starting up a new virtual server:

And then selecting an Ubuntu Image:

There’s a spot here for ‘Add launch script’ where you can drop in these commands to automatically install dokku on first boot:

wget https://raw.githubusercontent.com/dokku/dokku/v0.7.2/bootstrap.sh
sudo DOKKU_TAG=v0.7.2 bash bootstrap.sh

Give it a name and press Create to start booting up the server. You should be able to SSH to the new server very quickly though you can connect before dokku and package updates have been applied (it’ll take a couple minutes for the dokku command to become available)

After a couple of minutes have passed and things are installed and running visit your server in a web browser:

For the public key you’ll want to grab the key on your computer.  if you have linux or macOS you can grab the contents of ~/.ssh/id_rsa.pub.  If you need to generate a key there’s a good How-To on Github about generating them.

Set the hostname you’ll use for the server if you have one and Finish Setup.

Next step is to SSH to the server and fiddle with it there using the private key you can download from Lightsail:

ssh -i LightsailDefaultPrivateKey.pem ubuntu@<YOUR PUBLIC IP ADDRESS>

And create the app you will be deploying:

dokku apps:create your-app

Add a postgres database (there are other great plugins available for dokku too)

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
dokku postgres:create database-name
dokku postgres:link database-name your-app

Now, back to your local app add a git remote to this new server and deploy it:

git remote add dokku dokku@<PUBLIC IP OR HOSTNAME>:your-app
git push dokku master

If that is successful then the project should be visible online. Yay!

Then there are some next steps to help complete the app. Set any environment variables you need for the app:

dokku config:set your-app ENV=prod

You can install an SSL cert using Let’s Encrypt very easily:

sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
dokku letsencrypt your-app

You can configure some pre and post deploy hooks inside the app.yaml file in your project repository to run checks or execute database migrations.

That’s about it! git push deploys to update your project whenever you want.