Matt Warren

Author: Matt

  • Getting Started With Ansible

    Everyone who does web development will probably find themselves doing devOps tasks at some point.  I keep an eye on servers hosting several different projects at different cloud providers and some private servers.  Until recently management of those servers wasn’t difficult enough to warrant using tools to deal with things.

    Ansible is a commandline tool that allows you to manage your servers.  Lets say like me you want to introduce it into an environment with servers already in place.  Here’s how you can get started.

    Install the Ansible commandline:

    brew install ansible

    Create a hosts file (file that lists all the servers you want to manage):

    ansible_hosts:

    [ec2]
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user

    Ping all the servers to see if they respond to ssh:

    ansible ec2 -i ansible_hosts -m ping

    Ansible comes with a large library of low level ‘modules’ for doing interesting things on your servers. You can call simple things right from the command line as in the previous example (uses the ping module). If you want to upgrade openssl package on all your (Ubuntu) servers:

    ansible ec2 -i ansible_hosts -m apt -a "name=openssl update_cache=yes" --sudo

    or upgrade all updates to a CentOS server:

    ansible ec2 -i ansible_hosts -m yum -a "name=* state=latest" --sudo

    There are over 200 modules built into the core of Ansible to make interacting with lots of different programs easy – git, shell commands, postgres databases, supervisorctl, django, gem packages. Doing a git checkout of your application across a pool of web servers can be done with a single command line.

    If that was all you could do with Ansible I’d be happy. But that’s really just scratching the surface. Building on these low level modules is what Ansible calls ‘roles’. Roles encompass a series of commands to do a bigger task – like install and configure nginx. You can tie multiple roles together into a ‘playbook’ that can define how to build out your entire infrastructure.

    In my next post I’ll cover how to go from zero to fully deployed web application with Ansible.

  • The Quantified Programmer

    Expanding on the thoughts in my last blog post about measuring opposing indicators I wanted to explore things from a different perspective.

    There is a growing community of people that measure and record as much as they possibly can about themselves.  This quantified Self movement is predominantly focused on health related aspects exemplified by the growing market for activity trackers like FitBit.  The more information you have to track yourself the more you are able to identify longer term trends that would otherwise be twisted by your own fallible perception.

    Programmers strangely do not have much going on for measuring themselves.  Even though it would be quite easy to glean information from the coding output from any programmer.

    I would be quite interested to see a collection of stats about myself.  A daily digest of basic productivity, and charts of longer term trends would be motivating and insightful.  Would lines of code per day increase as I got better at a new language/framework?  Would there be plateaus in my productivity? Could I identify the impact of distractions? How much more productive am I working from home vs. at the office?  There are so many questions that would have valuable answers for me personally in my quest to improve.

    While no measurement is going to be perfect, by measuring myself against myself, for personal insight provides some incentive not to cheat and keeps the relative performance as an informative metric.

    GitHub provides a decent API which could be used to dive in and gather this information.  It would be interesting to launch a service that could mine through a user’s github account to generate a detailed productivity report.

    Unfortunately another project I don’t have the time to build myself.  If someone else builds it though. I would surely be a paying customer.

  • Taking a Machine Learning Course

    I started a Stanford course yesterday on Machine Learning.  It’ll be interesting to update my knowledge since things have changed a great deal in that area since I did an AI class at Waterloo 10 years ago.

    Machine Learning has really taken off in the last couple years as the number of problems and the technical capabilities of AI algorithms has evolved.  In particular services like voice and text recognition are like magic.  Applications like self-driving cars, flying drones, game bots, chat services, marketing segmentation, bio data analysis and astrophysics have exploded with new use cases for this type of approach to software.

    A good machine learning algorithm offers tremendous leverage, since it has the chance to solve problems that people have a hard time defining how to solve them.  A solid self-driving car system could disrupt a trillion dollar industry.  Self piloted drones could create a new trillion dollar industry.  These are algorithms that could impact everyone on the planet.

    So I’m kind of excited to refresh my knowledge and learn a few new things.

  • Measure Opposing Indicators

    All software developers know that measuring a developer’s productivity by counting lines of code written is not an effective way to measure their output.  The correlation between the technical difficulty of a problem and the number of lines of code is not always 1:1.  Which means that one developer can write 10 lines in a day and be very accomplished, while another could copy/paste a solution of 1000+ lines in a day which was trivial.

    On the factory floor managers always should be measuring opposing indicators.  If you’re measuring the volume of output – you also need to measure the quality of output in order to prevent creating an unbounded method for taking advantage.

    For example.  If a manager of Intel processor factory production line focused solely on the number of chips produced each day, and cancelled all quality control checks then production output would likely skyrocket.  In short order, however, the defects would be found by customers, the business would suffer.  On the other extreme, if the manager focused solely on quality and checked every aspect of each chip produced then costs would skyrocket and output would crash.

    The problem with measuring a software developer by the number of lines of code they produce is that there isn’t an opposing measurement of the quality/difficulty of those lines.  Gauging the quality or difficulty of the produced code would likely require peer review, and for that peer to grade the code.  Nobody wants to be the ass that gives their co-workers D- grades on their code so it becomes difficult to get honest scores.

    In my opinion, counting anything is better than counting nothing.  If Bill suddenly goes from committing 100+ lines of code per day down to 10 lines per day it may indicate something that a manager should investigate.  If you weren’t measuring output then it would be impossible to know about or fix anything that Bill was having trouble.

    Authors measure themselves by pages written per day.  Writing 10 pages of crap is better than ending the day without having written anything. At least once things are on paper they can be communicated and others can help edit things to find the gems inside.

  • Make Reading Inevitable

    One of my big goals this year is to read one book every 10 days.  That’s 36 books to read before Christmas.

    Inevitable thinking is about asking one question: “What can I do to make the outcome I want INEVITABLE?”  The phrasing of that question prompts you to think about things differently.  Instead of just stating the goal – I want to read 1 book every 10 days – you are forced to delve deeper to take actions that make sure it is guaranteed to happen.

    In order to keep pace with my reading goal I have leveraged the same tool that has kept me blogging every week for over 5 years and kept my GitHub streak going strong (currently at 84 days in a row!)

    I’m keeping my reading history in a spreadsheet on Google Docs:

    Books_-_Google_Sheets

    Then I have a small script scheduled to run every hour.  It reads this spreadsheet, finds the most recent date I finished a book and if it was more than 10 days ago I get a rather offensive email hounding me to read a book.  I’ll continue to get one email every hour until I finish a book and update the spreadsheet.

    This simple nag has proven effective at changing my behaviour over the long term.  With this system in place I’m confident I’ll hit my reading goal this year.

  • The Most Often Overlooked Programmer Skill

    Programmers often take something for granted that would make them significantly better at their job.  No it’s not communication skills, or management.  The biggest differentiator between the best and average programmer is how often they have to refer to google to solve a problem.

    Any programmer who knows the basics of programming in a C like language feels like they’re competent enough to write Javascript – even if they only have to program it occasionally.  What ends up happening is they stall on every other line of code to crawl stack overflow for solutions.  Despite their lack of skills they can still write complex code and get things done.

    No doubt that Google and Stack Overflow are awesome resources that make it easier to write things you don’t really have the knowledge to do off the top of your head.  Each detour to the search engines eats time.  It takes 10x more time to find a good piece of code on the internet than if the programmer knows how to write it already.

    The skill that most programmers overlook is memorizing the APIs and languages they use.

    When you take away all the crutches we use to help write code by writing with pen and paper, without the help of reference material gaps in your knowledge become brutally apparent.  Try this from time to time.

    Most programmers I know seem to think that picking up a new language or framework can be done in just a few weeks.  My experience is that unless they take deliberate time to read and learn and build expertise the typical result is getting to a passible skill level and then plateauing.

    Do not underestimate the value of depth of knowledge.

  • Don’t Write Code Unless You Use These 11 Tools

    ToolboxDo you want to write code faster, with fewer bugs, and feel confident that what you write is good?

    Over the years, programming has evolved, and many new platforms and tools have been created. Most of the tools aren’t very useful, but a few are.

    Here are the 11 tools that have helped make my programming a bit easier and more successful:

    Tool #1: Linting

    Every language these days has a selection of Linting tools.  These are programs that parse your code to detect and flag best practices.  Linting tools enforce consistency of code style among all those in your team, can detect complexity, unused variables, naming conventions, or a long list of other infractions that help you to keep your code clean.

    A side benefit from Linting tools is that they can hint at things which may just be general improvements.  Enforcing Javascript semi-colons, suggesting modernization of Ruby code or helping keep Python code PEP8 compliant can often trigger you to think twice and maybe re-factor to improve your code structure.

    If you’re code environment doesn’t have linting built in, get a better development environment, or install a plugin. Linting is a great tool for improving your programming skills.

    Tool #2: Dash

    Google and Stack Overflow are great resources for solving quick questions you have about something you need to do but having a local copy of documentation to reference can speed you up tremendously.  Dash is a Mac App that lets you search through documentation for various languages and frameworks quickly and easily.  It copies them locally so it works when you’re offline too.

    Everytime you search online for an answer to a question there are two things that can go wrong:

    1. You get distracted reading articles, browsing things and loose time
    2. You get the answer you want without enough knowledge to understand it, copy/paste and promptly forget how it works.

    Reading real documentation often forces you to learn something new.  Well written documentation will explain things enough to give you a firmer foundation to help you remember it the next time you encounter similar questions.

    Tool #3: A Solid Text Editor/IDE

    I have been using Atom for the last couple months as a coding editor.  I’ve always personally felt that IDEs can often do too much which makes me uncomfortable.  Which ever side of the fence you’re on, one thing is clear.  You need to get yourself a great editor.

    This is the primary tool you use when coding – don’t be afraid to spend a little bit of money to make your life better.  Take the time to truly master this tool.  Write your own plugins, customize the shortcuts, tweak the theme, read a book!

    Tool #4: Vim

    Yes, vim is a text editor, but regardless of your editor of choice you should also know how to use vim.  Especially if you work on servers.  Vim is installed by default on just about every Linux machine, so it’s the workhorse that is very nearly always available for you to use.

    Tool #5: Git

    There are many methods for managing source code versions. Git is the one you should be using.  CVS and SVN are long dead, Mercurial and Baazar lost out to git.  Even if you just are coding locally, without any other developers git is a gift that will someday save your ass.  There is really no excuse not to have your code under source control.

    If you want a GUI for interacting with git – checkout SourceTree.

    Tool #6: Commandline

    A master of the commandline can accomplish a lot of things with relative ease.  One of the nice things about a commandline interface is that if you know how to do something on the command line (as opposed to in a GUI) you know how to automate it or turn it into a script without any additional thought.  With a GUI you know how to do something by moving and clicking the mouse – and it’s unlikely you’ll every investigate how to do that task any faster even if you have to perform it 1000 times.

    Tool #7: GitHub

    You’re using git right?  GitHub makes git better in several ways.  By adding Issues, Wikis, Pages, you can plan your Todos and publish your documentation.  You can easily give access to repos to new people when they join the team, or make things public so anyone can fork your project.  Use Github webhooks to plug in and automate your own workflows.

    Have a public face to yourself and your company. Github is the first place I look when checking out a potential new hire.  Your github profile is the modern extension of your resume.

    Tool #8: Newsletters

    K, this isn’t really a tool, but I think it’s important.  Reading the various Google+ Pages, Reddit subreddits, Twitter, and blogs for the languages, tools and frameworks you use, is simply too much noise to reasonably parse.  I suggest signing up for a weekly newsletter instead.  A much more reasonable weekly digest of the most important new things will keep you up to date, without sucking up all your time.

    Tool #9: Books

    Books are often underrated by software developers.  Unlike blog posts which are often written quickly and published in the first draft.  Books are often written by multiple authors, reviewed by a panel of experts and edited by professionals.  These extra filters helps to lift up the quality of content in a book much higher than you find laying around on the web.

    If knowledge a $20 book gives you saves you 10 minutes then it’s paid for itself.  It’s much more likely that a good book will save you days.  A great book could help you double your efficiency.

    Tool #10: Continuous Integration/Unittests

    I have found that different languages and tools necessitate different levels of unit testing.  Python is relatively explicit and APIs are generally consistent – python code usually behaves the way you expect it to.  Ruby syntax on the other hand is prone to ambiguity, has multiple syntaxes for the same things, is poorly documented and often implicit in how things work.  Even after over a year of full time Ruby development under my belt I trust that any untested code is broken.

    Continuous Integration runs your tests after every commit, or every change you make locally.  By running tests often you can catch regressions quickly before they become an embarrassing bug.

    I recommend using a web service like Travis-CI.  Setting up your own CI Server can be a bit of a pain.

    Tool #11: Practice

    If you hired a coach to teach you how to play tennis, you expect to run drills, learn how to grip, swing, stand and step. You’d hit thousands of balls until the mechanics of how to hold the racket, swing and step become so instinctual that they can be performed without thought.  You would practice several times per week, for 1-2 hours each time just performing basic moves.  It would be months before you actually are ready to play a game.

    Nobody practices anymore.

    I believe that deliberate practice is what differentiates an average developer from a 10x developer.  Practice with the languages, frameworks and libraries you use and the tools you write with.  Every API that you memorize will add up to hours of saved time searching Stack Overflow.

    Conclusion

    Writing code doesn’t have to be hard. There are a ton of free tools out there that can help you write better code faster.

    The 11 tools I mentioned above are the main ones I use on my projects. Give them a shot. They’ll change your coding experience.

    What other tools do you use when coding?

  • Practice, Homeostasis, Resolutions and the Path to Mastery

    It’s that time of year again when everyone is making ambitious new years resolutions.

    I started by re-reading one of my favourite books:

    Mastery is about the journey – an endless one. Taking comfort in the practice, ritual and repetition of the fundamentals while endlessly pushing the edge of your skills and knowledge.

    Something that’s been on my mind has been how to get even better at software development. Several months ago I did a massive purge on my blog subscriptions in order to make room for reading more books. Split with reading time, for the last two months I have made an effort to freecode everyday (my github account is on a 66 day streak).

    Making real changes involves a disruption to your homeostasis. You can expect to encounter forces to bring you back to your existing patterns. Knowing to be on the lookout for this is half the battle. The other is to put in place new checks and balances to create a new homeostasis. My most successful long term changes started with something other than just a personal commitment. Blogging and Github commits have been made permanent by having scripts that email me when I get out of balance. Training for a marathon worked only when I had found a social group that expected me to show up.

    One thing that I have been trying to think more about lately is how to apply practice to software development. If you were taking tennis lessons you might hit thousands of backhands for hours multiple times per week before moving on to a different stroke. If you were learning piano you’d probably spend hours playing scales. If you were learning karate it might take a decade of repeating the same motions to reach black belt. Practice is key.

    I don’t know any software developers that practice their craft in the same way. It’s hard to find any information about other developers taking a similar approach to learning.

  • Asset Allocations

    Late last week my RRSP transfer from mutual funds into Questrade completed.  It feels as if I lucked out in that the 5 worst days of the year on the market happened while the cheque was in the mail to move money.

    Trying to devise a strategy for investing that is both safe, has the potential to beat the market, and protects me from myself lead me to read a couple of books on portfolio management theory and devise an overall approach to managing my investments.
    The take away from the world’s best investors is that the best approach for investing is:
    1. Diversification is the only guaranteed way to reduce risk and maintain good returns
    2. Rebalance your portfolio no more than twice per year to reset your risk exposures to account for drift
    3. Don’t trade on emotion – you are your own worst enemy
    4. Always consider the tax implications and costs to trade
    One of the biggest lessons I got from working at an investment management company is to hedge all your bets.  You never put all your chips on red because your analysts think it’s the best bet.  You buy the whole board and make small bets.  This way you maintain diversification – which lowers the variance of your whole portfolio.  And provided you can do better than 50% on picking winners and losers the portfolio will do better than it’s benchmark.
    Extending that to the asset allocation for a complete portfolio is relatively straight forward.  It starts with the question: What should my portfolio look like at a high level?  To answer that I’ll go through the break down of how I built my portfolio.
    The asset types I went for were:
    1. 40% Bonds
    2. 50% Stocks
    3. 5% Precious metals
    4. 5% Commodities
    Noticeably missing from this list is REITs.  I own a home which is more than enough exposure to real estate for my liking.
    I’m targetting 90% of my investable assets into broad low cost ETFs. 10% is individual stocks which I believe have good moderate term prospects to beat the market.
    Within each broad asset category things are further diversified:
    Bonds:
    • 1/4 High Yield corporate bond ETF (HYG)
    • 1/8 DEX index Canada government & municipal bonds (XBB)
    • 1/4 Canadian Real Return Bond ETF (XRB)
    • 1/4 Canadian Hybrid Bond ETF (XHB)
    • 1/8 Emerging Market EFT (LEMB)

    For bonds I wanted a good mix of corporate, government, and international bonds that were hedged against interest rate changes a bit with the Real Return bond (XRB).  Once I get a chance to compute the co-variances on this selection I may re-allocate.  In particular it seems a bit under-exposed to US and EAFE bonds.

    Stocks:

    • 4/10 TSX 60 ETF (XIU)
    • 4/10 EAFE ETF (XIN)
    • 1/5 Personal stock picks – currently (AAPL, BA)

    Admittedly this isn’t as diversified as I’d like, future purchases may go towards something very broad like the MSCI World ETF (XWD) or Vanguard World ex US ETF (VEU). I’m betting on Apple because they have a strong product lineup, entering new markets next year and have been doing a lot of stock buy backs.  Boeing is my bet that if oil stays low, airlines around the world will see billion dollar profits and funnel that back into growing their fleets with new planes.  Boeing is also a hunch that it’s possible to get ahead of the next trend by thinking critically about where profit from the current trend could end up.

    Other:

    • Precious metals are in gold bullion (CGL)
    • Commodities are in a broad ETF (CBR)
    There are so many options for things to invest in that it’s hard to choose.  My hope is that this portfolio will be low risk and have great returns.
    New money added to the RRSP will be used strategically to keep the portfolio in balance which will hopefully reduce the need to make big re-adjustments.  The way to think about this is that I will be buying the assets which have become the least expensive – thereby dollar cost averaging down.  It’s a way to frame things to avoid panic selling at the worst possible time.
    It was kind of a fun week to research, read and figure out how to invest things.  Learning is, of course, an endless journey and I’m excited to iterate and get better at investing.  It’ll be exciting to see how this portfolio does over the next year.

  • Why You Should Build Your Own Tools

    Something has happened in the manufacturing world that is nearly irreparable.  As America moved production to China and downsized factories to realize the profit gains from low wage Chinese workers it also decimated an entire skill set from the national tool belt.

    When a factory tools up a new line to produce a new iPhone, sneakers or toy it is the job of a skilled tool maker to decide what custom tools can be designed to make things efficiently.  These may be custom 1 of a kind drill bits, platforms, moulds, or machinery that someone with a creative and technical skill-set needs to design and create.

    When America outsourced it’s factories it also lost these very valuable skills.  Now, as a result of that loss of knowledge spinning up a new factory in America is a very difficult task to accomplish.  That loss will take a long time to recover from.

    In the world of software development, abstraction is often a panacea.  Abstraction has created a similar migration of technical skill sets over the last few decades.  We abstract out the complexities with layers upon layers of helpers, protocols, frameworks and libraries.  These extra abstractions can boost our productivity tremendously so long as we don’t encounter a use case that isn’t handled by our chosen abstraction technology.

    The danger when you completely ignore how those abstractions work is that the skills and knowledge it took to create them die out as well.  Who are the tool makers?

    I have found it incredibly useful when trying to understand a new tool to have a solid grasp of the underlying technology. Knowing how to build a HTTP service at the TCP level (and from time to time actually doing it) makes it easier to reason through issues and push the limits of a framework like Ruby on Rails.

    Software developers everywhere should spend just a little more time writing the tools they need to do things quicker in the future.  It is investments in the tooling that has the chance to compound to make future work exponentially easier.

    Create stand alone services that can be leveraged on multiple projects, write code generators, and libraries. Create your own frameworks.

    The style of programming required for creating a great re-usable library is very different from that used when doing a business application.  Good, flexible code often relies more on using the dynamic capabilities of a language.  You’ll often find yourself needing to use meta classes, introspection and other fun constructs.  The people that know how to wield these tools are going to be better at debugging, see opportunities to improve code, and create their own tools.

    My plea to software developers everywhere: create your own tools.  Know how to create the tools you use, and how to use the tools you create.