Matt Warren

Category: Software Engineering

Coding, Python, web development, architecture, and deployment

  • Opportunity Costs

    We all only have so much time and attention, and with constantly being pulled by work, friends, family in various directions we are only able to give our own causes and ambitions a percentage of ourselves.

    Opportunity cost is what we lose out on for taking one path over another.  In a way opportunity cost is infinite.  We could at any time take any of thousands of possible actions which could, in aggregate, lead to very different futures.

    In the present though, opportunity cost is speculative.  We can’t know for certain how much taking one action over another will be better or worse than another action.  Although we usually have enough information to help make informed choices.

    Making the optimal choice is non-trivial.  And so we apply tools to help us make smart decisions.  We go to school to understand how things work, study history to learn how things have been done in the past, and use frameworks to help guide us to use best practices.

    Going against these best practices are countless other factors.  Emotional impulses, reaction to external factors, others asking things of you, habits, and restraints. They act like friction preventing us from making better decisions. Some of these things we can easily do things about, recognize and counteract. While other things are very difficult to ignore or fix.

     

  • Best way to learn Machine Learning

    I’m going deeper in my learning about how to successfully implement machine learning algorithms this year by initally doing a survey of all the resources out there for learning this stuff.

    It is a fast moving area of expertise and as such newer techniques and tools wont be covered in older books or tutorials.

    MOOCs are now a great way to get up to speed on the Deep Learning approaches to Machine Learning.  And while there are some good quality general books out there about ML, most are currently in pre-order.

    The most appealing to me right now is the course on Udacity, presented by Google which uses Tensorflow in iPython notebooks to teach how to build and apply ML.  The best thing is that it’s

    1. in Python
    2. uses the latest ML library TensorFlow (developed at Google)
    3. Is free

    As with all learning, the best way to learn is by doing it yourself and practicing enough to make it stick.

    This is not the first resource I’ve used to learn about topics in Machine Learning and it won’t be the last.  Taking multiple courses, reading multiple books and tackling multple problems on your own is the best way to ensure you have no gaps and a well rounded deep understanding of the concepts.

    Actually mastering a new skill is hard and there are no shortcuts.  Accept that and jump into the challenge.

  • SSL All The Things

    If you’re a frequent reader of this blog you might notice that eveything is now going through https.

    Thanks to the awesome people at Lets Encrypt I’ve managed to get free SSL Certs for many of my websites.  If it’s free why not encrypt all the things.

    Nobody uses telnet anymore when ssh is far more ubiquitous and secure.  Yet the use of unencrypted http is vastly more popular than the secure https version.  Why? Mostly because of cost and complexity with setting up https.

    Lets Encrypt is a pretty neat tool.  you run it on the server directly and it will validate the site automatically, generate the keys, install them into apache or nginx and modify the configurations to enable https for those sites.  It’s free and just takes a few seconds to generate and install the certificates.

    Now that it’s easy and free to setup SSL Certificates and enable HTTPS, I’m hopeful that it will force other SSL providers to compete and encrypted web traffic will become the norm instead of the exception.

  • Project Mu

    In a one night spark of interest I thought I’d write a webframework that deploys to Amazon Lambda and API Gateway.

    A web framework that deploys to a serverless environment is kind of an interesting idea.  Costs scale directly with the users and limit the base costs in some cases all the way to $0.

    Now that Amazon Lambda supports Python I thought it was worth trying to make something that allows you to manage a larger project in a way that is familiar to most web application developers.

    To start with I based the framework on the awesome Falcon web framework because it is 1. high performance 2. API focused and 3. very easy to understand.

    What makes this framework unique is that deployment needs to be baked in.  The framework needs to understand how the APIs are built in order to chop them up into stand alone functions and upload them to Amazon.

    The net result is an easy to use framework for developing APIs that can deploy directly to a scalable platform.

    It’s still early alpha stages but I’m hopeful that this or something like it will catch on in the future.  For those interested, you can follow the development on github.

  • Why You Need a Meta Project

    I got this idea from the guys at Yelp and how they manage their deployments and several internal infratructure tools.

    The basic concept is this: If you had a source of information about all your projects in a simple easy to work with format what kind of tooling would be easy to implement with it?

    Yelp does this with a git repository full of yaml files. coupled with a handful of commit hooks, cron jobs and rsync they are able to provide local access to this information to any scripts that want it.

    If you had the ability to get information about all your projects what kind of things would you want to know:

    • where is the git repository
    • where are the servers hosted
    • who should be notified if there are problems
    • how is it deployed
    • how can it be monitored

    With this information what kind of easy to write scripts could be developed?

    • connect to all servers and check for security patches
    • check all git repositories for outdated requirements
    • validate status of all services and notify developers of problems
    • build analytics of activity across all projects
    • be the source of information for a business dashboard

    Also interesting about this approach is that it easily handles new strategies.  If you’re deploying to Heroku, Elastic Beanstalk, raw EC2, Digital Ocean, or through new deployment services it doesn’t matter.  Create a new document with the information needed for a new method and write the required scripts that know how to use it.

    By not using a webservice or database you gain the simplicity of just dealing with reading local files.  This low bar makes it trivial to implement new ideas.

    A meta project: a project with information about other projects is an intriguing and powerful idea.

  • Raspberry Pi Zero

    I’ve been intrigued by the stuff happening in the IoT space over the last 6 months because I’ve been spending considerable amount of time working on an IoT project for work.

    Yesterday the Raspberry Pi guys put out a new model before the holidays.  A smaller device that costs only $5.

    A $5 computer.

    So I ordered one along with a couple of extra bits to help make developing an application a little easier.

    Not sure exactly what I want to do with it yet but I have a long list of possible ideas.

  • Server Security: Lesson #1

    A recent project I have been working on involved a custom built Linux distro running on an ARMv6 piece of hardware. We figured we were fairly immune to getting hacked based on obscure old hardware and pared-down Linux distro.

    Unfortunately, early in development for ease of working on things we chose a guessable root password.  By the time (months later) that we wanted to plug our device onto the internet for testing we’d long since forgot the state that we had left things with the root user account.

    It took just 1 week of being connected to the internet for the device to be hacked and malware installed.

    An investigation uncovered just how unsophisticated of an attack was required to gain access.

    So a lesson was learned by everyone on the team. Basic security precations such as using a strong root password should be made from the start – not procrastenated.

  • The Bots Are Coming

    Slack is the fastest growing startup in history, and it is starting to drive a new kind of interface for users to make things happen within their businesses.

    If you remember way back the only way you could interact with a computer was through a command line.  With this interface you form very specific commands that require a manual for reference. Then with the advent of the mouse and graphical operating systems, these command line tools were mostly replaced with buttons and menus. This opened up computers to a new level of novice users, but there still was a need for manuals and books to learn how to use all these graphical applications.

    Today we have two new possible interfaces for how you can make computers do what you want.  Voice systems like Siri offer a compelling next generation way to talk to a computer with your voice but the technical demands for integrating this technology is still FAR too difficult to make it work with arbitrary custom software.  The other option is bots that run in chat systems like Slack that can offer natural language interfaces that blur the boundary between what we have come to expect of computers and what the future may be like.

    The other night I programmed a quick slack bot powered by the script from The Big Lebowski (one of the best movies ever).  This simple script interacts with the user in a conversational way and took less than 1 hour to implement.

    This type of interaction offers a future where we can casually talk (through typing) to the computer and have it do things on our behalf.  It doesn’t require a manual to know how to specify commands.

    This seems like a step in the right direction towards a broad range of tools that work in a more natural way.  Ultimately replacing the text input with voice will bring us to a Star Trek style future. But until that time comes we will be continuing to develop frameworks that allow developers to create interesting tools that know how to interact with users naturally.

    These bots can trigger plenty of things that might not have been possible with other interfaces.

    >> matt: hey @meeting_assistant, start a meeting with @jack and @jill

    >> meeting_assistant: ok @matt, join the Google hangout <here>, I’ll keep meeting minutes of anything you type in here.

    >> matt: @jack can you contact the client about that thing at that place tomorrow?

    >> meeting_assistant: @jack I added a reminder to your calendar for this ^^

    >> matt: @meeting_assistant, all done.

    >> meeting_assistant: ok, I’ll email the meeting minutes to everyone.

    This sort of theoretical chat bot is now entirely possible and it may just usher in a new age of business communications.  It’s now easy to forego the large enterprise application development cycles and instead focus on small text based services that can easily be deployed.

    The future of this kind of interaction may create a new opportunities to reduce costs and streamline business functions.  By triggering messages based on external events it’s possible to prompt employees to their maintain their workflows, or reward them automatically:

    >> time_bot: @matt, you haven’t recorded your time for yesterday.  please update your timesheet.

    >> overtime_bot: @matt, thanks for staying late, but it’s time to get back to your family

    >> kpi_bot: @matt, your performance has been exemplary this week, we’re sending you an Amazon gift card as a thank you.  Keep up the good work.

    I think this could be a big shift in how businesses run.  And companies that take advantage of this next generation technology stand to gain a tremendous competitive advantage in the coming years.

  • Being A World Class Programmer

    A year ago I started on a goal to write and publish code every day to my github account.  That streak lasted for 250 days.  It was a learning experience that gave me time to broaden and deepen my knowledge of Python.  Though the actionable goal was to maintain a streak of activity, the underlying purpose was to get better at programming (both speed and quality).

    It was in many ways a success.  I learned a bit and got better.  However, as a form of practice it had limits and ultimately improvement plateaued.

    Since then I’ve been thinking off and on again about what it would take to reach the top.  Not the top 1% but the top 10 globally of Python programmers.  Even though there is no measurement of the top as there is in something like chess, it’s clear that as an intellectual endeavour some people know more and can write better code than others.

    Then I read about how Ben Franklin learned to write better.  He started by finding a source of writing he wanted to emulate – in his case Spector magazine articles.  He would read the article sentence by sentence and make notes about each intention and meaning.  After a few days when he forgot about the article he would go to his notes and try to reproduce the article from them then compare against the original and identify where his version wasn’t as good. By digging into why specific differences were better he was able to learn how to write better.  He was able to expand on this technique to improve his vocabulary and techniques for overall structure of a story.

    I have on many occasions tried to equate learning to program with authorship skills.  This is another one where I think programmers have an opportunity to learn something.

    Becoming better at something takes deliberate practice. The keys to effective practice is focus and feedback. You need to identify things that need to be improved and focus on those individually and then get immediate feedback to learn what can be done better.  You then cycle this process for hours/days/years.

    Until reading about Ben Franklin’s technique I didn’t have a good answer for how to objectively get the focus and feedback needed for practice to work.

    The current approach to learning in a professional setting is:

    • peer review
    • books/reading
    • conferences

    Peer review is great if done right.  If people use the review process as a chance to coach instead of just scanning for overlooked bugs then there is a chance things will learned, discussions will be had and the product will get better.  However this is dependent on peers that want to offer this kind of feedback and who are sufficiently advanced to point out areas for improvement.

    Books and reading can be a source of great ideas but reading ideas is not the same as originating them. People learning a spoken language experience this difference.  Exceptional amounts of time spent learning vocabulary, verb conjugations, prepositions and sentence structures can make it possible to understand something they read, while at the same time actually being able to speak a sentence is stilted and slow.  The mental processes for taking an idea and turning it into speech or code is different from reading and understanding what you read.

    Conferences are much like books except you gain a few more mental hooks to hang memories on (I remember learning about that at a conference in SF last year), and the chance to discuss things with peers.

    None of these options really provide a great way to reach the top. But Ben Franklin’s technique for practicing writing may be a hint at something that could actually people achieve elite levels (with a lot of hard work).

    It’s an idea I want to explore further in the coming months.

  • Going Mobile

    Over the last couple weeks I’ve had a growing concern that my office is just not functional any more.  A standing desk in a room pulling double duty as a spare bedroom.

    So over the last week I have pulled out all the office equipment and turned the room back into just a bedroom.  The experience made me realize just how much clutter there is everywhere and it all creates distraction.

    The white tennis shoes scenario.

    There’s a classic book for writers called “The Artists Way”.  One of the anecdotes is about a pair of white tennis shoes.  Imagine for a moment that you have an immense project in front of you – months worth of work.  And in the dead of winter with a foot of snow outside you sit at your desk staring at your computer monitor.  Then, from the corner of your eye you see your white tennis shoes tucked in the corner. There’s a little bit of mud scuffed into the toe;  you look back at your screen.

    Moments later there is a nagging feeling, you must clean those shoes.  Never mind that you won’t be able to play tennis for months.  This must be done now. “It’ll only take a few minutes” you tell yourself.

    Several hours later you’ve cleaned the whole room and accomplished none of your work goals for the day.

    A messy office is full of white tennis shoes and as a result I haven’t been getting much work done for what feels like forever.  To address this I’m taking the unusual approach of getting rid of an office entirely.  I’m focusing on getting my mobile office in better shape so I can easily re-locate to a coffee shop, the bedroom or the kitchen table to get work done.  By just moving to a place that has limited distractions I’m hoping I can stay more focused and get more done.