Matt Warren

Category: Software Engineering

Coding, Python, web development, architecture, and deployment

  • Small Projects

    There’s nothing quite like the feeling of starting a new project idea and seeing it all the way through to finished and published.  It’s a feather in your cap that you can look back on and say “I built that”.  Regardless of if it is a big hit or not, it will make you a stand out – very few people get something all the way to done on their own.

    Ambition can act against you in this.  The larger the project the more opportunties there are to hit roadblocks which derail it. The size of a project is a risk that should be minimized.

    That’s why I believe it’s important to create momentum with smaller projects.  A small win still gives you a great amount of confidence.

    This applies both to home projects, or code projects or hobbies.

    Small is a relative term.  You may be able to handle a small 40 hour project, while someone else cannot yet tackle something that big.  Small may be as simple as fixing a wall hook or creating a pull request to fix a typo in the documentation of an open source project.

    By putting a lot of these small projects together you create something bigger than the sum of them.  Fixing all the small things around your house can turn it into a relaxing home, Contributing to Open Source projects could gain you some notoriety and help you get a dream job.

    Derek Sivers said “the best option is the one with the most options” and doing many small projects gives more options than one big one.

    37 signals (now basecamp) started out with 6-10 individual products. When starting they didn’t know which would be a success so creating many smaller ones diversified their risk and helped them succeed.

    Small projects are going to be a core part of my strategy for 2017.  Launching micro-sites, simple tools, or open-source libraries that can be finished in 8-10 hours of effort.

    Think small, get out there, and finish it.  It’s a step to something bigger.

  • Finding Appropriate Side Projects

    Time is our one major limited resource and so it’s important how we decide to use it.  One of the biggest questions we ask ourselves as developers is often what kind of cool stuff could we do on evenings and weekends.

    Here are my considerations for any side projects that I feel are worth saying ‘yes’ to.

    • It should align with your broader goals
    • It should ideally provide an incremental step towards those goals
    • It should be fun and interesting
    • It should have a number of quick wins
    • It should be something that you can’t or won’t get experience with during work hours

    One thing to consider is conflicts with your job.  It would suck to be working on a passion project at night and find the need to hide it from co-workers and your boss or risk some sort of discipline. Co-workers can be great sounding boards and strong supporters of your success with these personal projects and it’s awesome to be able to take advantage of that.

    The key to being one of those people who seems to get a lot of stuff done is to be continually walking in the same direction rather than running scattered from place to place.  It’s not as easy as it sounds.  There’s always a new hot idea to pursue or a reason to give up on the project you haven’t yet finished.  For that reason it is good to pick a project that hits as many checkboxes as possible for you and that you believe you can see through to the end.

  • Web Design Fallacies and the Failure of CSS

    Having the experience now of building dozens of different web sites over the years I have come to the conclusion that there are some major flaws in how web design is done and the underlying assumptions have proved inadequate to make web design better.

    The first of these is the about CSS.  The goal of CSS was to create a language that designers could use to apply their vision to a website to separate the concerns of design from the logic and structure of a website.  It was meant to be simple enough that someone without a technical background in computer science would be able to write and maintain.  Of this goal, CSS has failed miserably.  Finding a web designer who has a design background but also knows CSS is nearly impossible.  Most designers skills are based in Photoshop and Illustrator, few have any interest in writing or maintaining CSS files.  The flip side of this is that most programmers are tasked with writing in a design language when they would be much more comfortable with JavaScript and few engineers have a good eye for design.

    Another web design gotcha that is easily overlooked is that the web has different constraints from paper.  A static design spec put to a snapshot pdf or image of the final design is only 2 dimensional.  Web design reality is that there is at least 3 dimensions and sometimes 4 or more to implement.  This puts the onus of implementing missing design onto the developer.  How the page should behave at widths of 300px up to 2000px? What interactions should exist? What animations are there? Responsive design cannot be done with Photoshop and so these rendered design mockups represent perhaps only half of the design.

    I would like to see one of two things happen:

    1. Make changes to the CSS tools so that it can be managed in much the same way that Photoshop does.  Fulfil the promise of putting design implementation into the hands of the designers by making the tools used to craft it similar to the tools they’re already familiar with.
    2. In cases where engineers have to do the design implementations provide a method that is more familiar to them.  It should allow for unit testing, it should be designed to reduce the likelyhood of regressions as the design changes. Debuggers and linters should identify unused code.

    I think what we have with CSS is a bad compromise.  It is too technical, daunting and uninteresting for most designers to want to work with it, and it’s too unstructured, untestable and imprecise for engineers to have much love for it.  As a result CSS exists in a no mans land where few people have an aptitude.

    The future looks interesting though.  Will there be a next generation language after SaSS that improves things for the lives of developers, and how will the React.js approach impact the future direction of web design.

  • Programming Should Be More Like Authorship than Engineering

    Most programmers I know come at the discipline from an engineering background and this leads to all sorts of pre-conceptions about what we do and the guarantees we offer.

    Clients often believe that paying for software development is like building a bridge.  You have some meetings to decide what kind of bridge you want to build, send it to get architected and the design approved by engineers and then hand the blueprints off to a construction company to build it.  Once built you expect the construction workers to go away and the new bridge will not need to be serviced for 50 years.  This is the mental model that many people have about how software development works too.

    This is partially our fault and partially a quirk of history. Computers and programming languages came out of engineering and mathematics and the early developers brought with them their background of being engineers.  Now we are deeper than ever into this mode of thinking.  Software patents are treated like inventions.  Teams of programmers try to apply engineering styled development plans to writing software, sometimes with gant charts an other times with scrum.  None of these approaches have proved ideal and as a result expectations often do not match reality.

    Instead I believe that software should be thought of as being closer to writing.  If you commissioned someone to write a book about the Golden Gate Bridge you might expect them to not be experts and require some time to gather research, distil that into notes and further organise their source material before they could really even suggest a structure for the book. If the book was going to be written by multiple authors then sections of the book would be assigned to each person in a way that required little sharing of the same content.  As each section of the book is drafted you might get an early read and have plenty of edit requests to make. Entire sections may need to be re-written if you don’t like the way something is presented.  At the end the full manuscript is again reviewed and edited. Reviewers and Editors expect to find typos and awkward sentences that should be fixed.

    Publishers of the book also usually have a very different pay structure than software clients.  Publishers often provide the writer with an advance to cover expenses while the book is being written and then give a royalty on sales of the book.  This seems like a fair arrangement. Up front payment allows for full-time focus without having to take a part-time job or use credit to live and the promise of royalties provides extra incentive to write the best and most successful selling book you can muster.

    Software development consultants rarely put any skin in the game. Time and materials based projects encourage a consultant to go over time for more money, and fixed budgets encourage them to cut corners to save time. The lure of ongoing maintenance contracts encourages developing something that requires updates and monitoring to keep running.  Notice that there is no direct incentive for the project to be a success, and the developer needs to cover their living expenses until a delivery date is hit. The developer who wrote the software rarely gets any public credit.

    I think it would be refreshing if the publisher style model was applied to software developers. An upfront advance to cover expenses, and royalties from the profits of a successful product would help put the right economic incentives in place.  Books often have an about the author blurb tucked somewhere in the first few pages; websites and applications should do this as well since it’s another incentive to build something you are proud enough to put your name on.

    The business model for book publishing seems like a good fit for software to me. What do you think?

  • Web Scraping With Scrapy

    After reading a reddit thread about interesting scripts people have written to automate things I was inspired to put together a quick web scraping script to check the sales at the LCBO and send me a message (on Telegram) if there is anything interesting.

    Back in the day I would have done something like this with BeautifulSoup, but Scrapy is a full-on framework to help build these kinds of tools. It makes it easier to write a web scraping or spider script in a more normalized way and decompose the scraping into a more consise and easier to read functions.

    This was also the first time I tried to integrate with Telegram as a bot to send myself messages.  Having built bots for Slack and Facebook Messenger in the past, Telegram’s integration was by far the simplest and quickest to get up and running.  A quick conversation with @BotFather and I had my bot created and a key to connect with.  Using the twx.botapi package I was sending messages in just 3 lines of code.

  • Writing Bots

    This past week I had a chance to work on a conversational UI.  It’s a neat way to interact with software and presents an interesting set of problems to implement.

    Before even starting to build something I needed to decide on some tools to help with managing the conversations. There are a small number of libraries and services that can help with the natural language processing which would normally present a technical hurdle.

    I evaluated 3 options:

    wit.ai is a service owned by Facebook that provides some very impressive ways to train a language parser to categorize and extract information from sentences.  It operates as a web service that if you send it a message it will return with a confidence measure and any extracted values.  It provides a level of fuzziness in it’s matching and as a web service it allows you application to improve over time without deploying new code.

    Api.ai is a similar Saas based solution that provides a way to convert a message into a more easily actionable format.  It has slightly different semantics than wit.ai in how you describe the message parsing.

    RiveScript – is the option I decided to work with. It specifies what is comparable to a markdown type language for defining messages and responses. Unlike wit.ai or api.ai it is more simplistic in the language parsing aspects (closer to regular expression matching).  But has the nice property that it doesn’t rely on an external service. Implementations have been written in a bunch of languages which make it easy to get started writing bots.

  • Fun With Chat

    Chat is becoming the next platform. Interacting with computers through channels that you already use to talk to people (Facebook Messenger, SMS, Slack etc) in a way that feels natural is a powerful way to accomplish many things for which web pages, mobile apps, desktop applications, or commandline scripts are cumbersome.

    Chat is not a panacea for all user experience problems but there are many day to day things that might be able to transition to this way of interacting with computers or new ways that as yet haven’t been able to work.

    In some ways this is evidence of more encroachment of technology into taking ever more jobs from humans.

    Facebook has a bot they run on Messenger (available limited to Silicon Valley area) which can answer ‘any’ question and perform any number of activities on your behalf. Right now it is operating on a hybrid model. There is an ever growing list of queries for which it can automatically handle, and anything which the AI is not yet able to handle is directed back to an actual person to do. “what is a good pizza place nearby?” can be sourced from Yelp reviews. “Can you call and reserve a table for this evening” -> maybe can’t be handled automatically and will go to an actual person to do. As development continues the features that can be handled will be incrementally expanded.

    I have a service running to maintain my healthy habits. Persistence is an app that messages me to do things when it can tell that I haven’t. It’s like a dependable friend that holds me accountable to my goals. There’s no reason why all this can’t happen through a communicative medium like chat. There’s no reason why this couldn’t feel like chatting to an actual friend.

    After playing with the Slack and Facebook APIs around chat, it’s proving to be both more difficult and more interesting than I imagined. Difficult because in these early days creating interfaces that process natural language and conversation threads is missing a lot of tooling. Interesting because this is all so new that you really get a chance to build something that no one has really thought of yet. This is a new platform and there is still much to learn.

  • Fear Of The Unfamiliar

    A number of things have come up in the last few weeks for a project I’m involved in that has shaken my beliefs and shifted my perspective.

    The team I work with have all developed a deep level of experience within their domain of expertise.  This is part of the competitive advantage we use to win clients.  Smart people with a tightly focused specialization can work faster and produce better work than a generalist that hops to the new hotness on every other project.

    The problem is that these specializations prevent individual developers from breaking out of their circle of perceived experience without great effort.  Specialization codifies itself into the how the team functions, who can work on what and who cannot work on what.

    When a problem that demands some technology or tools which are outside of the normal comes up it can throw a wrench in things. Sometimes there are justifiable questions – can we support a solution written in Lua if the original developer leaves?  Other times it can devolve into a rather insulting “‘We’ don’t know how to do that” for something that can be learned in a few hours of reading or working through a tutorial.

    The perceived difficulty and risk of something new can prompt only the most senior developers to get assigned to work on new things.

    In the past at various different jobs we did one of two things.  We noticed that the existing technology stack was no longer meeting our needs so we evaluated some alternatives and then everyone was given some insight into the decision, how it was made and then either:

    1. Everyone was given training to get up to the same level of proficiency at the same time
    2. A hard cut over to the new project on a new stack and forced everyone to pick it up quickly on their own

    Most good developers have no problem picking up a new language quickly. A significant portion of the knowlege you have as a Computer Science or Software Engineer is not tied to the syntax of a particular language.

    Part of a good education in Computer Science is experience with a wide range of types of applications.  I did AI algorithms, wrote a real-time operating system, OpenGL and ray-tracing,  web applications along with the basics of algorithms and data structures.  Part of being a great developer is having the breadth of experience to know when to apply certain technologies over others.

    A perceived specialization can negate all that past experience, and hinder an individual’s opportunity to tackle new challenges.  It demands a balance that must weigh company goals and efficiency won from deep expertise with the desires of each individual developer to work on interesting things and continue to learn.

     

  • Writing Algorithms

    Ah algorithms, that fundamental part of computer science education (and job interviews) which seem to very rarely come up in real world, day to day software development.

    Sometimes I have a  real yearning to tackle a technical problem that involves writing an implementation of a red-black tree or a bucket sort, or hell, just write a function that uses recursion.  These opportunities come up so infrequently that when they do it’s a real pleasure to be savored.

    It’s worth making those moments happen if they don’t come up very often at work.  Go out of your way to find projects that create the challenges you want to take on.  It’ll really remind you about what you love about writing software.

  • Chat First Design

    They say that the market for Mobile Apps is now mature. The stores are topped by the big companies – Facebook, Google, Microsoft and the design languages are now more or less stable for mobile platforms.

    The next hype’d interface is chat. For a generation of kids txting is the most comfortable way to communicate and having services that can tap into that already popular mode may have a good chance of sticking.

    Text bots are apps that interact with users almost exclusively through chat applications – Facebook Messenger, Google Hangouts, WeChat, SMS, iMessage etc.  But even though these platforms are relatively new, text bots certainly go back a long way.

    I can recall one particularly smart grade 9 kid in school wrote a computer chat application back in 1995.  Not that it was very sophisticated.. more or less just 1000’s of hard-coded text responses to your questions.  By the time that IRC started to become popular there were plenty of hackers writing bots that would try and pass off as human, or just do useful things.  It’s hardly a new idea to have text conversations with computers over chat.

    Yet here we are in  2016 and the new hotness in UX and interactive design is to go back to text chat. But there’s one thing that is being overlooked.

    Despite it’s long history, it’s still really hard to do a good job.  There are few if any frameworks for building applications like this.  Sure, you can pull out some NLTK code and parse sentences, but writing something to reliably handle arbitrary strings of text from users is still very difficult.

    I had gotten my hopes up, with the growth of Slack and the buzz around chat, only to be dashed when it came time to implement an app idea I have.  Suddenly I’m finding myself inventing algorithms for maintaining the context of a conversation.  There are still hard problems to solve before chat interfaces will be something that are easy to build.

    And until they are easy, they won’t be prolific.

    What the world needs is a Ruby On Rails for chat.  A conventional approach for building complex chat applications that will allow any developer to take their ideas to production.

    It’s something I wish I had time to work myself, but am instead asking the wider community to pick up the torch and help us all out.