Matt Warren

Category: Software Engineering

Coding, Python, web development, architecture, and deployment

  • The Future of Work

    When looking at how automation, robotics and AI will change the future of how people work and what jobs they will have there seems to be two main camps of thought:

    Capitalistic/Free Market view that looking back at the history of economic transitions have shown that each time we faced productivity improvements there were always a corresponding shift in the labor force that maintains the level of employment. Somehow we transitioned from 90% famers to 2% during a great urbanization trend. Later, we transitioned to a largely industrial economy, and most of this industry is alive and well today. Constantly innovating and developing, materials and products are carefully produced using machinery that requires Induced Draft Air Cooled Heat Exchanger Units in order to operate at peak performance under closely monitored and controlled conditions. For our generation, a new industry has flourished as well, that being a knowledge and services one. Not only have we navigated those shifts and today have a relatively low unemployment rate, but compared to 100 years ago, women have entered the workplace. There are more people than ever with a job. The capitalistic view that this has held true since Adam Smith suggests that perhaps it is a universal economic law that will continue to hold in the face of the AI boom.

    The pessimistic view is that AI is different, that knowledge work is the last human product that we have to offer so once it is replaced by an AI there will be no jobs left for us to do. AIs will work all day and night, with better accuracy, faster and smarter than we ever could. The algorithms will chip away at our most demanding careers until one day we reach full unemployment and the world collapses.

    In truth, we have been living in a world of increased knowledge automation since the 80s. Software has been automating people out of a job since MS-DOS. And while software engineering is a significant industry that has absorbed a many of those people, not everyone has jumped ship to become a computer programmer. Just prior to the current pandemic situation, unemployment was at record low levels, despite 30 years of software automation.

    Where people have ended up I think is an indicator of the future we might have to look forward to as the programmers themselves are replaced by AIs.

    The major trends seem to be towards the gig economy, side-hustles, and contracting. Everyone young wants to be a youtube star, and I think the next generation is getting an early lesson in how to promote products. Patreon shows how people have been willing to send money to people who are producing interesting content. This seems like a universal. People will always be able to pay others to see them do something – play sports, video games, build crazy inventions, or simply tell their story.

    The economy will continue to function so long as people find reasons to send each other money.

    Unlike the impossibility of a 1950’s era farmer worker forecasting a future of software developers. Today, I think we can see what’s coming next. People will become natural sales people, refering customers using the same language they pickup from watching youtube and instagram. “smash that like button” or it’s evolution will become even more common as people become independent publishers in big or small ways.

    The future of work is that it will look less like a job, and more like ways we can earn money.

  • Pace of Innovation

    Our ability to be productive is the most critical metric that determines quality of life. It’s a fundamental measurement of how the economy is doing, identifys where we are stagnating, and shows where the biggest gains are.

    There are many factors that go into improving productivity – capital investment in tooling, factories and better software, also education and training. Those examples all suffer from limits. With the best people, using the best tools and methods you’ll hit an upper ceiling of productivity that cannot be surpassed without an innovation.

    Here’s a story I heard about the UK cycling team. In 2002, the UK had an abismal record of only 1 gold medal in the previous 76 years. It had gotten so bad that not only did they have a difficult time finding a bike company to sponsor the team, but several bike companies actually requested that the UK team NOT use or be seen using their bikes for fear of being associated with such poor performance. However, in 2002 they got new leadership from Sir Dave Brailsford who focused intensely on innovating in every minute detail. They found new ways to shave weight from the bikes, new clothing, better diets for the athletes, improved routines, tested supplements, tweaked aerodynamics, and tried various greases and lubricants. Every possible variable that could be optimized was systematically researched for innovations and many of them eeked out small improvements of less than 1% performance gains. In aggregate however these small changes had an outsized impact and resulted in a dramatic turnaround from absolute failure to domination.

    Not only are the individual innovations important but the pace at which those innovations happen can quickly become a massive advantage.

    So what kinds of things can you do to increase the pace of innovation. Here’s some ideas:

    • shorten iteration cycles of build-test-learn-repeat
    • focus on quantity early on rather than doing it right the first time
    • apply project planning on what can be done in parallel
    • be creative at un-blocking people and their tasks – avoid being stuck waiting for external people/services/deliveries etc.
    • replace external dependencies with more reliable internal ones
    • reduce parts
    • simplify or eliminate processes
    • everything is wrong, it is only temporarily the best solution
    • look out for local maximums and have the courage to back out to try something better (one step back, two steps forward)
    • put things in place to reduce the time it takes to get a new person up to speed – good documentation, simpler onboarding
    • always try to find analogues in other industries
    • apply A/B or multi-variate testing where possible
    • know the goal, keep it in focus
    • calculate the fundamental limits
    • periodically step back to get a 10000ft view
    • periodically dig into the minutia
    • use the right words for things
    • make change easy
    • encourage ideas from everyone and everywhere
    • even small changes are worth doing

    I think it’s worth spending considerable effort on things that improve the pace of innovation. Leave a comment if you have more ideas for things that will accelerate innovation personally or where you work. I’d love to collect as many ideas as I can

  • Ideas, big and small

    I have 3 big ideas. The kind of legacy making change the world ideas that are big and bold. These ideas are things I would pursue if I had the means to execute on them but it requires vastly more capital than I have access to, or that others would put me in charge of. And more skill than I have the ability to muster.

    Those big ideas are in the areas of construction, access to space and energy.

    To set myself up to achieve those things in the future I look at what other successful business entrepreneurs before me have done, and model them. It is common to have started with a more modest businesses and leveled up several times before developing the skills and confidence to achieve unreasonable success.

    This was on my mind recently. That before tackling the big ideas I need something that hits exactly the right level of challenge that I don’t get discouraged, but hard enough that I learn what I need, and that is within my means to succeed at. It would be an added benefit if success also brings with it the assets required to tackle a bigger idea next time.

  • Current Thoughts on Software Testing

    When I was a younger developer I didn’t see the value in writing tests for the projects I worked on. I was focused on being as productive as possible during the day to turn out new features and fix bugs. Tests doubled or more the amount of code that needed to be written, it added delays and distractions waiting for tests to run, and sometimes they were actually very difficult to write (figuring out how to mock out things is not always easy).

    That approach works for a while, but has it’s limits.

    There are 3 main things that tests solve for me and it took a while to appreciate:

    1. It catches stupid mistakes earlier
    2. Improves ability to take on or hand off a codebase to new developers
    3. Helps stave off fear of making changes in the future

    Catching mistakes early is accomplished because, tests force you to think through some of the error cases, it makes sure that you’ve actually run the code before launching in front of users. I have come to see unit tests as comparable to double entry accounting in some ways – you write the credit and the debit separately, and then can do a validation step (running the tests) to confirm that both sides are in agreement. This is valuable but it is the least valuable of the 3 points that make tests worth writing. For non-critical software, and if it is quick and easy to identify and fix issues in the deployed software then it’s easy to see how fixing errors after the fact, and writing half the amount of code is a productivity trade off that has a strong argument.

    Being a developer that comes onto an existing project, having tests are a life saver. If you don’t know how to quickly identify and fix bugs, then every bug becomes a lengthy investigation. This process of having new developers have to re-learn how the existing software was built is one of the biggest cost factors in software development. The time it takes to understand how previous developers decided to solve problems in a complex pile of files can be upwards of a year or more. If the average tenure of a developer on a project is only 2 years before they are promoted or move to another project, then the company gets very little time of a developer working efficiently. At this point in a projects life, having tests improves the productivity of developers. Tests allow new developers to catch issues they wouldn’t have known about, helps them learn faster about why things are done the way they are, and reduces the likelihood of introducing a bug that takes a long investigation to diagnose and fix in unfamiliar corners of the software.

    As developers our job is to write software that adds new features and continues to provide value to users and that stays competitive. After a few big bugs have slipped through from lack of test coverage both developers and project managers will lose confidence in the ability to develop new things and make changes to the code. Simple library upgrades and security patches may be delayed or ignored for years out of fear of breaking things. When a team has fear of making software changes it can doom the project unless addressed. The ability to work on innovative new ideas is replaced with more mundane fixes and patching bugs. It won’t take long for things to get so out of date, so unmaintainable and so boring to work on that two things will happen: The best developers will jump ship and move onto more interesting projects and the remaining good developers will start to push for a re-write from scratch. This is a doubly bad spot to be in as a business. The parting developers take with them the knowledge and expertise that’s required to be productive leaving holes in the team that will require new people to relearn what has been lost. And secondly, the cost of a re-write can be massive and not viable, it’s risky and time-consuming. Not addressed, this will rot a project until failure.

    These are the lessons I have learned from working on dozens of software projects over 15 years about the value of tests. I started as a huge skeptic. It took a while, but I’ve come around to see them as indispensable.

  • Global Kick Towards Abundance

    The utopian future portrayed in Star Trek takes place is a post scarcity society where money has lost it’s importance and people more or less have the access to food, energy and housing they need to live as well as the freedom to acheive their personal life ambitions without worrying about where the next paycheque is coming from.

    We’re not there yet.

    But we do live in a time where technology is advancing very quickly, and around the forseeable corner is things like self-driving, automated factories, delivery drones and AI that could smash millions of jobs out of the economy faster than we can adapt. Technology is driving a deflationary effect that is causing prices to drop at an accelerating pace. This has been happening for a while now (though not always apparent) and the only way that the global finance system has staved off deflation is by printing massive amounts of money to counteract price drops. This has been critical to avoiding the situation where the massive amounts of debt in the world cannot be possibly paid off with future dollars that are more abundant due to inflation. For over a decade now, we have had near-zero interest rates and massive government spending attempting to goose some inflation, but it’s been less and less effective at doing so.

    The gobal situation with massive job losses, and work from home, lockdowns are a potential trigger that could cascade into a future where deflation finally takes over and prices start to fall. Suddenly people are being forced to use online shopping, click and collect groceries and finding out that there are a lot of products and services they don’t need. Stores may start to adapt by making contactless delivery the preferred way to purchase. and as a result start scaling back on retail presence. Ultimately the cost savings from this will turn into lower prices for consumers.

    I’ve always thought that things will transition to the point where ultimately the end-to-end creation and delivery of a product will not require humans in the loop – automated mining machines composed of high-quality mining equipment parts or farming or tree cutting will deliver materials to factories built and operated by automated robots, powered by electricity from solar panels made and installed by robots. The only scarcity will come from patents, information access, and land-use.

    What would happen to the cost of electricity if robots could turn sand into solar panels and ship them to your house with zero humans required? What if the robots were open-source designed and mostly 3D printable?

    Despite the unprecedented amounts of money that governments around the world are using to keep the current system going, I’m not sure it will be enough to restore consumer confidence and get people back into their old habits of spending and borrowing and investing. Time will tell if the current experiment with helicopter money directly into people’s pockets keeps things rolling forward for long.

    Right now millions of people are finding ways that working from home can be productive. Businesses are going to be looking at the productivity benefit of having office space and many will come out of this forced work from home with a realization that the ROI on office space is not worth it in some cases. retail shopping will take another shift towards Amazon for everything.

    We will have perhaps a year or more of people investing in ways to continue operating their businesses without needing to have people interacting with eachother with physical proximity. These investments will accelerate towards more robots and drones.

    The march towards abundance is unstoppable. Staving off deflation will necessarily fail at some point as a result. This current crisis will be the kick that forces a re-think of many of our most established institutions and accepted norms.

  • Lockdown Projects

    Making the most of things when stuck at home for the last couple of weeks, and plans for the next while. I’ve been chipping away at various small projects. I’m looking for things that I can get done with the things I have at hand and not going to the store.

    One of the recent purchases I made that has made a good difference was a package of 8 T5 LED strip lights for the garage and crawl space.

    replacing some flickering flourescent tube lights with new LEDs has made the garage a place that is bright and workable.

    Continuing with the garage parjects I modified my workbench to build in a table saw, add hooks for rolls of tape, wheels were added to make it possible to move, and the shelves were improved:

    Indoors I spun up the 3D printer to create a soap dish for the bathroom:

    Most recently I created a small box/shelf so that I can put the tool box on the hand truck and move it around easily.

    Improvements to the garage are turning it into a space that is clean and functional. One of the design motivations I have with the garage is to put everything possible on wheels. With a small space, having the ability to move things around on carts helps make the tools more useful. I can pull things out to the driveway for bigger cuts, and tuck things away to make space.

    There’s more things in the in the plans to get done.

  • The Frequency of Disaster

    In the midst of the current global crisis for COVID-19, I was reminded just how often we have found ourselves in Bug-in situations over the last decade. We have been personally impacted by 4 significant events in the last 12 years.

    In 2018 a tornado ripped through a power distribution station in Ottawa knocked out power for the better part of a week. All our frozen food thawed, we were without internet and hot water for long enough to be worried. The limited communication and power made it difficult to know what was going on, or keep our phones charged.

    In 2013 we were in Calgary for the flood of the century. My office was flooded and closed for several weeks when all of downtown Calgary was under several feet of water. Luckily our house was on high ground.

    And back in 2008 we were in Vancouver when the power substation downtown exploded. This time as well, my office was impacted and shut down for a week.

    These events that last long enough to outlast all our laptop batteries, or the normal week worth of food in the house are where it starts to get into the territory of an emergency scenario. And given just how often these events seem to come up, having some preparation is probably a good idea.

    We’ve averaged one emergency scenario every 3 years, there’s no reason to believe that those are unusual odds to continue into the future.

    Probably not a bad idea to have emergency preparation as part of an annual re-assessment calendar reminder.

  • Making Things Happen

    There’s a limit to the things a single developer can accomplish as part of a project. I get a great deal from checking things off my todo list. You can write great code and propose great ideas, but there’s a limit to what a single person can accomplish without a team.

    A team can accomplish a lot. Multiple people with different perspectives and expertises can pull together in the same direction to build amazing things. Even still there is a limit to what a single team can do.

    Making things happen can take the effort of someone to reach outside the boundaries of the team they belong to bridge the gap between teams. Getting teams to work together towards a common vision is how mountains can be moved. Getting developers to work in coordination with sales/marketing, and management. Getting developers that work on different individual goals to all pull together in the same direction is an amazing challenge.

    Communication is the key skill that can accomplish so much. An individual team member can often perform amazing things quietly working on tasks. But the communication of who you talk to, the words you use, how you connect people and ideas you push out of your brain can build teams, can inspire people, and it can make bigger things happen.

    But saying that communication is the the critical main component to make things happen is not particularly helpful without understanding the vast complexities of the individual tasks. How to run a meeting, the various types of meetings, how to handle difficult conversations, how to distribute ideas more broadly, how to propose ideas, how to shut down ideas, how to keep people on task and focused, how to know what conversations need to be had, what people need introductions and what people should be on a team or taken off the team. Much of this come down to practice, experience and organizing your thoughts. But for anyone who doesn’t stumble onto being good at this kind of thing, it’s very difficult to find books or resources on getting better at them.

    If there was a formula for what to say when to whom in all situations, then humans wouldn’t be relevant. Until then what you say when you say it, how you say and to whom you say it to is perhaps one of the most open-ended problems we can think through. Saying the right thing at the right time in the right way to the right people can change societies or more simply, help you get a promotion.

    In an effort to make things happen, sometimes those things are simple physical things like fixing a squeaky door in which case some how-to knowledge is what it takes. Other times it’s convincing someone to build a small business with you. But getting 1000 people working together is a completely different game, and this is something that I’m spending some more time thinking about.

  • Finished a Game – First time in a Decade

    Finished a Game – First time in a Decade

    For the first time in maybe a decade I played a game all the way through to the finish. I don’t have much time to play games, but I’ve been chipping away at Portal 15 minutes at a time.

    Not sure if it was intentional, but it seemed a little too easy. Like geared towards a teen. But they did a good job of making it seem like you have to be smart to figure out the puzzles. Having challenges that look hard but are actually not gave me a sense of confidence. which I think is part of the brilliance in the game design.

    This was also a game that I was able to play with Ada (5 yo) which was cool. She doesn’t play much games other than baking/decorating cakes on the iPad. This was a completely different style of game-play that I don’t think she wrapped her head around yet. In particular the end of the game is a long sequence of puzzles as you try to escape, the computer voice is trying to convince you not to escape and that it has some cake for you.. but there are scribbles on the walls (presumably from other escapees) that says “the cake is a lie”. She was completely confused about whether or not there was cake. 5 year olds still have a hard time with sarcasm and deception. And it was interesting how uncomfortable it made her.. she wanted me to start over at level 1 and try again rather than finish.

  • Producing Video and Audio

    Over the last several weeks I have been doing quite a bit of learning. One of the things I’m doing more of is going deeper on producing video and audio content for podcasting and YouTube channels. My home office now has enough equipment to do an adequate job with audio and video production.

    My YouTube channel has been reconfigured and re-branded for a podcast and video channel I’m building about the moon.

    The first 2 episodes of the audio podcast are recorded and scheduled for early February. The first video episode is developed and scheduled for YouTube.

    One big lesson in this, is just how much work can go into creating good videos. Assembling videos from stock clips and images is time consuming. It took a full day of editing to make a relatively simple 15 minute video. It is making me think about changing the format to more of a video of me – talking head style.

    In addition to my moon project podcast I am also helping to produce a series of podcasts about meditation.

    All in all, I’m quickly ramping up my knowledge of podcast syndication and publishing, as well as YouTube strategies, as well as how to use video and audio editing software.

    One of my primary goals with all this, as I mentioned before, is to improve my speaking and presentation skills. So far it seems like it’ll be well worth the effort.