I often get into a debate with people about the type and the extent of testing that has to be done for software. One of the key arguments I hear is that developers should focus on creating only the happy path and leave the rest to the testers to find or have SDETs write extensive automated tests while developers can skip unit tests and focus only on writing useful code.
I have used the swiss cheese model to explain to people about the need for layered coverage instead of concentrating all the effort onto one layer. I first heard this term while watching an air crash investigation episode in nat geo. The narrator explains that though there are millions of parts in commercial airliners, the chance of getting into an accident is far lesser than motorcycles.
How an aircraft with millions of parts is able to get back on to the sky within a few hours after landing from a long trip? The answer lies in layered testing. The pilot has a pre-flight checklist, the ground staff do a routine check for visible damages, the maintenance crew notice the hours of usage and do preventive maintenance, there is a strong network of weather monitoring and routing, ATCs manned with trained staff and modern radars, aircrafts fitted with collision avoidance systems, a long series of protocols for ground and sky movements etc.
It is the same across many engineering and medical disciplines. The risk of something slipping through all the layers of test is very rare and if it happens then it is immediately plugged. In software engineering it is a combination of static checks, unit tests, integration tests, automated ui tests, heuristics & exploratory tests, performance tests etc. No one type of test can be removed altogether and replaced by another. We need the layers to move fast and yet deliver what is required.
Image courtesy: Davidmack, Swiss cheese model of accident causation, CC BY-SA 3.0
I went to a restaurant for a buffet which was a bit crowded. I thought the service at some of the live counters will be slow but was surprised that the people at the counters were able to serve large number of customers in a short span of time, especially the salad counters.
I was curious so visited one of the salad counters to see how was it possible for them to serve that many people at once. It was shocking to see that in the name of speed the person at the counter was cutting fruits in such a way that about half of them got thrown away along with the skin and the seeds. If this person had maximized the fruit content then it would have taken 4-5 times the amount of time taken now on fast cuts.
It is very clear that the restaurant can afford to waste as much of half of the food because they still gained from servicing a lot more people than operating efficiently. It was optimized for time not cost.
This is something people don’t understand while choosing tradeoffs, people often choose both cost and speed as key without giving a second thought that both cannot go hand in hand. If the same set of people had to do things much quicker and at a larger scale there has to be expenditures in tools, training and also some change in processes where there will be huge wastes before optimization kicks in. This is what happens in software development teams, often there is a tight budget and an impending doom if something does not happen; leading teams to easy burnouts.
I did not include the word quality here as it is non negotiable, you can do things quick and cheap but with a poor quality of work like serving the fruits with seeds still intact or skins not peeled well. That is not work done, there is no work without quality; eventually it drives away customers.
Next time when you have a debate about speed consider moving the cost sliders.
If you are a salaried person, then imagine that you are a business owner who serves only one client at a time. Won’t you keep looking to improve the product/service you offer, won’t you keep looking at emerging trends and stay on top of your business domain, won’t you find out who your next prospective client is? It is not that hard to answer these questions, a business owner always has to stay aware of the market and stay relevant.
When it makes sense as a business owner to explore the market, find the options and stay relevant; why should salaried employees not be aware of what is going on around them? Each salaried person should always stay aware of the market and keep themselves relevant. I have often come across people who scamper for opportunities or rush to update themselves when they are either disgruntled or unable to perform with their current employer. This will lead to people making bad decisions as survival instincts kick in to stay employed. Your best decisions will come to you when you are least stressed.
If we treat our employment as a business that has to be sustained, we won’t be relying on the learning & development programs of the companies. We will constantly be in touch with the industry and will continue to stay aware and stay relevant. To be aware of what is going on in the market, time to time conversations has to happen about opportunities like any other business. If we are aware of the market, then we are aware of our abilities and will shape up quickly to always be a market fit.
This is a follow up of my previous post Gamers, musicians & artists. The TL;DR of that post is; people tend to self organise, find what they have to do and also hold their peers accountable to an expected standard. All they need is to work in small teams, some clarity on what their objectives are and leave them to be in a state of flow where they are constantly learning & improving while going behind their objectives.
I used to go to my college by town bus service. Town bus services are mostly run by governments. There are no incentives for the government employees to run a great service; but the ones we took to our college were run by enthusiastic drivers and conductors who treated their buses as their second home. These guys kept the buses squeaky clean and in best shape though it was an old bus, they pooled in their own money and added a radio when new FM channels were introduced. They would enquire about our health if we did not turn up for a few days. They would also signal us through the horn when they approached the college gate so we could run outside the blind spot and catch it on time.
The students became fans of these services and started celebrating bus day every year. All the regular passengers got together on a day along with the staff and distributed sweets & gifts. I still feel these guys will be running their buses like how they did before even though they are nearing their retirement age.Nothing prevented them from checking in and out, run a poor service and still be paid for; but they wanted to enjoy something they do day in day out and went for an ever improving quality of service even when they were not expected. I have rarely seen these people fall sick even though the climatic conditions varied from cold to too hot within few months.
Software development teams also if left on their own, will always try to punch above their weight when they buy in to the purpose they have signed up for. The reason people will stay engaged and go for it is a light tension to constantly be challenged, learn to overcome failures and the support to bounce back stronger. People need to have business knowledge, be competent in their tech domain and also have room for failures which promotes creativity. Eric Schmidt in one of his talks mentioned that his teams always try to shoot for the stars, but many times they fail and land on the moon, but landing on the moon itself was impossible and at times it seemed to be more easy to reach what was impossible, when work was allowed to happen. He calls his people smart creatives and there are people to facilitate decision making to make them go fast.
I always quote this from Semco’s management style. They call their managers ‘Facilitators’ who help the team to set unimaginable goals and equip them to go behind. To go behind audacious goals, we need to communicate well to the teams and make sure they are equipped both technically & people wise; which will then let work happen itself with very less intervention. By trying to make work happen through a great plan, we will be predictable and deliver as per plan week on week and quarter on quarter but it won’t be anything disruptive or far fetched. Even if we achieve a great objective tracking to a plan, in an ever changing environment plan will soon become obsolete. There is this cliched story of cathedral builder vs person laying bricks. Everyone would want a cathedral builder, but many will choose the cathedral builder to just lay bricks as told by the planner.
Let work happen and see the magic
When I joined ThoughtWorks, a common scene I witnessed in the evenings were playing games. Age of Empires was the most commonly played one and the gamers were very fanatical about it. When I wanted to play, I was given a chance to take part but when the fellow gamers found out I was a rookie, no one wanted me in their teams. They forced me to be at some level so that I can play along with them and be part of their teams, I was made to train myself in the weekends and then get back to playing with them. Same for other games like Cricket. If there was some level of seriousness, then better be prepared.
There was also a music band, I had practiced playing movie songs on a computer keyboard and some software for a very long time, may be around 8-10 years. So I thought I had a hang of music when I expressed interest to join the band. I could readily see that I was moved out of singing, drumming and keyboard to merely being an MC for the event as I lacked skills that were good enough for an office band. One of the band members made me join a music school and after about two years of practice I was finally taken back into the team. I have seen this among other disciplines of art, people are not ready to take you into the team if you cannot match up to their expectations.
Workplace is different, because it is does not have an easy way to measure productivity or understand expectations. Basic expectations of being a good team member is understood well, but when people don’t see those behaviours generally it is tolerated or seen as the job of the manager to interfere there. There could be many reasons to this one, one of them is work is seen as a job description to be fulfilled for a salary; not something that is done because you want to do and you are compensated for it. Workplaces won’t be difficult to run if people know what to expect from their team members and be very strict with the outcomes just like gamers, musicians and artists.
I was cruising down the highway around 110~120 kmph, though the car was capable of running at 150+ kmph I chose to keep it below 120 as the thought at the back of my mind always says it is not going to stop quickly or control the direction well in case I need to. When I am very sure that I have an open & straight road, I test the limits of the car, but will quickly pull back to manageable speeds when a turning comes in sight. During one of those high speed bursts of 160 kmph, a sports car overtook me. It did not just overtake, instead it zoomed past and disappeared out of sight. Enjoying speed was not much about the road, it was the control available in a vehicle for a driver. Sports cars don’t just go fast, they turn well, stop quickly and have lots of safety bits to protect occupants from a crash. You could ram a sports car at a high speed into a wall and walk away from the crash. If I use my passenger car downhill at 200 kmph (which I still can), that is insanity; it is not going fast.
It was when I had these thoughts that I stumbled on an article pointing out that developers who are eying for speed often compromise the safety aspects. In software development there are plenty of aspects to take care. In simple terms it is taking a problem and solving it using computers by people with various skill sets. You have analysts, developers, designers, operations etc. The very nature of different people getting involved means there is lots of communication, if there is lots of communication between people of different skill sets then there is translation loss. If there is translation loss then there will be misunderstanding and rework. If you need to rework often, then the speed at which you can code matters. If speed matters, then better be safe.
Test harness consisting of unit, integration and functional tests, static analysis, performance checks, automated deployments, coding practices all together form the safety package for software development. As the code base grows and the number of people increase the more important the safety checks become. It will always be tempting to avoid the process and get something out quickly but the price to pay will be bad. There is nothing prudent in crash landing.
Another aspect of speed that is also often compromised is sustainability. The common example given to agility and speed is Cheetah, Cheetahs can maintain its top speed only for about 90~120 seconds followed by a long dip in physical activities. Any activity that requires a spike in the output is followed by a dip. There is nothing called sustainable peak performance.
Violating safety or sustainability of speed removes control out of the equation, it makes sense only if we are crash worthy and have the energy and resources to get back to normal. Speed for the sake of speed will thrill and eventually kill.
Toyota introduced the andon cord in their manufacturing lines to help people to stop the production line and alert the people around if an abnormal situation arises. It is a cord that hangs above the head within easy reach of any to help to immediately gather the attention of the others. While the rest of the industry was treating production line is something that never should be stopped and let quality control take care, this one put power in the hands of people on the line to take a call to improve quality by attacking the problem at its source.
The idea then widely got adopted in different forms like ‘Help’ buttons in various manufacturing sectors. It was easy to adopt in manufacturing as we can see what is going wrong and gather together to immediately fix the problem. In software development it is not obvious when to pull the andon cord and how to collect thoughts of people on what to solve.
Dev huddles are the answer to the andon cord in software development. Huddles are very common in any team sports, team members quickly huddle to celebrate a goal or discuss a plan. The team also optimises over time to communicate very effectively in fewer words and quick time.
When to call for a dev huddle?
We need to call for a dev huddle when
- Our programming has deteriorated from a flow to brute force or trial/error method. Manuals did not help to resolve and even a quick help from another team member did not help.
- We find a badly developed code and need to bring it to attention of the team when it is still fresh in mind.
- We are about to make a major change in the code base and every body needs to be aware of the incoming change so they are not surprised.
How to make sure that we don’t waste other’s time?
Team time interruption is expensive, so before calling for a dev huddle we need to make sure that we have all our show and tell pieces ready for discussion. One example is when stuck at a problem, quickly jot down the problem and the steps tried to resolve that did not work on the whiteboard in diagrams or words; then call for the huddle. If the resolution or direction is not found within 10~15 minutes, break the huddle to come and do a detailed research on the problem.
Why should dev huddles work?
Dev huddles work on the idea of crowd wisdom, the average output of a group is always higher than best individual in the group in most cases. Ideas & solutions can come from any one provided they are given a good explanation about the problem. Dev huddles also help in knowledge sharing in a terse & effective manner.