If you enjoy Software Engineering Tidbits, it would mean the world to me if you invited friends to subscribe and read with us. If you refer friends, you will receive benefits that give you special access to Software Engineering Tidbits.

How to participate

1. Share Software Engineering Tidbits. When you use the referral link below, or the “Share” button on any post, you'll get credit for any new subscribers. Simply send the link in a text, email, or share it on social media with friends.

Refer a friend

2. Earn benefits. When more friends use your referral link to subscribe, you’ll receive special benefits.

• Get 30 minute Zoom chat for 50 referrals

• Get 1 hour Mock interview for 65 referrals

• Get access to my 3-day live course - Software Engineering from the Frontlines - for 80 referrals

Visit the leaderboard

To learn more, check out Substack’s FAQ.

Thank you for helping get the word out about Software Engineering Tidbits!

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

I'm excited to announce that a new cohort is now open for enrollment!

This course is divided into three sections:

1. Building Software (Code Reviews, Unit Testing, Engineering Requirements Document)

2. Maintaining Software (Observability & Monitoring, Release & Outage Management, Migrations, Monolith/Microservices)

3. Team and Culture (How to onboard team members, How to conduct an interview, How to scale yourself, Software Engineering Career Ladder)

You can find the complete syllabus and enroll by visiting this link: https://maven.com/georges-el-khoury/mastering-software-engineering

Cheers,

-Georges

Today’s post is about how to release software to production.

First, we will cover the required steps before landing a pull request to the main branch.

• Build is green

• Linter is green

• All unit tests are passing

• Pull request code reviewed and approved

  Software Engineering Tidbits

  Code Reviews

  One thing you will spend the most time doing in your software engineering career is code reviews. Unfortunately, it is not something software engineers learn in schools or are specifically mentored about. Below is a detailed guide on how to conduct a code review from both the perspective of the author and the reviewer…

  Read more

  4 years ago · 10 likes · 1 comment · Georges El Khoury

• Storybooks are passing (Frontend only - Optional but recommended). These are unit tests that capture unwanted changes in the UI

• Integration tests are passing (Optional but recommended). These are high-level tests that invoke the top-level APIs on environments similar to the production. These tests usually have minimal or no mocking at all. Ensure they run quickly and are not flaky; otherwise they will degrade the developer experience. No engineer likes to resubmit a pull request.

Once a pull request is landed to the main branch, it is time to deploy it to production. The main recommendation is to deploy to the fewest number of machines first and then gradually increase the number of machines to the deployed version.

The deployment usually follows the below patterns:

• Canary environment (few machines that get the deployed version first)

• Data centers/regions with 1%, then 2% then 5% of the machines at a time. As an example, you first deploy to us-west, then us-east then Europe, etc. It is good to be able to fasten the deployment if necessary through a configuration. This should only be used during an outage mitigation.

Between every step, you either have a manual approval to move to the next stage or automate the approval based on metrics/instrumentation mimicking what the engineers will be looking for to approve deployment to the next stage.

Quick note for mobile apps. You want to release internally to employees first and then also have a gradual rollout on the App Store and Google Play.

The next question is who does the deploy and at what cadence?

At one extreme is a CI/CD (continuous integration/continuous deployment). This means the second a pull request is landed, it gets deployed in the way it was described above.

There is a lot to like about CI/CD.

1. It ensures you have one pull request/change per deployment. This makes it easy to detect the root cause during an outage.

2. It puts the deployment on autopilot, freeing development time

3. It incentives engineering excellence, especially in terms of observability.

The main disadvantage is that it increases the chance of deployment to happen unattended and it couples the landing and the deployment, which is sometimes not what you want to do. Sometimes you just want to land the pull request but not to deploy.

At the other extreme are scheduled deploys (e.g daily or weekly), usually done by the on-call engineer. These are not desirable for many reasons primary among them is that they pile multiple pull requests per deployment, increasing the odds of an outage. In addition, they create a bottleneck on the on-call engineer where everyone starts to ask them when they are going to deploy and whether they have deployed yet.

My preference for small startups is to have CI/CD and to move to on-demand deployments done by engineers once the customer base is bigger. I think it’s the best of both worlds. It ensures all deployments are attended to and are happening when desired. It minimizes having many pull requests per deployment as they happen more frequently, although it still allows deployments to have more than one pull request at a time. Finally, it does not create a bottleneck on on-call engineers, etc. since every engineer is able to deploy whenever they want on demand.

The last topic to discuss is the different environments. I recommend the following:

1. Development on a developer machine. Big bonus point if this environment is runnable in only one command (e.g. npm run dev boots up everything needed on the local machine)

2. Staging. This environment is similar to production with a non-production database/auth/configuration, etc.

3. “Next”. This environment is part of production but is usually not exposed to the outside world.

4. Production. This is the public environment used by users/customers.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

In fact, it is a prerequisite.

Software Engineering Tidbits

Software Engineering Career Ladders

This is my take on software engineering ladders and what it takes to be promoted from one band to another. Software Engineer I: You are tasked and are able to deliver a feature part of a bigger release. Special skill to acquire to be promoted to the next level is clear communication and status update…

Read more

3 years ago · 3 likes · Georges El Khoury

In an office environment, it will be someone almost always at your desk. In a remote one, it will be continuous internal messages on Slack/Team.

So how can you support everyone and at the same time deliver on your deliverables? How can you scale yourself?

Well, the most obvious one, and not the one I am recommending is to work two shifts. One for the company and one for you. This is directly inspired by Paul Graham's classic essay: Maker's Schedule, Manager's Schedule. A must-read.

To more realistic options, one easy win is to pack all the requests into weekly or twice-a-week office hours. You can redirect all requests to the prescheduled time slots and address them during this time.

Another option that will pay off long term is to direct all questions to a wiki. You can automate this in a Slack/Team channel. StackOverflow for Teams is a good solution I have used in the past (not an ad). With time, you can encourage people to start searching there first and this will create a good knowledge base for everyone at the company.

Taking this option further, investing in a good internal search that will surface wikis, docs at various locations, slack messages, etc. will pay up a premium in time.

At one point, you will have to delegate. This is the only way to be able to scale. First, I recommend you do not delegate high leverage activities such as interviewing (minus initial screening maybe). Another thing I recommend you do not skip doing is 1:1s. You want to keep a pulse on the organization especially now that you are focusing more and have less visibility on the overall picture. A third area that I also recommend you do not delegate is whatever gives you a pulse of the business and allows you to get product feedback. These are still important to get and are really valuable.

How/what to do delegate?

If you have on your team a solid engineer II or senior engineer that you respect, give them more responsibility and start directing all requests to a certain area to them. They will be the first point of contact and you will be the second tier of support. This should free up a lot of your time. As an example, if you are owning a full stack, start sending all front-end (or back-end) requests to another engineer on your team or if you own two areas, start sending all requests related to one of the areas to someone else.

You can also delegate to your manager or counter part in other disciplines. A good manager will gladly take some your responsibilities to allow you to focus more on critical initiatives. These could be things like capacity planning, budgeting, nudging different teams to do something like upgrading to the latest library, etc.

This is hard to do especially for individual contributors. Managers are more experienced in this and can do it better. You have to resist the urge to be the first to respond and allow the new owners to assume their responsibilities. This being said, delegating this way has multiple values for the organization. First, it frees your time to be strategic on how to make an impact or have a better work/life balance, it removes you from always being a bottleneck, and provides more opportunities for junior engineers to grow and make an impact of their own.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach similar hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

In the code review post, I recommended to help the reviewer help you (the author) by making it easy for them to approve the pull request.

Software Engineering Tidbits

Code Reviews

One thing you will spend the most time doing in your software engineering career is code reviews. Unfortunately, it is not something software engineers learn in schools or are specifically mentored about. Below is a detailed guide on how to conduct a code review from both the perspective of the author and the reviewer…

Read more

4 years ago · 10 likes · 1 comment · Georges El Khoury

Today, we will cover a bug report in a way that will help the engineer help you (the person who submitted the bug report) by making it easy for them to quickly find the root cause and fix the bug.

This is the outline I recommend.

Clear and concise title

Priority: How quickly should we tackle this bug? Usually a range between 1-5.

Severity (optional): What is the impact of this bug? Usually a range between 1-5

Repro Steps: Describe every single step needed to repro the bug.

Expected Result: What was expected to happen? What is the expected behavior?

Actual Result: What is actually happening

More details: Add as much details and context (user, environment, version, platform, time, repro once or every time, repros consistently or not, etc…) as possible. Add any screenshot that will be useful. Finally, attach any logs, network requests/responses and instrumentation that will be useful for the engineer while debugging.

I have seen people record their screen and comment which is really helpful for the engineering team. Loom is great if you want to do this (not an ad)

Software Engineering from the Frontlines on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

1. Idempotent: It’s every engineer favorite word :) Meaning it can be run over and over and is always green.

2. Independent/Atomic: Meaning it does not dependent on any other tests to run before and can be run at any order or randomly within a test suite

If we are not able to achieve the above, then our unit test will become flaky and a bad (very bad) unit test. It will be flagged with an ignore tag or worst either commented out or deleted all together.

In addition, our unit test should be fast: It should run quickly so it does not delay the development process. This means that all the unit test dependencies should be mocked.

Side note - the only place where not mocking might be okay (and it is debatable) is for an orm heavy app where we can launch a database in memory. I still think it should be mocked but there is a counter argument to be said that not mocking might lead to a more accurate test through relying on actual db data and operations rather than orm mocks.

The unit test title should explain what it does. I recommend testWhenShould pattern in the title but other patterns are okay too. I also recommend having three sections: setup, execute and validate.

In the end, your test should be similar to this:

test*When*Should* {
//setup
.
dependencyOne = mockDependencyOne.return({…});
dependencyTwo = mockDependencyTwo.return({…});
objectToBeTested = new ObjectToBeTested(dependencyOne, dependencyTwo);
//execute
result = objectToBeTested.someFunc();
//validate
assertEquals(result, expectedResult)
}

Unit test code is no less important than product code and should follow the guidelines of elegant code too.

Software Engineering Tidbits

Elegant code

Elegant code is hard to put your finger on, describe or quantify. In his book clean code, Robert Martin spent a full chapter with multiple people trying to do just this. Each of them ended with different definition. Here is my take on it: Clean code is readable, sectionable and effortlessly extensible…

Read more

4 years ago · 4 likes · Georges El Khoury

Finally, one last tip I posted about before. When you fix a bug, check why the previous unit tests did not uncover it and update them accordingly.

Software Engineering Tidbits

Fix a bug / Write a unit test

One solid practice to have while coding is to fix a bug and at the same time write a unit test that would have uncovered it. It is a sign of solid experience in software engineering. The direct benefits are to avoid regressions for this bug, to ensure the fix is correct and to increase code coverage and the reliability of the test suite…

Read more

3 years ago · 3 likes · Georges El Khoury

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Software Engineering Tidbits

First things first

Returning a service to a green status is the priority in an outage. This mean looking first into rolling back recent changes or falling back to unaffected regions. Avoid the temptation to forward fix or spending time to identify the root cause before you do this…

Read more

4 years ago · 6 likes · Georges El Khoury

You should first try to rollback to a previous deploy or configuration if the outage matches with a deployment time or a configuration change.

In addition, you should check if the outage is happening in all data centers/regions. If not, you should fail over to the healthy one.

Anything you do while failing over or rolling back should be done gradually as this might cause further deterioration or introduces new problems.

If this solves the issue, you can breath a sigh of relief and catch up your breath. Otherwise, it is time to start digging.

If you have a secondary on-call, it is a good idea to get in touch with them. Your team culture should encourage this. Handling outages alone is unhealthy and everyone benefits from an extra set of eyes.

If the outage is a high severity one, you should start an incident page, join a zoom outage and start actively paging folks and on-call engineers to join the meeting.

Next, you should try to find the health of your downstreams, upstreams and see if anything there is unhealthy. After this, you should see what changed in your systems. You should check deployments, configuration changes, experiment running, logs, etc…

If this still does not help, you should try to get a repro either directly or reach out to impacted users and try to see what is special for them that might be causing the outage.

You should try to identify if the impact is effecting all users, some users, regions, features, etc.. This will help you close in on the root cause.

In parallel, you should also deconstruct your systems and try to see which ones are healthy and which ones are not

Software Engineering Tidbits

A good way to debug

One of the best software engineering tip I received is a good way to debug. Once you are investigating something broken, you keep peeling/removing functionalities until things work again and then you add them back one by one until things re-break again…

Read more

4 years ago · 8 likes · Georges El Khoury

Communicating during the outage is important. If you are the incident manager (we use to call them commander at Uber), you should keep iterating the knowns/unknowns, update the incident page when possible and delegate tasks. Staying calm and level headed is crucial.

In reality though, the best way to prepare for outages is through work you do prior to them happening. Things like instrumentations, dashboards, rollout strategy, redundancies, etc… Engineering excellence in your organization is highly correlated to them.

Finally, no discussion on outage management can end before we talk about Post Mortems. It is such an important topic that I dedicated previously a post to it by itself.

Software Engineering Tidbits

Blameless Postmortem

Putting together a blameless postmortem culture is one of the most effective way to improve quality in an engineering organization. One of the best manager I had once told us that the only sure way not to have an outage is not to deploy to production and I surely expect you to ship and deploy a lot of features. The only thing I care about once we have an…

Read more

4 years ago · 1 like · Georges El Khoury

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

1) Declaration close to usage

Instead of declaring a variable and use it way below and have random code in between, have the declaration and the usage as close as possible.

2) Guarding

Instead of having code inside an if and maybe an else, do a quick one liner guard check in the beginning of a function and have the majority of the code unnested outside of brackets in the top scope of the function.

3) Reduce the number of parameters in a function

A function with a long list of parameters usually mean that it is highly coupled and doing a lot of things. 1, 2 or sometimes 3 parameters is okay but more usually mean that the function or the parameters need refactoring.

4) Naming variables

A lot of time naming variables to exactly describe what they do goes a long way into making the code more readable Also breaking logic and encapsulating them into intermediate variables improves readability too Do not be afraid in having long variables names.

Did I miss anything? Feel free to add them in the comments.

I will keep adding to this list on this post.

If you want to stay up to date, subscribe below to the newsletter.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

For software engineers, our code base is our station. This is where we spent 8 to 16 hours a day. In my career, I have seen around 10 code bases and I know for a fact that team morale was much higher in a well maintained code base. It felt good to come to work and work on clean code base. Messy code bases are out there and they might be generating huge revenues. Yet, just like a messy kitchen, they will require great energy to find anything in them or to get anything good out of them. This is no environment for a chef, a software engineer or anybody to be in.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Mastering Software Engineering” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Layoffs are necessary when the future of a company is at risk and need to reduce its expenses to survive and compete in the marketplace.

They are unfortunate but a necessary evil for the continuation of the company.

Managers in layoffs have two responsibilities:

First to deliver the news to the impacted employees.

Second is to reassure and reenergize remaining ones.

It is important to be compassionate but direct while delivering the news. A new manager should practice their message prior to the announcement. They should explain the why and the how for both the impacted and the non impacted who will be watching closely.

It is crucial to identify who was and was not impacted as soon as possible. Announcing layoffs and lingering on identifying who will be impacted is absolutely awful for the productivity and the morale of the team.

Hence the importance of tight lips while planning the layoffs in upper management.

It is also important to communicate clearly and have management available and accessible for the larger team. This is a time of high uncertainty and having management around is valuable and comforting for the remaining employees.

A generous severance when possible is also incredibly helpful. It shows that the company cares about its employees and will be appreciated by the impacted and non impacted ones

During Airbnb layoffs in the beginning of the Covid pandemic, although I was not impacted, I witnessed how generous they were. I think they pioneered accelerated vesting for the impacted employees. I will never hesitate or feel bad about staying in an Airbnb because of this.

On a personal level, the manager should reach out to his personal network and try to refer impacted employees especially the good ones that were just impacted for financial reasons.

This goes a long way in helping impacted employees and build a lasting relationship with the impacted employee and the manager.

After doing the announcements and going thorough the layoffs, a manager should move into comforting the existing team members and explain to them the roadmap and how the company is set for success having gone through the layoffs.

Another incredibly important practice here for the credibility of the management and the health of the company is to avoid doing layoffs multiple times.

It is extremely destabilizing for the company. Hence the need to cut deep and wide when doing layoffs to avoid the need of doing another one in the future.

Companies that goes through layoffs when done in a compassionate and correct manner might emerge stronger, leaner and more competitive out of them.

Unfortunately the opposite is also true and when done incorrectly can lead to loss of talent and spiral the company into bankruptcy.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Software Engineer I:

You are tasked and are able to deliver a feature part of a bigger release. Special skill to acquire to be promoted to the next level is clear communication and status update.

Software Engineer II:

At this level you should have delivered multiple features with high quality including unit tests and instrumentation. You should be viewed and have the reputation of an engineer who can deliver consistently with high quality with minimal guidance. You should also start mentoring interns or new engineers and start designing end to end features as a stepping stone for the next level. You are a great asset for your team.

Senior Engineer:

This is a terminal level at many companies. At this level, you should own a critical area in the product. You usually own a code area or repo and code review changes in this area.

You are battle tested and have some scars to show for it usually from over engineered systems. You are the go to person and the authority for an area of the product and receive multiple requests from different people looking into integrating into your area or needing support.

You have been a commander in outages and are one of the few who can return a system to a green status especially in your area of expertise. You mentor junior engineers and are able to drive the design and release of a complex product or set of features.

You set patterns or design software that are leveraged or adopted throughout the company. You are called upon for feedback and advice from upper management. You are a valuable asset for your company. You leaving the company is a considerable loss for the company.

Staff Engineer:

This is a continuation of the senior engineer role but your charter is now multiple teams and orgs. You avoid duplication and pursue opportunity for consolidation of redundant systems in multiple team. You build systems that improve developer velocity.

You have clout in your organization and lead by example. You set an example of good behavior and reprimand bad behavior. Your endorsement lead to promotions for junior and senior engineers.

You are able to architect and deliver end to end products from infra to user facing including developer experience, operations and release management. Example of deliveries is to move APIs from Rest to GraphQL or migrate a repo from a monorepo to microservices.

You leaving the company is a tremendous loss for the company.

Principle Engineer:

This is a continuation of the staff engineer role but you are closer to the executive layer at the company with a direct access to the CTO. You can prioritize initiatives and impact budgeting.

You are tasked with initiatives that are critical to the company. Failure to deliver on them have serious repercussions on the competitiveness of the company. You can reach deep and wide and commandeer any resource or engineer when pursuing company critical priorities.

Example of deliveries is moving from on premise infra to the cloud or responding to the emergence of web, mobile or AI as an example.

You leaving the company has financial and competitive impact on the company.

One last note is that every engineer wherever they are and no matter how successful in their careers will remember the manager that promoted them (knighted them) into their new band. This is a manager that recognized their contribution and trusted them into new responsibilities.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

The way a final interview loop happens is that you will be part of a group of engineers that will be each interviewing a candidate separately for one hour. After this, you will meet your colleagues in a debrief meeting where you will decide collectively whether to extend an offer or not to the candidate.

The worse you can do for the candidate, yourself and your team is to have no opinion in the debrief meeting. You would have literally wasted everyone’s time and been unfair to the candidate.

What this means is that the second you enter in the interview room you are actively collecting positive signals that the candidate will be a great team member or negative signals that the candidate will not be a good fit for the team.

First, ask them about what they are currently working on. If they are in college, ask them about their final project, etc… This will allow you to gauge their experience, command and complexity of their current work. Ask them follow up questions, ask about their learnings, give them space to tell you about their experiences. Unfortunately, a lot of people do not ask about this and miss a great opportunity to learn a lot about the candidate.

Second, if you are doing a coding round, never ask an aha question, one of these that needs a trick or a workaround that once you get, you would be able to solve the question. This usually do not tell you much. You also want to minimize algorithmic question. They are hard to do in one hour and do not mimic how engineer think about algorithms in their day to day work. What you want is to get them coding as quickly as possible. You also want them coding on their IDE rather than a white board. Conducting an interview on a white board will be a missed chance to collect valuable signals on their debugging and troubleshooting skills.

Remember, any chance to collect signals increases the odds of a successful interview.

If you are doing an architecture interview, allow the candidate and give them space to show their experience and knowledge of different patterns. Ask a lot of follow up questions and try to gauge and connect their architecture to their work and production experience.

In the end, leave a little time for the candidate to ask you questions about the team and company. In addition, if they did really well in their interview, use also this time to sell your team.

Before we go to the debrief, two things to note. First always be respectful, only focused on the interviewee and never ever cut an interview short. Interviewees are under a lot of stress and being emphatic to them is of utter importance. They are also current or potential customers and will not forget a bad experience (rightfully so). Second, be open minded especially with candidates with a non typical background.

In the debrief, come back with one of the four options:

1. Strong Yes. You would fight for this candidate and hiring them is a competitive advantage for your team.

2. Yes. They are solid and will be great team members. We should definitely extend an offer to them.

3. No. Unfortunately, anything that is not a solid yes should be a no. Hence your weak yes needs to a be a no. Due to this, you should end up in the long run with a little more noes than yeses.

4. Strong No. This is the opposite of a Strong Yes. Basically, in your debrief, you should go out of your way to explain why this candidate is a bad hire for your team.

Good luck in your hiring!

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

It is a sign of solid experience in software engineering.

The direct benefits are to avoid regressions for this bug, to ensure the fix is correct and to increase code coverage and the reliability of the test suite.

The indirect one is to promote a culture of quality and reliance on unit testing in the engineering team.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

I am grateful to have started this newsletter this year. It has given me a medium to share my technical posts and publish them online.

Second, I was not planning to share this but it ended up being a great technical discussion on many software engineering topics and worth sharing. I was part of a Twitter Space to discuss the implication of Elon Musk acquiring Twitter from an engineering perspective with Bassem, Ahmed and Mohamed. Bassem is a great host and contributor and Mohamed and Ahmed gave great insights too.

You can listen to it in full here: https://twitter.com/i/spaces/1BdxYybmBONxX?s=20

I am hoping to hear back from you in upcoming posts. Do not hesitate to push the reply button and get back to me.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Mastering Software Engineering” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Below is a step by step guide on how to approach it.

First step is to standardize and consolidate the API of the old system. This is an important step and critical to the success of the migration.

Second step is to implement the API of the new system.

Third step is to invoke the old system and the new system keeping the old system as the source of truth. In this step, you compare the old and new system output and fix any discrepancies between them. This is called Shadowing.

Most of them will be from bugs in the new system but do not be surprised if it is also due to existing bugs in the old system that got surfaced by the shadowing process.

The longer you shadow the more confident you can be of the new system. This comes at the cost of delaying the migration of the new system, maintaining the old system and calling both the old and the new system at the same time.

A huge problem in the shadow phase is what to do with new features? Do you setup a feature freeze on the old system? Do you tell teams to wait for the end of migration and implement in the new one? Do you implement in both?

This is why mostly shadowing needs to run for a limited period only.

The forth stage is similar to the third stage but uses the new system as the source of truth for the returned output. This is called Reverse Shadow. The benefit of this stage is to know that it is possible to fallback to the old well tested system in case of an outage.

The fifth and last stage is to just call the new system, remove the shadowing and reverse shadowing logic and deprecate the old system.

Bonus point is to have a logic to do gradual or targeted rollout from the old system to the new system in the shadow and reversal phase

Another bonus point is to have an ability to compare and ignore fields from the response of the old and new system to model expected discrepancies. This will improve the discovery of issues and will fasten the matching of responses between the old and new system during the shadowing phase.

A third one is to build a migration dashboard that shows the rollout between the new and old system, error/success ratio and percentage of mismatch.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Today I want to tackle: Errors should never pass silently.

Let’s say you are coding a logic that expects one element in an array. Now, let’s say you found an empty array. What do you do? Do you throw an exception or do you return null since there is nothing in the array?

Similarly, let’s say you are reading from a file and you got an error while accessing the file. What do you do? Do you fail at this time or do you return an empty String instead?

If you decided not to throw an exception, you most probably did it to avoid an outage or something around this line. The thing though is that the outage has already happened. You just passed the buck to someone/somewhere else.

If you had thrown, what would have been a short time to find the root cause in an outage becomes multiple hours as the exception is masqueraded now into null exception, weirdness in the system, etc…

It is always better to throw/fail or try to recover rather than fail silently.

I think this is so important that Golang decided to get rid of exceptions and have the following pattern.

resp, err := methodThatThrow()

The reason this is great is that it pushes you right now and there to decide on what to do with the err.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

I wanted to share a podcast I did with Bassem at Glich. Bassem does a great job at keeping the discussion technical and fun at the same time. I really enjoyed doing this interview and we covered a lot of topics. You can watch it on the links below.

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course

Below are things that are good to do while on-boarding.

Meet everyone you can. This includes team members and managers. Ask them what they do, the issues they are seeing and what they plan to work on next.

This will give you an understanding of the business, the current issues and the product roadmap.

Understand your company org chart. This will let you understand how the company is structured and how it operates. Try to figure out the charter and responsibility of every team

Understand the architecture and the different services. Draw an end to end flow of your product and the different areas it touches.

Enlist in the repo. Get your IDE setup. Get couple of bugs fixed ideally across the stack. Do not skip this step. It is hard to get a feel and true understanding without getting close to the code.

Understand the monitoring dashboards. These usually end up capturing the heartbeat and priorities of the business.

Finally, ask a lot of questions. Be curious. Be humble. Show professionalism. Be on time. Add any missing info to the onboarding doc. Show and share progress of your onboarding. Enjoy and make new friends.

Good luck!

Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course