Software Engineering Tidbits

Software Engineering Tidbits

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
Migration
Copy link
Facebook
Email
Notes
More

Migration

Georges El Khoury's avatar
Georges El Khoury
Nov 18, 2022
1

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
Migration
Copy link
Facebook
Email
Notes
More
Share

If you work long enough as a software engineer you will end up doing a migration from one system to another.

Below is a step by step guide on how to approach it.

First step is to standardize and consolidate the API of the old system. This is an important step and critical to the success of the migration.

Second step is to implement the API of the new system.

Third step is to invoke the old system and the new system keeping the old system as the source of truth. In this step, you compare the old and new system output and fix any discrepancies between them. This is called Shadowing.

Most of them will be from bugs in the new system but do not be surprised if it is also due to existing bugs in the old system that got surfaced by the shadowing process.

The longer you shadow the more confident you can be of the new system. This comes at the cost of delaying the migration of the new system, maintaining the old system and calling both the old and the new system at the same time.

A huge problem in the shadow phase is what to do with new features? Do you setup a feature freeze on the old system? Do you tell teams to wait for the end of migration and implement in the new one? Do you implement in both?

This is why mostly shadowing needs to run for a limited period only.

The forth stage is similar to the third stage but uses the new system as the source of truth for the returned output. This is called Reverse Shadow. The benefit of this stage is to know that it is possible to fallback to the old well tested system in case of an outage.

The fifth and last stage is to just call the new system, remove the shadowing and reverse shadowing logic and deprecate the old system.

Bonus point is to have a logic to do gradual or targeted rollout from the old system to the new system in the shadow and reversal phase

Another bonus point is to have an ability to compare and ignore fields from the response of the old and new system to model expected discrepancies. This will improve the discovery of issues and will fasten the matching of responses between the old and new system during the shadowing phase.

A third one is to build a migration dashboard that shows the rollout between the new and old system, error/success ratio and percentage of mismatch.


Software Engineering from the Frontlines Course on Maven

If you liked this article, I will be teaching a “Software Engineering from the Frontlines” course on Maven where I will teach hard-learned lessons I acquired developing large-scale products at companies such as Uber, Airbnb, and Microsoft.

View Course


Thanks for reading Software Engineering Tidbits! Subscribe for free to receive new posts and support my work.

Daniel Jarzębowski's avatar
1 Like
1

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
Migration
Copy link
Facebook
Email
Notes
More
Share

Discussion about this post

User's avatar
A good unit test
A good unit test should be:
Feb 20, 2023 • 
Georges El Khoury
18

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
A good unit test
Copy link
Facebook
Email
Notes
More
5
A good way to debug
One of the best software engineering tip I received is a good way to debug.
Apr 28, 2022 • 
Georges El Khoury
16

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
A good way to debug
Copy link
Facebook
Email
Notes
More
2
Outage Management
Outage management is a core skill for a software engineer to acquire and is critical to achieve high availability of an online service.
Feb 13, 2023 • 
Georges El Khoury
13

Share this post

Software Engineering Tidbits
Software Engineering Tidbits
Outage Management
Copy link
Facebook
Email
Notes
More
3

Ready for more?

© 2025 Georges El Khoury
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More

Create your profile

User's avatar

Only paid subscribers can comment on this post

Already a paid subscriber? Sign in

Check your email

For your security, we need to re-authenticate you.

Click the link we sent to , or click here to sign in.