Microsoft Fixes Insane Exchange Y2K22 Bug That Disrupted Global Messaging

Microsoft Fixes Insane Exchange Y2K22 Bug That Disrupted Global Messaging

Getty Images

Microsoft released a fix for an insane Exchange Server bug that shut down on-premises mail delivery around the world as the clocks struck at the start of the New Year.

The massive disruption was the result of a date verification failure in Exchange Server 2016 and 2019 that prevented servers from adjusting to the year 2022, prompting some to call it bug Y2K22. Mail programs stored dates and times under signed integers, whose maximum is 2147483647, or 231 – 1. Microsoft uses the first two digits of an update version to indicate the year of its release. As long as the year was 2021 or earlier, everything was working fine.

“What is Microsoft? “

However, when Microsoft released version 220100001 on New Years Eve, the on-premises servers crashed because they were unable to interpret the date. As a result, messages got stuck in transport queues. Admins around the world have found themselves frantically trying to solve problems instead of ringing in the New Year with their friends and family. All they had to do was two cryptic log messages that looked like this:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

“What is Microsoft!” »An administrator wrote this Reddit thread, which was one of the first forums to report mass failure. “New Years Eve !? The first place I check is Reddit and you save my life before we even have an engineer on the phone.

The next day, Microsoft posted a fix. It comes in two forms: a PowerShell script, or a manual fix in case the script doesn’t work properly, as some admins have reported. In either case, the fixes should be performed on each on-premises Exchange 2016 and Exchange 2019 server within an affected organization. The automated script can run on multiple servers in parallel. The software maker said the automated script “may take a long time to run” and urged administrators to be patient.

The date and time check was performed when Exchange checked the version of FIP-FS, a scanning engine that is part of Exchange malware protections. Once the FIP-FS versions started with the numbers 22, the check could not be completed and the mail delivery was abruptly interrupted. The hotfix stops the Microsoft Filtering Management and Microsoft Exchange Transport services, removes the current anti-virus engine files, and installs and starts a corrected anti-virus engine.

On Monday, things were back to normal for many affected organizations. It’s unclear how long the buggy date store was in place, but judging from the two affected versions, it may have been introduced during the development of Exchange Server 2016.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Trending this Week