Japanese university loses 77TB of research data following buggy software update

In letter : Losing data due to a backup error can mean worrying about losing years of personal photos or, as in the case of Kyoto University of Japan, losing 77TB of critical research data. The incident happened with the university’s supercomputer which received a faulty software update for its backup system, accidentally erasing 34 million files over a two-day period.

The culprit for this huge data loss was a faulty script originally intended to remove old, unnecessary log files from the Cray / HPE supercomputer at Kyoto University as part of a software update. However, it ended up deleting 77TB of search data between December 14-16, 2021 from the computer’s high capacity backup disk / LARGE0.

The university initially estimated it was losing up to 100TB of data after the buggy update erased almost all files older than 10 days. The 77 TB of search data that was deleted contained 34 million files that affected 14 hunt groups. Although Kyoto University has not disclosed the nature or details of the erased research data, it Noted (Japanese) that files belonging to 4 groups were unrecoverable.

The university’s supercomputer supplier Hewlett Packard Japan (HPE) admitted 100% responsibility for the incident and issued a letter of apology later released by the university. HPE said a modified script was released in its update to “improve visibility and readability”, such as The Stack reports.

However, HPE said it was not aware of the side effects of this behavior, which caused the modified shell script to reload in mid-run, resulting in “undefined variables” and deletion of files on. the supercomputer backup disk / LARGE0.

Kyoto University has since suspended the safeguard process, as it seeks to make improvements and add preventative measures to deal with such incidents in the future. In addition to mirroring backups, the university also plans to maintain incremental backups once it resumes the backup schedule later this month.


Please enter your comment!
Please enter your name here

Trending this Week