No.161 / Dec 12, 2022
At approximately 3:30 p.m. on October 15, a fire broke out in the electrical room on the B3 floor of SK C&C’s Pangyo Internet Data Center (IDC), building A, which is used by Kakao, causing an interruption to Kakao’s overall service. The fire was extinguished approximately 8 hours later, but the disruption lasted for approximately 5 days.
Cause of the fire and service failure
CCTV showed a spark near the Uninterruptible Power System (UPS), a device made up of batteries that supplies electricity for a short period of time in the event of an emergency or interruption. It was suggested that the cause of the fire was due to an electrical fault with these batteries, but this is not yet clear. When using water to extinguish the fire, fire men must turn off the main power to lower the risk of electronic leakage. Therefore, power could not be supplied to the server, and the service was interrupted.
The situation at that time
Kakao’s overall service (Kakao Talk, Kakao Talk Channel (Kakao Talk Plus Friend), Kakao Pay, Kakao Gift, Kakao Login (Melon and Upbit’s Kakao account, etc.), Kakao Map, KakaoT, Kakao Mail, Tistory, etc.) was interrupted. Thus, there were disruptions to self-employed workers using Kakao Talk Channels, taxi drivers using KakaoT, and everyone using Kakao’s service. On the day of the incident, Naver Map ranked 1st, LINE ranked 2nd, and UT ranked 3rd on the Apple App Store and similarly on the Google Play Store, because Naver Map can replace Kakao Map, LINE can replace Kakao Talk, and UT can replace KakaoT. In addition, Naver posted the phrase “Global messenger LINE that does not stop when urgent contact is needed” and Toss posted the phrase, “Toss also has a chat function, contact your friend” at the top of their access screens.
Why recovery took long: insufficient server dualization
People have pointed out insufficient server dualization for the delay in recovery. At a press conference on October 19, Kakao’s co-CEOs (Nam Gung-hun and Hong Eun-taek) said “Most of the major service operation programs have been duplicated, but the work tools dealing with them have not been duplicated. Kakao had 90,000 servers distributed in 4 IDCs and SK C&C’s Pangyo IDC was main with 32,000 servers, so it caused a lot of damage.” During the recovery, a Kakao official said, “As the entire IDC is affected, traffic is so heavy that it takes time to move data to another server (a server for dualization).” Even so, it seems that there was no “perfect dualization equivalent to large companies”, that is, a Mirror Site strategy which formed a Disaster Recovery (DR) center that has duplicated servers as the main center. Also, it seems “investment in backup and distributed systems in case of failure” was not well established.
Public Apology
Nam Gung-hun, co-CEO of Kakao, resigned but he will lead an emergency committee for Kakao instead. Kakao is considering compensation for paid services and for free services. Meanwhile, the National Assembly approved the bill on IDC, which failed two years ago, on November 15. In addition to telecommunication companies, Internet companies such as Kakao and IDC operators such as SK C&C are required to build servers and power lines with various routes.
By Noh Hyun-jin, AG Reporter
noh0605@ajou.ac.kr