中文版：2023/1/21博客受攻击宕机事件分析与复盘 – Frank’s Weblog

On 1/21/2023, my blog was attacked and went down for 4 hours. This article will cover what the incident was like, the root cause analysis and improvements.

On that day, I woke up in the noon and saw the alert email from UptimeRobot. Sometimes a network or server glitch can trigger an alert as well, but it have been 2 hours since alert triggered, so apparently that’s not the case. I found I was not able to connect to the website, while sometimes I could connect but got 504.

I ssh-ed to the server and restarted all the Docker containers, but the problem persists. top showed that all the load average were 6.xx and most of the CPU usage were from php-fpm. I checked the graphs in nginx amplify and found that nginx have received large amount of requests during past few hours.

I planned to go grocery shopping for the lunar new year dinner with my girlfriend, so I didn’t want to spend too much time on this. I simply turned on the Cloudflare reverse proxy(orange cloud icon) and “Under attack” mode and left home.

After a while I received the alert clear email from UptimeRobot and website was back online.

Over last few years I’ve implemented a set of monitoring and security measures for my site and automated scripts to mitigate common issues.

1. UptimeRobot for monitoring downtime. I’ll receive alerts if the website cannot be reached or returned HTTP status that indicates a malfunctioning(5xx).
2. nginx amplify for monitoring nginx and OS metrics. I’ll receive alerts if some metrics(eg. disk usage, requests per second) goes over the threshold.
3. If requests per second goes over the threshold, it will automatically turn on Cloudflare proxy and increase security level.
4. WordPress security plugin automatically blocks malicious requests.

Benefit from these measures, my site have maintained a uptime of nearly 100%. Being a blog that only have 2 digits of visitors everyday, 4 hour downtime is nothing to worry about. But my professtional habit have been wondering what happened behind the incident, especially why these measures failed to prevent the incident from happening.

(more…)

English version: 2023/1/21 Blog Incident Postmortem – Frank’s Weblog

2023年1月21日，我的博客受到攻击宕机了4个小时左右。本文将介绍事件的经过，对根本原因的分析，及改进方案。

当天中午，我起床之后看到了UptimeRobot的报警邮件。有时一些网络或者服务器的短暂故障也会触发报警，但是当时距离收到报警邮件已经过去了近两个小时，所以事情显然没有这么简单。我简单检查后发现访问博客时有时完全无法连接，有时会返回504。

我ssh上去之后重启了一下所有Docker容器，但是故障依旧。top显示全部load average高达6.xx并且大部分的CPU使用来自php-fpm。检查nginx amplify图表之后发现过去几小时内nginx收到了大量的请求。

因为当天下午我计划和女朋友去采购年夜饭的食材，不想在这上面花费太多时间，于是我打开了Cloudflare的反向代理（橙色云图标）和Under attack模式，然后就出门了。

过了一段时间后就收到UptimeRobot的报警解除邮件，访问恢复。

我的博客上有一套监控和安全措施，以及一些自动化脚本来mitigate一些简单问题：

1. UptimeRobot用于监控可访问性，如果出现无法连接或异常的HTTP状态（5xx）则会发邮件报警
2. nginx amplify用于监控nginx和操作系统的指标，其中一些指标（例如磁盘使用，每秒请求数量）超过阈值之后会发邮件报警。
3. 如果每秒请求量超过阈值则会自动开启Cloudflare反向代理并升高安全等级。
4. WordPress的安全插件会自动block恶意请求。

受益于这些措施，博客在过去几年一直保持着近乎100%的uptime。作为一个每日访问量两位数的博客，4个小时的downtime并不是一个需要担心的问题，但是职业习惯还是让我想知道背后到底发生了什么，尤其是为什么这一系列措施都未能阻止宕机的发生。

(more…)