• Use Cloudflare Load Balancer with Cloudflare Tunnel

    中文版:配合Cloudflare Tunnel使用Cloudflare Load Balancer – Frank’s Weblog

    Cloudflare Load Balancer is a global load balancing product provided by Cloudflare. It can connect to origin servers in traditional ways by DNS name or IP addresses, it also can be integrated with Cloudflare Tunnel to create a seamless and secure network infrastructure.

    Using Cloudflare Tunnel with Cloudflare Load Balancer is more complicated as we need to configure the DNS name and host header to make sure the routing and monitoring work correctly.

    In this post, we will use an example to demonstrate how to use Cloudflare Load Balancer with Cloudflare Tunnel.

    (more…)
    Read more: Use Cloudflare Load Balancer with Cloudflare Tunnel
  • 配合Cloudflare Tunnel使用Cloudflare Load Balancer

    English version: Use Cloudflare Load Balancer with Cloudflare Tunnel – Frank’s Weblog

    Cloudflare Load Balancer是Cloudflare提供的一个全球负载均衡产品。它可以以传统方式(域名或IP地址)连接源服务器,还可以与Cloudflare Tunnel集成,以创建一个无缝和安全的网络基础设施。

    将Cloudflare Tunnel与Cloudflare Load Balancer一起使用的配置与传统方式相比略微复杂,我们需要正确地配置域名和Host头,以确保路由和监控的正常工作。

    在这篇文章中,我们将用一个例子来演示如何配置将Cloudflare Load Balancer与Cloudflare Tunnel一起使用。

    (more…)
    Read more: 配合Cloudflare Tunnel使用Cloudflare Load Balancer
  • 2023/1/21 Blog Incident Postmortem

    中文版:2023/1/21博客受攻击宕机事件分析与复盘 – Frank’s Weblog

    On 1/21/2023, my blog was attacked and went down for 4 hours. This article will cover what the incident was like, the root cause analysis and improvements.

    On that day, I woke up in the noon and saw the alert email from UptimeRobot. Sometimes a network or server glitch can trigger an alert as well, but it have been 2 hours since alert triggered, so apparently that’s not the case. I found I was not able to connect to the website, while sometimes I could connect but got 504.

    I ssh-ed to the server and restarted all the Docker containers, but the problem persists. top showed that all the load average were 6.xx and most of the CPU usage were from php-fpm. I checked the graphs in nginx amplify and found that nginx have received large amount of requests during past few hours.

    I planned to go grocery shopping for the lunar new year dinner with my girlfriend, so I didn’t want to spend too much time on this. I simply turned on the Cloudflare reverse proxy(orange cloud icon) and “Under attack” mode and left home.

    After a while I received the alert clear email from UptimeRobot and website was back online.

    Over last few years I’ve implemented a set of monitoring and security measures for my site and automated scripts to mitigate common issues.

    1. UptimeRobot for monitoring downtime. I’ll receive alerts if the website cannot be reached or returned HTTP status that indicates a malfunctioning(5xx).
    2. nginx amplify for monitoring nginx and OS metrics. I’ll receive alerts if some metrics(eg. disk usage, requests per second) goes over the threshold.
    3. If requests per second goes over the threshold, it will automatically turn on Cloudflare proxy and increase security level.
    4. WordPress security plugin automatically blocks malicious requests.

    Benefit from these measures, my site have maintained a uptime of nearly 100%. Being a blog that only have 2 digits of visitors everyday, 4 hour downtime is nothing to worry about. But my professtional habit have been wondering what happened behind the incident, especially why these measures failed to prevent the incident from happening.

    (more…)
    Read more: 2023/1/21 Blog Incident Postmortem
  • 2023/1/21博客受攻击宕机事件分析与复盘

    English version: 2023/1/21 Blog Incident Postmortem – Frank’s Weblog

    2023年1月21日,我的博客受到攻击宕机了4个小时左右。本文将介绍事件的经过,对根本原因的分析,及改进方案。

    当天中午,我起床之后看到了UptimeRobot的报警邮件。有时一些网络或者服务器的短暂故障也会触发报警,但是当时距离收到报警邮件已经过去了近两个小时,所以事情显然没有这么简单。我简单检查后发现访问博客时有时完全无法连接,有时会返回504。

    我ssh上去之后重启了一下所有Docker容器,但是故障依旧。top显示全部load average高达6.xx并且大部分的CPU使用来自php-fpm。检查nginx amplify图表之后发现过去几小时内nginx收到了大量的请求。

    因为当天下午我计划和女朋友去采购年夜饭的食材,不想在这上面花费太多时间,于是我打开了Cloudflare的反向代理(橙色云图标)和Under attack模式,然后就出门了。

    过了一段时间后就收到UptimeRobot的报警解除邮件,访问恢复。

    我的博客上有一套监控和安全措施,以及一些自动化脚本来mitigate一些简单问题:

    1. UptimeRobot用于监控可访问性,如果出现无法连接或异常的HTTP状态(5xx)则会发邮件报警
    2. nginx amplify用于监控nginx和操作系统的指标,其中一些指标(例如磁盘使用,每秒请求数量)超过阈值之后会发邮件报警。
    3. 如果每秒请求量超过阈值则会自动开启Cloudflare反向代理并升高安全等级。
    4. WordPress的安全插件会自动block恶意请求。

    受益于这些措施,博客在过去几年一直保持着近乎100%的uptime。作为一个每日访问量两位数的博客,4个小时的downtime并不是一个需要担心的问题,但是职业习惯还是让我想知道背后到底发生了什么,尤其是为什么这一系列措施都未能阻止宕机的发生。

    (more…)
    Read more: 2023/1/21博客受攻击宕机事件分析与复盘
  • Deploying Budget Friendly ARM-X86 Hybrid Kubernetes Cluster with K3s

    中文版:使用K3s部署预算友好的ARM-X86混合Kubernetes集群 – Frank’s Weblog

    Kubernetes were used for enterprise level services, which is heavy and expensive. Even for the least-expensive Digital Ocean, its managed Kubernetes starts with $12/month per node. Then I learned about K3s, a lightweight Kubernetes distribution that removes or lightens many of the components in Kubernetes, allowing K3s to run on smaller VMs and even Raspberry Pi while still having the Kubernetes’ features and scalability.

    This article will demonstrate at how to build a budget friendly Kubernetes cluster using K3s on Oracle Cloud Free Tier and find a balance between fancy technology and cost. The goal is to migrate some of my personal services that I previously ran using Docker, including Matomo, MariaDB, Mastodon, and several personal services to Kubernetes. This article will focus on the high-level design and concepts, and will not cover specific implementation details.

    (more…)
    Read more: Deploying Budget Friendly ARM-X86 Hybrid Kubernetes Cluster with K3s
  • Create Personal Financial Sankey Diagram from Intuit Mint Data

    Throughout 2022, the financial statements from tech companies were extensively discussed, and you probably have seen diagrams like this:

    I later discovered that this diagram is known as Sankey Diagram, which is a type of flow diagram in which the width of the arrows is proportional to the flow rate of the depicted extensive property. The Sankey diagrams are often used in the visualization of material flow analysis. It can also visualize cost breakdowns.[1]

    I’ve been using Intuit Mint to manage my finance, however I feel that the budget and trends feature is not good enough: The pie chart and histogram that Mint provides doesn’t provide enough insight, and its budget feature is limited, for example, it’s not good at managing expenses that happens once but covers an extensive period of time.

    (more…)
    Read more: Create Personal Financial Sankey Diagram from Intuit Mint Data
  • 2022 Year in Review

    ,

    中文版:2022年终总结 – Frank’s Weblog

    2022 was a somehow a bumpy ride. Although many goals were not achieved, it was fortunate that I did not encounter any major difficulties in such a tough environment.

    Career

    In February, my company hosted the Sales/Engineering Kick Off for the new fiscal year and invited employees all over the world. After one year’s working from home I finally met some of my teammates, but lots of colleagues didn’t come because of another wave of COVID.

    Because of the success of the previous project, I got my first promotion in my career. Along with the promotion comes larger scope and higher expectation for ownership, as well as taking on more team service including supporting customers and mentoring new hires.

    In the mid of the year I successfully delivered a large project, This project revamped the network architecture of our cloud product and added the capability to serve enterprise customers from network security perspective, unblocked a few customers. Being the project owner, I’ve learned a lot in both technical and management perspective.

    (more…)
    Read more: 2022 Year in Review
  • 2022年终总结

    ,

    English version: 2022 Year in Review – Frank’s Weblog

    2022年从某种程度上是乏善可陈的一年。虽然很多目标并没有达成,但在糟糕的大环境下也万幸没有遇到太大的困难。

    工作

    年初我司在Las Vegas举办了新财年的Sales/Engineering Kick Off,并邀请了全球的员工前来参加。WFH一年来终于第一次见到了部分同事,遗憾的是因为年初Omicron变种的造成的大流行,很多同事并没有前来。

    由于去年年底和年初项目的成功,我如愿以偿地获得了第一次升职。随之而来的是更大的scope以及对于ownership的要求,以及承担更多的组内工作,如客户支持和mentor新人。

    今年年中成功交付了一个大型项目,这个项目改进了我们的Cloud产品的网络架构,从网络安全的角度上使我们的产品具备了服务企业用户的能力,unblock了一些客户。作为项目的owner,从整个过程中学到了很多东西,无论是技术上的还是管理上的。

    (more…)
    Read more: 2022年终总结
  • 使用K3s部署预算友好的ARM-X86混合Kubernetes集群

    English version: Deploying Budget Friendly ARM-X86 Hybrid Kubernetes Cluster with K3s – Frank’s Weblog

    Kubernetes大多被用于企业级的服务,十分沉重且昂贵。即使是最便宜的DigitalOcean,其Managed Kubernetes最低也需要$12/月每节点。后来我了解到了K3s,一个轻量级的Kubernetes发行版。K3s移除或轻量化了Kubernetes中的很多组件,使得K3s可以运行在较小的VM甚至Raspberry Pi上,同时仍然拥有Kubernetes的可扩展性。

    本文将研究如何使用K3s在免费的Oracle Cloud Free Tier上搭建一个预算友好的Kubernetes集群,并在炫酷技术和成本之间找到一个平衡,目标是将我之前使用Docker运行的一些个人服务,包括Matomo,Maraidb,Mastodon以及若干小工具迁移到Kubernetes上来。本文将侧重选型及概念,不会涉及具体实现细节。

    (more…)
    Read more: 使用K3s部署预算友好的ARM-X86混合Kubernetes集群
  • 使用Velero Restic集成备份及恢复Kubernetes数据卷

    English version: Backup and Restore Kubernetes Volumes with Velero Restic Integration – Frank’s Weblog

    我搭建了一个Kubernetes集群,使用OpenEBS作为存储后端。我选择了Jiva作为存储引擎,Jiva是一个高可用的存储控制器,数据被复制到所有节点。为了确保数据的安全,我使用Velero及其Restic集成将卷备份到AWS S3。

    安装

    首先需要在本地电脑上安装 Velero CLI以控制Kubernetes集群上的Velero控制器,请参阅Velero Docs – Basic Install了解安装说明。

    准备Kubernetes配置文件,其应位于.kube/config 并确保 kubectl get pod 返回正确的结果。

    使用如下的格式创建一个AWS密钥文件,该密钥应具有访问在下一步骤中提供的S3存储桶的权限,记下该文件的路径。

    (more…)
    Read more: 使用Velero Restic集成备份及恢复Kubernetes数据卷