小事记：Hypixel紧急维护

https://hypixel.net/

Update #3 (June 20th – 12:45PM ET): All of the new hardware has been configured to the new network and all the core servers and minigame servers are working great. We are now entering the final steps of setting up and configuring the last bits of the network and DDoS protection, this is the more tricky part because we heavily rely on external partners to do their parts so we have no clear vision on progress at this time.

We are still seeing continuous large-scale DDoS attacks across the board in the Minecraft server scene, which means our decision to make this major change was the right path forward. Read more


Tonight, if all goes well of course, we will be doing tests assuming our partners meet their deadlines. Please note that these tests will require reboots and downtime – the player count will be severely limited. As soon as the situation is fully back in our hands, you will see more frequent updates from us and then slowly increase the player count to allow all of you in.

Our team has done everything humanly possible, spared no expense and worked with incredible partners to make this happen as quickly as possible for our amazing community that we are grateful for.



Update #2 (June 19th – 11:15AM ET): We’ve added a progress bar to give you a better understanding of where we are in regards to bringing the Hypixel Server back to all players. Please note that this is not representative of time, and will be manually updated, so don’t completely rely on it. This will, however, give you all at least some transparency on where we are with the work. It’s currently too difficult to give an exact ETA as several external teams are involved. As we are passed updates, we will quantify the % and add it to the progress bar. Read more
This morning, the on-site team moved most of the core hardware to be on the Cloudflare network, and our servers will no longer be on any network of our host. We will be effectively 100% Cloudflare, with full DDoS protection from them. Unfortunately, in order to do this, we have had to move (and still are) quite a lot of hardware from the current host’s network; this has taken a fair bit of time due to the large number of servers and racks we have. However, all critical machines required to begin initial setup and hold a decent player count have been moved. Today, external network teams are being dispatched to assemble and configure the new network. This new setup will be on Cloudflare, and your ping should technically be somewhat the same. However, because this is the first time we will be using full Cloudflare, there’s a possibility that we may have some issues at first. Despite that, their team is assisting us and we have worked with them for years with great success. Note that the forums, API and some other things might be broken at first, but will be sorted later on. We will try to give another update as soon as possible. Keep an eye on the progress bar – we will do our best to update it whenever possible! Thank you for your support!


Update #1 (June 18th – 3:35PM ET): There are still several ongoing large-scale DDoS attacks going on in the Minecraft server scene at this time and we are not quite done with our changes before we can come back online safely. We do not have a precise ETA at this moment, as we are relying on several external partners to help us finalize the DDoS protection and major changes to our network infrastructure. We will do our best to give an ETA by tomorrow morning, but overall we aim to get back online at some time this weekend if all goes well. Once again, we appreciate your support during these difficult times, we will keep you guys updated the best we can.

Hey guys!

We will be taking down the Hypixel Network for emergency maintenance. This maintenance begins now and will last for at least 24 hours. Should we choose to extend this downtime, we will post an update on this news post.

During this time we will be reworking the network to ensure the issues we have been having these last few days don’t happen again.

We would like to apologize for the inconvenience the last few days have caused and we hope that our community understands the difficulty of the situation.

The situation is: Our host is currently under large-scale DDoS attacks and they are actively working to resolve the situation. We are assisting them the best we can, which is why we have to put the server on maintenance during this process.

We have already done many network improvements that have lessened the impact of the attacks that are either directed at us or not, but it has proven not to be enough. Our top priority is to mitigate the impact of these attacks as quickly as possible while working on permanent solutions that will shield us better from future attacks and isolate our network.

The good news is that we have dealt with DDoS attacks for well over 8 years now and we have a great understanding of the situation. Each of these attacks is helping us better understand weak points that need to be improved from our side of things and also for our host. As much as this situation is bad for everyone at this time, it is giving us very good data to further understand and lessen the impacts of future attacks.

In reality, about a year ago, these attacks would have had no impact at all, but some recent changes at our host caused a flaw in our setup. so the host is working with us to resolve the situation. We all know what to do to resolve the situation because we have done it before, but it requires quite a bit of work to get to that place again. We’re hoping 24 hours is enough for this upgrade, but please understand it may take more than that.

To be fair, we haven’t been as alert or ready as we should have been as it’s been about 2 years since our last DDoS attacks. This is definitely a good opportunity for us to set some improvements in motion and deal with this once and for all.

We made sure any time-based services on the network are taken care of. This means pausing boosters, pausing the SkyBlock Auction House and Bazaar timers, pausing timers on MVP++ and daily rewards, etc. We’re hoping not to forget anything, but if we do, we are always open to a chat on our Help Desk.

Furthermore, when we bring the server back online, we will be turning on Triple Coins and Triple XP for at least a week.

As the situation progresses, we will be posting any updates to our status page. Once we are ready to go live again, we will make an update with more details and share as much as we possibly can with our community.




As always, we are extremely thankful for everyone’s support in these difficult moments and apologies once again for the issues. We want to reassure our community that the current situation is our top priority.

https://status.hypixel.net/incidents/tcsyp6sl632g

Update
We have the new hardware configured, but we still need more time for set up and for our external partners to do what they need to do. But you can read more about it on the website > https://hypixel.net/
Posted 17 hours ago. Jun 20, 2021 – 13:46 EDT
Update
A progress bar has been added to the website to show how much more needs to be done before we can get up and running again. Additionally, we posted an update regarding what’s going on with the maintenance, so be sure to check it out > https://hypixel.net/

If you are having trouble seeing this new update, make sure to press CTRL and F5 at the same time while on the page to refresh the cache.
Posted 2 days ago. Jun 19, 2021 – 11:44 EDT
Update
The Hypixel Network will be down for emergency maintenance starting now. We are sorry for any inconvenience and thank you for being patient with us. You can read more at https://hypixel.net
Posted 3 days ago. Jun 17, 2021 – 20:33 EDT
Update
We are continuing to work with our upstream provider to resolve this. We have resolved the issue of some players being unable to connect.

We appreciate your patience and support as we work to resolve these issues.
Posted 4 days ago. Jun 16, 2021 – 19:31 EDT
Update
We are continuing to work with our upstream provider to implement a way to fully resolve this issue. We are also aware of some players being unable to connect completely at this time.

We appreciate your patience and support as we work to resolve these issues.
Posted 4 days ago. Jun 16, 2021 – 18:38 EDT
Identified
We are aware of the issue still existing and are working on a larger solution with our provider. This may take some time and we’ll keep this incident updated.
Posted 5 days ago. Jun 16, 2021 – 07:30 EDT
Monitoring
Upstream has implemented a change and we are monitoring the results.
Posted 5 days ago. Jun 15, 2021 – 21:55 EDT
Identified
We are aware of the connection issues occurring again and are working with our upstream provider.
Posted 5 days ago. Jun 15, 2021 – 20:37 EDT
Monitoring
Upstream has implemented a change and we are monitoring the results.
Posted 5 days ago. Jun 15, 2021 – 19:38 EDT
Update
We are continuing to work with the upstream provider to resolve the issues as soon as possible
Posted 6 days ago. Jun 15, 2021 – 15:33 EDT
Identified
We have identified the issue with an upstream provider and are working with them to resolve it as soon as possible.
Posted 6 days ago. Jun 15, 2021 – 15:03 EDT
Investigating
We are currently investigating an issue with intermittent connection issues to the Minecraft Server and Forums.
Posted 6 days ago. Jun 15, 2021 – 15:00 EDT
This incident affects: Minecraft Server (SkyBlock, Hypixel SMP, Bed Wars, SkyWars, Murder Mystery, Housing, Build Battle, Arcade, Prototype Lobby, The TNT Games, The Pit, UHC Champions, Classic Lobby, Mega Walls, Blitz SG, Cops and Crims, Smash Heroes, Warlords, Duels) and Website, Public API.

然而非常巧合的是我收到了一封来自CloudFlare的邮件，长这样：

Dear Cloudflare Customer:

We are reaching out because of the recent resurgence in ransom-driven DDoS threats that are targeting organizations of all sizes.

This email is to notify you that your Cloudflare-protected Internet property is already safeguarded against these threats. Please know that if your organization is ever threatened with a ransom DDoS attack, we stand ready to help.

What is a ransom-DDoS attack? 
A ransom DDoS (RDDoS) attack is when a malicious party attempts to extort money from an individual or organization by threatening them with a distributed denial-of-service (DDoS) attack.

Most ransom DDoS attacks start with a ransom note sent to the target in which the attacker threatens the business or organization. In some cases, an attacker may carry out a small demonstration attack to illustrate their seriousness before sending a ransom note.

What to do if you receive a threat?
Do not panic and do not pay the ransom: Paying the ransom only encourages bad actors — and there’s no guarantee that they won’t attack your network now or later.
Notify local law enforcement: They will also likely request a copy of the ransom letter that you received.
 
How can you prepare now for this threat?
Enable Under Attack Mode during an active DDoS attack: All Internet properties proxied by Cloudflare are already protected against DDoS attacks of any size and kind. The Under Attack Mode performs additional security checks to help mitigate Layer 7 DDoS attacks. Validated users access your website and suspicious traffic is blocked. Learn how to enable the Under Attack Mode here.
Enable DDoS protection for network infrastructure: If your organization’s network infrastructure (Layer 3/4) is exposed to the Internet, consider adopting Cloudflare Magic Transit to extend the Cloudflare protection you get for your web assets to your entire IP infrastructure.
Enable DDoS alerts: If you are on a Cloudflare paid plan, you can be notified immediately in the case of an attack on your Cloudflare protected Internet-property. Click here to enable DDoS alerts from your dashboard.
Review our support docs: Learn best practices to secure your Cloudflare-enabled site and review how to respond to ransom notes threatening a DDoS attack here.The Cloudflare Team

更巧合的是Bedless的这个帖文：

我就不说什么了，毕竟Hypixel自己都没说受到勒索，但最近这些事情确实有点恐怖，居然有人组织去攻击Minecraft服务器，简直是奇了个大闻，如果什么时候我也能享受到这种待遇就好了（doge）

可惜了这个周末啊，明天开始就要期末考试了，这两天都要无聊死了