Arcade news, bug fixes, and general site updates.

Hardware failure! 05-28-24

Moderator: Systems Admin

Hardware failure! 05-28-24

Postby galled » Thu May 30, 2024 12:14 pm

Hello!

Our server suffered a total SSD meltdown about 9:30am WST on 05-28-24. Unfortunately it completely made the system unbootable. We worked with our remote tech to verify that drive was uncoverable--unfortunately we can't even look at the drive. RIP

At this point we had to restore an OS image to a new drive just to get back being able to look at what happened (it took 10 hours). This set us (Windlyn) back to that image date 01-11-24 (the "time travel" issue some posted about--boy it was only like that for a few hours! Don't you guys ever sleep?! :cheer: ).

The next step was to restore database files. We had a nightly backup and have been restoring each database one by one and checking to make sure everything is OK. So we're back, but there are still some issues (mostly missing assets (items, emojis, etc.), and any posts made between the last database backup on the 28th and when it was restored on the 29th 4pm WST is unfortunately gone forever. It was only possible to post during the transition about 6 hours or so, so we don't believe there was a lot of posts that have been lost.

Of course we do value each and every post and it is unfortunate this was panic mode time and should have placed Windlyn in maintenance mode so no one could post during that period and avoided the issue. Live and learn!

As of this post there are still assets that need to be recovered. We're working on things, but it'll take some time. Thank you for your patience and understanding!

tldr; Server had a fatal hardware problem. Diagnosed issue, replaced hardware, working to restore backup. Going to take some time. Thanks for patience!
User avatar

Server Administrator

Local time: Mon Nov 11, 2024 8:01 pm


Image
~Windlyn Daydream by Mouse

Windlyn is so fun,
Windlyn is so grand,
So come on everyone,
All across the land,
Come and join us now,
In this awesome site,
We will show you how,
Talking through the night.
~npixelz


Hardware failure Update 06-04-24

Postby galled » Tue Jun 04, 2024 9:48 am

Hello!

We restored all missing assets a couple of days ago and have been checking and testing for problems. The only one that we can't do anything about are some Winners have noticed that their avi has changed. This is a remnant of the time machine issue--restoring a system drive image from 01/11/24--your avi cache is showing what you were wearing on that date. The solution it so save/wear whatever you have it'll bring you "back to the future" so to speak!

Thank you for your patience, understanding, and continued support!

tldr; Everything replaced, repaired, and restored. Re-save (wear button) your avi. Many thanks!
User avatar

Server Administrator

Local time: Mon Nov 11, 2024 8:01 pm




Return to Site Updates

Who is online

Users browsing this forum: CCBot [Bot] and 1 guest