by Our server suffered a total SSD meltdown about 9:30am WST on 05-28-24. Unfortunately it completely made the system unbootable. We worked with our remote tech to verify that drive was uncoverable--unfortunately we can't even look at the drive. RIP
At this point we had to restore an OS image to a new drive just to get back being able to look at what happened (it took 10 hours). This set us (Windlyn) back to that image date 01-11-24 (the "time travel" issue some posted about--boy it was only like that for a few hours! Don't you guys ever sleep?!
).The next step was to restore database files. We had a nightly backup and have been restoring each database one by one and checking to make sure everything is OK. So we're back, but there are still some issues (mostly missing assets (items, emojis, etc.), and any posts made between the last database backup on the 28th and when it was restored on the 29th 4pm WST is unfortunately gone forever. It was only possible to post during the transition about 6 hours or so, so we don't believe there was a lot of posts that have been lost.
Of course we do value each and every post and it is unfortunate this was panic mode time and should have placed Windlyn in maintenance mode so no one could post during that period and avoided the issue. Live and learn!
As of this post there are still assets that need to be recovered. We're working on things, but it'll take some time. Thank you for your patience and understanding!
tldr; Server had a fatal hardware problem. Diagnosed issue, replaced hardware, working to restore backup. Going to take some time. Thanks for patience!
