HaCkeD by MuhmadEmad
KurDish HaCk3rS WaS Here
FUCK ISIS !
I have officially downgraded the cloud cluster. It’s now longer using PHP7 or HHVM since neither was providing a stable environment for which we could properly operate Bigcloset/Topshelf.
Belle (front end server) is now operationally running PHP5.6
Belle, can be switched over to HHVM for limited testing as needed as we try to track down HHVM issues and fix them in code, but right now, the full/operational stack is as follows.
Belle (Front End)
- Apache 2.2
- PHP5.6 (remi stable release)
- Percona 5.6 (release 76.1 Revision 5759e76)
We tracked down what seems to be crashing HHVM and are researching possible fixes.
We are currently up and running using HHVM via Cloud in a “cobbled together” way. It is a less than ideal setup so I still need the VM re-imaged.
The method we have cobbled together right now seems to “fail” for HHVM every 8 hours or so in our initial tests, so I’ve written my own set of scripts to monitor the site and take automated action based on results. It’s kinda primitive but works 🙂
Wed Feb 24 15:00:01 CST 2016 HTTP Status when killed was: 503 Wed Feb 24 15:37:08 CST 2016 HTTP Status when killed was: 503 Wed Feb 24 23:43:01 CST 2016 HTTP Status when killed was: 503 Wed Feb 24 23:45:01 CST 2016 HTTP Status when killed was: 503
One of the longest steps in this whole process is one known as “Rebuild Permissions”. The BIGGEST issue with it was it seemed DESTINED to fail while the “WALL” was up, so to do it properly, we felt the need to take-down the wall and start letting people in.
Untill the permissions database is fully built, you will notice some stuff missing. Some OLDER stories will show up on the front page with new dates. Same with comments and such. But it IS being worked on.
The three of us pulled a Marathon 2/3 days getting this done, bulldozing every roadblock we ran into at full speed, and then rebuilding using the pieces left behind at each step.
I will post again here, and on BigCloset itself when it seems to be finished with it’s permission rebuild. and hopefully we can finish the lengthy process of fixing any lingering errors.
-Piper, Erin & Cat.
We are on the down-hill run at this point. I’m actually hoping to get some sleep tonite! 🙂
Not all the new features will be available at the launch, but the site should be back shortly 🙂
We’re still hard at work behind the scenes. I managed to grab about 3 hours sleep (not three hours straight through, but basically 3 hours where I blacked out enough that I’m going to call it sleep).
Things are progressing on the database and we are working on other issues as well right now simultaneously.
We are aiming for mid-day west-coast (USA) time for the site to come back online, and while it could be sooner, I want to warn everyone that it very easily could take longer depending on how everything goes.
Right now, we aren’t anticipating any issues that will cause major delays.
-Piper, Cat, Erin and the BigCloset Band
Hi Everyone! Quick Note!
I’m on my way up to the datacenter, to install some new equipment.
Part of the Equipment Package I’m delivering includes Infrastructure improvements, namely new network cabling.
While the downtime should not be noticeable, less than 30 secs at a time, you may have periods where The site is shown as offline, or the database offline, or everything may LOOK fine, but because the slave database server is offline, you get a 404 error when you click on a story.
The outages should be less than 30 secs at a time, and the whole process shouldn’t take me very long at all, so please bear with us!
-Piper, Erin, Cat and the BigCloset Band
We are currently experiencing issues with the server again. We are steadfastly working to correct the situation. We will try and keep you updated as things progress.
-Piper, Joyce, Cat and Peeps.
UPDATE: It is hardware failure having corrupted the database, we’re trying to bring things back up without losing anything. Gifts to the Hatbox or the BC Hardware fund to help us out can be sent via Paypal to Joyce at QnEZ.com. Thanks.
SHORT UPDATE: BC has a gigantic database, we’ve been able to read the failed drive but the info is not in the form of a database anymore. It got scrambled, to put it in simple terms. What we’ve been doing for the last many hours is reassembling the database. We had hoped to be done by this evening but it may take a bit longer.
MORNING UPDATE: Just so everyone knows, we are making progress. 🙂
MONDAY UPDATE: We’ve made considerable progress in restoring the story database and now need to get the other parts of the data in order. Then final testings and probably some adjustments. No timeline at the moment, but soon is something that can be hoped for and looked forward to. Thanks to everyone who has contributed, that will help enormously in making plans to see this sort of thing does not happen again.
TUESDAY UPDATE: We’ve got the site up and running with most stories recovered (not publicly available, sorry), but lots of stuff still needs to be done. The internal links, kudos, read counts, private messages and a number of other things still need to be recovered and we seem to be missing a few stories so we are still working on this. Hope to have another update later today. — Joyce
TUESDAY EVENING: We’ve got the site 98% recovered, we just have to get some details done so that all the stories are accessible. Tomorrow sometime is beginning to look more than just possible.
WEDNESDAY EVENING: More progress. Fixing the little niggling problems. We do hope to have the site up tomorrow. 🙂
THURSDAY MORNING: Here is a tease to keep you going.
Here’s an Amazon Shopping Link for those that asked for that: TG Fiction at Amazon US
But remember, we won’t get the commission on shopping until April. I wish I could put up the Hatbox Amazon buttons here but they won’t work without the BigCloset site to return to, it would end up flagging the account.
So the best way to donate to help us out is to use PayPal at the moment. My account there is Joyce @ Qnez.com.
Thanks and Hugs, Joyce