When it rains, it pours. Another network file server (dribbler) that serves the janky cluster was running into issues that didn’t require any upgrades (unlike the earlier issue with swiffer). As it turns out, Murphy and his blasted law apparently wanted to kick us while we were down. The servers that were seeing problems because of this were:

abraxor adbo adonis agamemnon ajax alboz ampedsports aphrodite apollo arda aries atlas atra avenus avi2l azcuehnos baachus backtrack3 barefoottours bashtage batserver battle bbd ben berniemedia bettyboop bgportal bimbodulcop blackfalcon-web blackfday blackhole blackman blizzard blogshare bluebox bogus borealis breckgear buychina caiman calypso caparazon casaclubtv casper celeb centaurus cfndc charon charybdis chens cheritek chipsiweb chronos chubu ciclo22 concorde coolblue daedalus dalliance danyun designerbrickmarble dexau dh01 dione dionysus diversity djm djpreet dusqi dusty ecards eduagora electra elvis emanuel entropia erebus erida eros evildead ewok farnsworth fastserver fiercee fifachile flamjam flowstudios forkpile fresh fuzzacademy ganesha gazihost gencer gl gladys gnot gorgon grasshopper grimbald gumoz habbid hades hb02 hdotnet hecate hector helios hephaestus hera heracles hermes hestia hypatia hypnos i2 icarus ichigo iinst1 instantcenter iota iotatec iris itauna jarcas jbutton jelly jlaurinphotography jobbr jove kaysnature kermit kesido keyiflimutfak keysersoze kiley kitwe kopi kratos kronos kztixo lagrange leichtman leto linkmoney linneo lolicg lucent lucy madotter managedq marcdorcel marhgil maserver mediadirect medusa mf1 mf2 mho midas minerva mll mooshi morpheus muchmedia mwl nandobhz nazgul nemesis netseyredin noodledee numbers oceanone odysseus odysseus1 odyssey oedipus optimusprime oriza osiris pan pandora peaceful pegasus persephone phi pichoster pimp plubble pmbd poseidon privateserver prospero proteus proxybilisim ps3854 ps3865 ps3868 ps3908 ps3930 ps3944 ps3953 ps3957 ps3982 ps3994 ps3997 ps4053 ps4072 ps4098 ps4121 ps4131 ps4168 ps4206 ps4337 ps4555 ps4571 ps4582 ps4583 ps4584 ps4585 ps4693 ps4721 ps4729 ps4730 ps4731 ps4978 quifflet quiknet reactor reuben rhea rhinoco rvooz serieonline shyfry simflight sisyphus sitesablaze sixtytwo smutbox socket somaticvision sturman sturmgroup stusserver sunnywood svetluska synergia tailfin tamiyaclubmovies tartarus tcb tethys thanatos thetis tic tonni triage triton twunbbs tyche universoprivate viigmt vivenet vrt7 wally warppipevps webben webhostservice weblogstematicos webserver webxl wolkanca xiondesigns yestoronto yuxiong zdki zentastic zeus

To keep this from happening again, some offloads will be taking place. Those should be pretty low key tho. Most servers should be back and operating correctly now — but a few are being pulled up by the scruff of their neck. If you are on one of these machines and happen to still see issues, please contact support and we’ll gladly help you get your sites back up as quickly as possible.

We are currently experiencing some problems with the file server Swiffer. The machine is having problems with some ram that needs to be replaced. We are on the way to replace it right now. The affected machines are:

abrahydroplanes aims arizona artofmanliness asle bambi boba brisk cactuscooler calpico caprisun cedar chai clamato codered dasani dietrite downside eggnog evian eyedock fabsilv fiji flipside frances francis-piragua frappe freewebsites fresca gannon geyser gordini grapico grassroots gravatt horchata inko intenselighting mastequila mccreery minutemaid mrpibb mug musclemilk nehi nesbitt netsolution niagara odr oj orangebang ovaltine polnetwork pom powerade ppps ps1 ps3673 ps4316 refresco right seltzer shasta silk sparkletts stacyfayehan sunnyd tampico tang tea tizer tropicana uncletapa welchs xecova zakidesign zoot

We will post as soon as this issue is resolved.

Update 13:26 PST: The ram has been replaced and the file server came back online without a hitch. We are cleaning up the webservers that tanked due to I/O backup, if your sites are not back now they should be shortly.

The server, capone, is currently having some hardware problems. One of our admins suspects that the Nic card went out and is currently on his way to the datacenter to swap the hardware out. We should have this back up shortly and we’ll update this post once it’s resolved. We’re sorry for the inconvenience this causes you.

Update 12:15pm PST: This server is now back up and running as of about 10 minutes ago. We’ve tested several sites hosted on the server and they’re back up and running, so things should be back to normal now. Of course, if you still have problems please contact support and we’ll be happy to look into it further.

The following MySQL servers will be going down shortly for some brief maintenance. Estimated downtime is between 15-20 minutes:

dewey, huey, janky-vsql01

If you have a PS named any of the following you are on janky-vsql01 and this will affect you:

psmysql1829, psmysql1861, psmysql1863, psmysql3011, psmysql3016, psmysql3029, psmysql3166, psmysql3171, psmysql3228, psmysql4330, psmysql4588, psmysql4975, psmysql744, psmysql745, psmysql797, psmysql799, psmysql802, psmysql961, psmysql962, psmysql963, psmysql964, psmysql965, psmysql966, psmysql967, psmysql968, psmysql969, psmysql970, psmysql972, psmysql975, psmysql978, psmysql979, psmysql986, psmysql991

I apologize for the short notice of downtime.

Update 3:18 AM PST: Ok, dewey and janky-vsql01 are back up so all services should be restored on those. Huey is doing an fsck due to the fact it has been a considerable amount of time since the last one was ran and a check was forced. MySQL machines are fast so I would estimate it will be back up within 5 minutes.

Update 3:24 AM PST: Ok. huey is back up and happy again. All service has been restored on all machines. Please contact support if you are on any of the above machines and still having any issues.

This server will be going down in the next 30 minutes or so for some short maitenance. This will effect you if your PS is named any of the following:

psmysql1014, psmysql1475, psmysql1476, psmysql1479, psmysql1686, psmysql1801, psmysql1832, psmysql1869, psmysql1872, psmysql1886, psmysql1887, psmysql2430, psmysql2431, psmysql2433, psmysql2435, psmysql2436, psmysql2543, psmysql2674, psmysql2676, psmysql2677, psmysql2845, psmysql2846, psmysql298, psmysql3172, psmysql3196, psmysql332, psmysql3398, psmysql3404, psmysql3733, psmysql3812, psmysql383, psmysql386, psmysql387, psmysql3875, psmysql389, psmysql3899, psmysql391, psmysql3933, psmysql3935, psmysql3941, psmysql401, psmysql4037, psmysql4136, psmysql4177, psmysql4182, psmysql4244, psmysql428, psmysql4600, psmysql4688, psmysql4689, psmysql4736, psmysql4873, psmysql4972, psmysql746, psmysql800

Total downtime should be approximately 10 minutes and happen in the next 20-30 minutes. I apologize for not giving a more advance notice.

Update 01:44 AM PST: Ok all services should be restored. Total downtime was around 12 minutes.

A super huge monster query has made our central database choke, and thus the Webpanel is very slow right now. We are working to clear out the query and track down where it is coming from so this does not happen again.

We apologize for the inconvenience. We will post more information as it becomes available.

UPDATE: We’re shutting down Webpanel web services for a little bit to give the database a bit of breathing room while we investigate this issue. This will shut down the Webpanel down completely for awhile.

UPDATE: Our central database is back up and running. We’ve altered the query that caused the problem and so this shouldn’t happen again.

Shemp is one of our older servers and it seems to have met it’s inevitable fate. It is being failed over to new hardware, estimated downtime is 30 minutes.

Update: New shemp is up and rocking a new pair of shorts, apaches configurations are running but should complete shortly

We do apologize for the inconvenience…our Web Panel is currently down, and we have an admin working furiously to restore it as soon as possible.  We don’t expect this will generate much support, seeing as how no one can file a ticket at the moment…but this is a heads up to let you know we are aware of the problem and are fixing it.  Again, sorry for any hassle!

Update 1:44 AM PST The panel should be back up and working again. Thanks for your patience.

We’re quite sorry to say that users on watanabe will be running into problems due to hardware failure.  A member of our admin team is currently working on resurrecting the server from the dead - but doing so has proven to be a bit problematic.  Please know that we’re doing whatever we can to get the sites of users on this server back up as soon as possible.  Further updates will be forthcoming as this matter develops.

Update - 8:39 PM PST: Good news!  The drive recovered!  However, in the interest of preventing data loss, our admin team has decided to fail the machine over to totally new hardware and copy the files over from the old drive.  This does mean that your data will be unavailable while everything copies over. Since this is a newer server tho, I’ve been assured that it shouldn’t take too long for everything to be migrated.

I know this might be a very big problem for some folks — but we’d rather prevent the server from having to recover a larger amount of files somewhere down the line. Expect more updates and progress reports here shortly.

Update 12:35 PM Pacific Friday Nov. 14 2008 We are still restoring data, we are up to the “t”’s in the alphabet assuming rsync is going in order. You may see skeleton directories present without your data if you are after “t” in the alphabet, the data from the old server should fill in those gaps!

Update 3:00 PM Pacific Friday Nov 14 2008 The rsync onto the new hardware finished. If you are still missing data please contact technical support and we will see if we have it on the old server. We apologize for the inconvenience of your data being down for a whole day like this.

Dalitz was unstable and had to be rebooted. While it was booting back up a file-system check was forced which is causing a delay in it coming back up. This typically takes 1-2 hours. Please accept my sincerest apologies for this inconvenience and check this space for updates.

Update 4:20 AM PST: The file-system check is about 80% complete.

Update 4:49 AM PST: The file system check is complete and all services are restored. If you are still having any issues connecting to your site please contact support.