Don’t Give Up On Plastic

Carl (verified to be logging in from chat.plastic.com) comes into #plastic and provides a few updates.

WARNING: We’ll be the first to remind people not to get excited just yet - we could still be in for a wait.

Some choice tidbits:

“…At some point I decided it’s down, everyone hates me, might as well keep it down until I have things working well. Up until now it’s always been so much hurry to ‘fix’ things that some basic infrastructure issues just never got addressed.”

“…people have been nothing but supportive. I just know what y’all really think.” 

“Shouldn’t be too long now. Just got the syncing of the clones working. I think the VPN’s next on my list.
 Or, no, I ended up doing everything via ssh, I guess. So I just have to do a remote sync test. And drill through some remote terminal session stuff.”

“…I was hoping to get Plastic back up Thursday, then Friday, then Monday, and now, well, if I’m talking to someone besides myself, I won’t say, but. I don’t think there are any unknown unknowns left.”

“I think only a single unknown known, in that I haven’t set up replication on the database. Really it’s a known known, since I’ve had that all up and running before, only took an hour or so, but I’ll want to look at pulling a slave out for dev work, re-inserting it for sync. Not really sure about that, except I already decided a while ago that failover belongs on the application level.”

(carl is asked when Plastic will be up)

” When it’s up. I mean, it’s all been working for a while, now. Might be tomorrow. Might not.” 

“Here’s the sort of thing that makes delivery dates so hard: so I’m setting up ESX Server across multiple machines, yes? Everything is going along well. And then: The documentation just sorta runs out. And then I realize. This is the place where one department would usually hand things off to the next department. Except I ain’t got nobody to hand-off to. No, it’s just time to deal with a whole new set of issues.”

These were the more relevant excerpts from what Carl mentioned in chat. We’ll keep you informed of anything new.

A collection of the verified and close to verified updates on Plastic’s status. For more information, check out the FAQ. 

July 15th

“We’ll probably be back up by late Monday. Really.”
Carl posts this update on the 15th, spends most of the weekend in Plastic Chat.

July 7th

Carl pops into #plastic and asks those present to test out a URL.

June 21st

“We now have a working server image; currently testing across various hardware, possibly using virtual devices and drivers; also need to yet make some decisions on RAID. Another day or two is all.”

Plastic Chat returns, with the aforementioned update message.

Early June

Plastic Chat goes offline for a couple of weeks. 

May 23rd

As promised, the site goes down again.

May 16

Although the site is back up, it seems the tech issues are still not entirely solved. The following excerpts are reportedly attributed to carl.

“The thing is with these expectations is that  I need the slack…”

“I don’t want to say  anything until I know what I’m saying. Because if I say the  wrong thing, make a bad estimate, then I’m savaged. Pilloried.  Crucified.”

“Anyway, we had no machine from which to broadcast said status message…We just went from three to two [locations].  I have to find another sucker.”

“…I’m not done, and the site’s gonna go down again.”

May 13th

Plastic springs back to life.

May 10th

An explanation for the downtime appears on Plastic Chat. 

“The old custom kernels from a couple years back are incompatible with the packet inspection that’s now taking place in hardware before it reaches the web application servers. So the web app has to be patched to not rely on the custom environment anymore.”

Sometime around May 1st

Plastic goes down. Initial uncertainty about the reason, talk of a possible DDOS attack on the site.