breakdowns

Blaine Cook still working at Twitter, according to Twitter jobs page

Owen Thomas · 06/02/08 04:00PM

Since leaving Twitter, former chief architect Blaine Cook has been sparring with his former employers over the cause of Twitter's outages. It's a peculiar battle of words, with Cook never mentioning Twitter and Twitter never mentioning Cook. But perhaps things aren't that unfriendly. According to Twitter, Cook (second from top and from right) is one of the reasons people should come work for Twitter.

Twitter's existential crisis a masterwork of fingerpointing

Owen Thomas · 06/02/08 02:00PM

Twitter's founders are waging a behind-the-scenes war with Blaine Cook, the blogging service's former chief architect. The subject: Who's responsible for the service's perpetual outages. TechCrunch's Michael Arrington ran a series of leading questions about Twitter's infrastructure, attributing them to "people who say they’ve seen Twitter’s architecture." I don't think that's true, if only because I received a similar set of questions, before Arrington's post went up, from a source who identified himself as a "friend of Blaine." In their official response, Twitter cofounders Jack Dorsey and Biz Stone — they're the two one always forgets about, because they're not as interesting as Evan Williams — go out of their way to avoid naming names.

PayPal closes the border

Owen Thomas · 05/26/08 03:00PM

When Peter Thiel launched PayPal a decade ago, he had a vision of a global payments mechanism which would accelerate the withering-away of the nation-state. And then he sold it to eBay. eBay's latest failure to transform the international monetary system is quite literal; for almost two weeks, PayPal has had a bug which prevents it from collecting cross-border payments for subscriptions — this while its new president, Scott Thompson, has been touring the globe. The error: a bit of code in a drop-down menu. Subscriptions are a small part of PayPal's business, though vital to the complaint-prone blogging class. Regardless, it's a trivial bug that should have taken minutes, not weeks, to fix; that eBay has not yet done so would seem to speak to a profound rot in its technical organization — which Thompson headed up as CTO before his promotion.

Red Herring website outage an unfortunate coincidence

Owen Thomas · 05/23/08 03:20PM

Alex Vieux's Red Herring isn't just poorly managed; it's unlucky as well. I just got off the phone with Vassil Mladjov, CEO of Blogtronix, the blog-software company which hosts RedHerring.com. He blames the site's outage — which comes the same week as the Herring's eviction from its offices and the cancellation of a Herring event in China — on a bug involving log files, and says the site will be back up shortly. Mladjov adds that unpaid bills aren't the issue; Blogtronix arranged to get paid through a barter deal.

Google Sites open for everyone — but closed for current customers

Owen Thomas · 05/22/08 01:40PM

Google engineering manager Andrew Zaeske is all but breaking his arm patting himself on the back for launching Google Sites to the public, after testing the wiki tool among Google Apps customers, a group of mostly business users who use customized versions of Gmail and other Web-based Google software. One of those customers would just as soon have Zaeske talk to the hand:

Twitter mission-critical for Michael Arrington's emotional stability

Owen Thomas · 05/21/08 03:00PM

Here's a hefty guilt trip for Twitter's overtaxed engineers to bear: TechCrunch editor Michael Arrington has blamed a recent outage for making him feel bad. He writes: "I'm in a particularly bad mood because I have food poisoning (thanks very much Grand Hyatt Seattle) and Twittering it was going to make me feel marginally better because a bunch of people would say something nice in a reply. But they take even that away from me." It's official: Arrington is self-medicating with Web 2.0. Folks, should we stage an intervention? (Photoillustration by Jackson West, from an original by Scott Beale/Laughing Squid)

NSA site goes down in blaze of badly-engineered DNS glory

Jackson West · 05/16/08 03:40PM

The website of the National Security Agency went down yesterday. No, it wasn't subject to "cyber terrorism," just bad network management — specifically, its domain-name servers, according to Arbor Networks' Danny McPherson. But don't worry, everyone, the site is back up! Still, the fact that it sports a Flash interface doesn't exactly inspire confidence in the engineering ability of our information overlords. (Photo by Ryan Lackey)

Yes, ex-Twitter architect Blaine Cook knows what scalability means

Nicholas Carlson · 05/12/08 04:40PM

Before he left, former Twitter architect Blaine Cook's job was to assure the service's scalability," or the ability to expand to meet growing user demand. Didn't happen. Instead, Twitter leads all social networks in downtime so far this year. In a post to his blog today, Cook wants to make one thing very clear: He knows what scalability is, OK? If one eliminates database chokepoints and throws enough servers at the problem, even crappy code written in Ruby on Rails will work. Cook just couldn't make it happen for Twitter. Here's the 100-word version:

Yahoo's real leadership problem: David Filo

Owen Thomas · 05/07/08 07:00PM

Everyone's piling on Jerry Yang, saying Yahoo's founder-CEO needs to go. Why? The weak stock that provoked Microsoft's unsolicited bid may have been the result of his absentee ownership over the years. But Yahoo's deeper problem is the rot in its technical prowess. And that has everything to do with the quieter cofounder, David Filo. Filo has stayed behind the scenes, but wields considerable power over Yahoo's infrastructure. Requests for more hardware go through him, for example. When Yahoo executive Jeff Weiner joked in an internal all-hands movie about not going through IT because it was "too much paperwork," the audience surely laughed because they knew exactly what he meant.

MySpace's technical triumph

Owen Thomas · 05/06/08 03:00PM

The conventional wisdom in Silicon Valley is that MySpace, based in Los Angeles, is a tech nightmare, blaring songs through a user's speakers while crashing all the time. Skilled engineers are in short supply down south, so the website must be falling over all the time, right? Not so. Pingdom, a website-monitoring service, has tracked how often some of the top social networks have gone offline. Twitter, based in Web-savvy San Francisco, has been down for 37 hours from January through April. MySpace has been up 99.96 percent of the time. That's 33 percent less downtime than Yahoo 360, and 60 percent less than Google's Orkut. Score one for the LA crowd. The chart:

Facebook posts advertiser's driver's license for all the world to see

Nicholas Carlson · 05/02/08 12:40PM

Musicians can promote their work through Facebook's Musician Pages. But before allowing them to upload music files, Facebook requires administrators to submit scans of their driver's licenses, to keep on file in case claims of copyright infringement come up. Last night, one of these administrators, an employee at Ping Pong Music, discovered Facebook had posted his license publicly on EMI artist This World Fair's page. He took a screenshot, which we've included below.

Twitter cans another engineer

Nicholas Carlson · 04/24/08 10:40AM

When Twitter hired Lee Mighdoll as VP of engineering and operations in January, cofounder Biz Stone called him the "perfect match" for the company. Not anymore. Mighdoll is out after just three months of the job. "The match was not perfect," Stone told SAI in an email. Mighdoll is the second engineer reported to have left Twitter in the last two days; architect Blaine Cook fled the country yesterday. Neither was able to fix Twitter's oft-reported propensity to crash. We hear the final straw to break Biz Stone's back was not the breakdown yesterday that TechCrunch described as a "privacy disaster". Makes sense, because isn't that Twitter's raison d'être?

Ning fires VP of operations two days before major outage

Owen Thomas · 04/23/08 10:40AM

Here's how things usually work: Have a major outage, then fire your operations guy. At Marc Andreessen's Ning, the social-network Web host best known for its porn sites, things run a bit differently. On Monday, CEO Gina Bianchini fired VP of operations Alexei Rodriguez. On Wednesday, the company saw all of Ning's networks go offline. We hear Rodriguez failed to deliver a promised upgrade to Ning's systems that would have avoided the problem; the outage was coincidental but almost inevitable, given Rodriguez's omission. The larger problem for Ning: No one seems to care that it was down. When you offer porn and still no one complains that they can't get to it, you have a problem which goes much deeper than database configurations.

Web host experiences temporary truth in advertising

Owen Thomas · 04/22/08 07:20PM

Media Temple, a Web host which counts several prominent bloggers among its clients, offers an accurate description of its service in a recent downtime announcement: "(mt) Media Temple engineers have brought db servers 13-16 back online and apologize for the temporary convenience." True: It would be convenient if more bloggers were brought offline. [Media Temple]

Clinton's campaign accused of hacking Obama blogs

Nicholas Carlson · 04/21/08 11:20AM

In the clip embedded below, an Obama supporter demonstrates how "someone hacked into Barack Obama's site" and changed a link into Obama's Community Blogs so that it instead directs users to Hillary Clinton's home page. We're shocked. Obama's Web presence is the product of Facebook cofounder Chris Hughes. Anyone familiar with that platform knows it's entirely resilient to human error or internal corruption. The video demonstrating the hack:

Tumblr security breakdown leaves scenesters exposed for 40 minutes

Nicholas Carlson · 04/15/08 06:40PM

While editing administrator code today, Tumblr founder David Karp and developer Marco Arment inadvertently published private user data for 40 minutes. Karp reports on his blog that 27 email addresses were exposed. He told us that four accounts — including popular Tumblr blogs by Julia Allison and Pete Nidzgorski — had their passwords changed. Karp told Valleywag he knows who changed the passwords. "He was a registered user, so we were actually able to look up his info," Karp said. The suspected hacker won't lose his Tumblr account. "I don't think we'll be taking this out on him," Karp said.

Oklahoma exposes sex offenders' Social Security numbers

Jordan Golson · 04/15/08 01:40PM

A blogger noticed the Oklahoma Department of Corrections was making the Social Security numbers of thousands of registered sex offenders viewable through a security hole on a state website. Even after he notified the department, the problem remained. Only when he showed how to make state employees' personal information visible did the state fix the problem. [The Daily WTF]