info
So yeah, put me in the “no way” column for this.
I had the pleasure of speaking with Brianna Wu earlier this week. During our discussion, we touched on some of the bullshit going on in her life.
I asked a simple question, “How is your husband doing?”
Her reply: “He’s a wreck.”
Now put yourself in his situation: how would you feel if this abuse was happening to your partner?
It’s time to start looking for ways to change our status quo. I know I am.
In my first post about the attack from the Great Firewall of China, I stuck to the facts. There was a simple reason for this: you don’t want conjecture when your site is down. You want to understand the problem and see suggestions on how to fix it.
This post will be different: these are my opinions and they are pointed. I’ll first note some of the reactions I received, then examine some of the technical subtleties, and conclude with speculation on the motives behind the attack.
By time you finish, you’ll also understand the odd title for this post.
Luckily, the server that provided the page you’re reading now was not attacked. I have purposefully not blocked any Chinese IP addresses at furbo.org. I wanted people there to see what their government is doing from a western perspective.
It’s hard to say how many people in China saw the post, but I do know that some have (at one point I saw twelve visitors from Beijing.) What I found most interesting was that every single person who contacted me used the same word to express their thoughts: “shame.”
Clearly, this attack was not an intentional act by the people of China. No one approves of what their government is doing. I can empathize with this shame: I’d feel the same way if a malicious third party made my browsing ruin the day of a random site owner. Grass Mud Horse!
What happened on my server was not an isolated incident. I have seen many other developers saying “me too!” in the past few days.
I suspected this was the case, but wondered why there wasn’t more discussion about what was happening. I’m guessing that when you’re fighting a fire, there’s little time to discuss the intricacies of how the fricken’ flamethrower is melting the fricken’ network interface.
Some of the discussion came from people who know a lot more about running servers than I do. The most telling was from John Adams:
This is happening, and happening to a number of the sites I manage. What the fuck, China? http://t.co/vPGON0QSgR
— John Adams (@netik) January 23, 2015
Look at his bio: he was one of the engineers that designed Twitter’s infrastructure. If a professional like John is saying “WTF?”, amateurs like me are pretty much screwed. Grass Mud Horse!
Another notable post was from Jamie Zawinski, one of the first people to write a web browser. I was hopeful that his clever response to the Chinese BitTorrent traffic would eventually make it go away.
Unfortunately, it appears there is nothing you can do to quiet the BitTorrent clients. I don’t know enough about this technology to offer any guidance, but someone sure as hell needs to look at the problem and deploy fixes across millions of machines in China. Unless, of course, the Chinese government decides to block BitTorrent client downloads.
The TorrentFreak website has a great overview of BitTorrent’s role in these attacks.
And now would be a good time to pray to your favorite deity that your IP address doesn’t show up here. Note that test is just against our friend “thepiratebay.org”. Your server’s IP address could show up for any other popular site on the web.
That, folks, is DNS poisoning in action. Grass Mud Horse!
Even with packet filtering in place, I still feel vulnerable. Why?
I’m not sure the blocks will withstand another 52 Mbps flood. Remember that up to 65,535 filter rules can be matched by code in the kernel. Your ability to block packets is only as good as the CPU that’s running that code. When I hear that dedicated Cisco firewall hardware is failing, it give me no confidence that my box with 6,500 rules getting 13,000 packets per second will be able to keep up. A back of the envelope calculation shows 84.5 million comparisons per second is needed (or one every 11 nanoseconds.)
For this same reason, don’t assume that any routers or load balancing schemes upstream from your server will be able to keep up with China. There’s no guarantee that your hosting provider will be able to protect your servers or VM instances at rates like we experienced last week.
Still don’t believe me? Look at the first comment on this post at the Internet Storm Center:
I had the same problem starting last Friday, the 2nd. Took out a full load balanced cluster of servers.
Grass Mud Horse!
The biggest unanswered question is why did this happen to the Iconfactory? (Apologies to visitors from China: you won’t be able to look at that link. Grass Mud Horse!)
Our only connection to China is that one of the partners, Talos Tsui, was born and raised in Hong Kong (during the years it was a Crown colony.) It seems unlikely that we’ve done anything to piss off the Chinese government. At least until just now…
The traffic spikes earlier in the week lead me to think that we were being randomly tested for our ability to handle a large volume of traffic. We have fat pipes without automatic DDoS protection. The duration and volume of the probes could determine both of those attributes.
I think James Moore nailed it in his tweet. (And he’s acutely aware of the implications of that analysis: we share a server cabinet with our friends at Panic.)
The Chinese government is not only being deceitful with IP addresses, they’ve also begun cracking down on a mechanism that lets its citizens avoid the bullshit: VPN. Grass Mud Horse!
This action, combined with the DDoS floods, is beneficial to a government that’s intent on isolating its citizens from the free and open Internet. They make it hard to get a packet out of China, but even if you succeed, it’s likely to be blocked by a server that’s been victim of their DDoS.
On the surface, this seems like a good strategy for creating your own private Internet: a network where no packets can enter the west or leave the east.
The Internet was designed to route around damage. While the ability to withstand a nuclear war is a myth, the protocols we use every day were created to be robust against infrastructure loss. Even when that section of the network is the size of China.
But even more important than the technology is the people who use the Internet.
The GreatFire.org website monitors the Great Firewall and provides information in both English and Chinese. An informed populace is a powerful one.
There are also efforts underway to redirect bogus traffic to mirror sites. Geeks have never had a problem staying one step ahead of those who attempt to control.
From a personal perspective, the DDoS attack from China made me acutely aware of how screwed up things are over there. The government’s actions have pissed me off and I’ll now do anything in my power to thwart their efforts. Like writing this piece.
And given the feedback I’ve received, I’m not alone with this point-of-view. People are fighting back. I’m hopeful that over the course of several years, we’ll find better ways to cope with the idiocy of the Chinese government than to tunnel under their firewall and block their IP addresses.
If you doubt people’s ingenuity in routing around roadblocks, take a moment to learn about the Grass Mud Horse:
(The whole video is informative, but be sure to watch the end.)
Dear Tim,
I’m writing today about the latest kerfuffle on the state of Apple’s business.
Marco is a brilliant developer and I’m proud to call him a friend. I also know that, like many of us geeks, sometimes his words verge on hyperbole.
You’ve always struck me as someone who relies on facts for your decision making, so I’d like to provide some background from a geek’s point-of-view. No hyperbole. No bullshit.
The reasons Marco’s piece got a lot of initial attention was because the basic message resonated with a lot of people in our community. But as usual, idiots with ulterior motives twisted the original message to suit their own purposes.
The good news is that none of the problems us geeks are seeing are show stoppers. We’re not complaining about software quality because things are completely broken. There’s still a lot to love about OS X and iOS.
But this good news is also bad news. Our concerns come from seeing the start of something pernicious: our beloved platform is becoming harder to use because of a lot of small software failures.
It’s literally a death by a thousand tiny little cuts.
Apple may not be aware of the scope of these issues because many of these annoyances go unreported. I’m guilty of this when I open a Finder window on a network share. While the spinner in the window wastes my time, I think about writing a Radar, but a minute later it’s forgotten. Until the next time.
As a developer, I know that Radar is my best channel to give Apple constructive feedback, but writing a report is a time consuming process. Those links above consumed about two days of my productivity. Nonetheless, I’ll keep harping on my 20K+ Twitter followers to do their duty. These problems won’t fix themselves.
I’m also wary of looking at past OS X versions with rose colored glasses: there always have been bugs, there always will be bugs. But I have a pretty simple metric for the current state of Apple’s software: prior to the release of Yosemite, I could go months between restarts (usually only to install updates.) Lately, I feel lucky to go a week without some kind of problem that requires a complete reset. I experience these problems on a variety of machines: from a 2009 Mac Pro to the latest Retina iMac.
A bigger concern than my own productivity is how these quality issues affect Apple’s reputation. Geeks are early adopters and effusive when things work well. But when we encounter software that’s broken, we have a tendency to vent publicly. Yesterday’s post by Marco was the latter (spend some time on his blog and you’ll see plenty of the former, as well.)
Because a lot of regular folks look to us for guidance with Apple products, our dissatisfaction is amplified as it trickles down. When we’re not happy, Apple loses leverage.
It also leads to a situation where your product adoption slows. To give you some personal, and anecdotal, examples:
Before I close this letter, I’d like to offer a final observation. Apple is a manufacturing powerhouse: the scale of your company’s production line is an amazing accomplishment. Unfortunately, software development is still a craft: one that takes time and effort to achieve the fit and finish your customers expect.
Apple would never ship a device that was missing a few screws. But that’s exactly what’s happening right now with your software products.
I’ve always appreciated the access Apple’s executive team provides via email, so thanks for your time and attention to this matter. Please feel free to contact me for any further information or clarification.
Yours truly,
Craig Hockenberry
How many apps on your iPhone or iPad have a built-in browser?
Would it surprise you to know that every one of those apps could eavesdrop on your typing? Even when it’s in a secure login screen with a password field?
Here is a proof-of-concept (ZIP file) that shows how an app can do this. For those of you who don’t have Xcode installed, here’s a video that shows what’s going on:
A few things to note about what you’re seeing:
No, this is not a WebKit bug.
The Shadow DOM does a great job of protecting static user content on a web page. It’s not possible to use JavaScript to view the contents of an input field on iOS since the current value attribute is actually being held in a platform-native control. The value of that control is uploaded when the user submits a <form>.
I don’t know for sure, but I suspect that the keyCode attribute of the KeyboardEvent in the JavaScript event handler is provided for backward compatibility. This API has been deprecated but there are still plenty of web pages out there that use it to handle keyboard input.
In fact, both the techniques shown in the sample app can be used for good as well as evil. Changing the content of a web page is a good thing when it’s done to make a page more readable or accessible. Handling keyboard events can also guide a user through a complex form or make viewing a slide show easier.
These are not inherently bad web technologies. The problem is that an iOS app has as much access to these technologies as the developer of the web page.
Websites have been dealing with username and password attacks for as long as there have been <input> fields on their pages. One of the primary goals of OAuth was to keep a user’s login information away from an external website or app.
OAuth does this by exchanging cryptographically signed tokens between the site where the user has an account and the app or web service that wants to access that account. A key factor in making this secure is that the exchange of these secure tokens is done through a trusted channel: the user’s web browser. Twitter has required third-party developers to use OAuth since 2010.
As early as 2008, the developers of OAuth recommended the following:
We’re trying to ensure that users are only exposed to the safest way to disclose their location using OAuth. To do this, it’s critical that a fundamental principal of browser-based authentication is followed; that the contexts of the third party application and the web service authentication remain separate. To allow users to grant trust to an application, they must perform the OAuth action within their web browser, not within the applications themselves. Otherwise, there is no way to verify the identity and authenticity of any page which asks for their username and password. Users must not ever enter their username and password into a third party application when a browser-based authentication API like OAuth is available.
There is always a tradeoff between usability and security. Doing the OAuth token exchange with an in-app browser makes it easier for a user to login, but they’ll have no idea if their personal information was captured. That is why Twitterrific did its token exchange in Safari, even though it’s a more complex user interaction and a more difficult technical implementation. As a user, I know that there’s no way for my login to be compromised when the transaction involves Safari.
Unfortunately, Apple’s current App Review policy does not agree with this recommendation or with Twittterrific’s previous implementation. This is why our update for iOS 8 was delayed—it was the first time since the launch of the App Store that we haven’t had a new version on release day.
(Apple folks can learn more about this situation by reviewing Radar #18419943)
Apple has taken a strong and welcome stance on privacy. They’ve recently been implicated in some high profile attacks so they definitely have skin in this game. Hell, they even want to protect us from the US government watching what we do online!
There’s no denying that the behavior demonstrated above could be very harmful in the wrong hands. It’s also Apple’s job as the gatekeeper for iOS to keep malicious apps out of the App Store. But how?
I don’t think it’s feasible to catch misbehaving apps at review time. There are a huge number of apps that need to be reviewed every day, especially when new versions of iOS are released. Many of these apps use in-app browsers which would require extra time and effort to vet. Longer review times benefit no one: developers, Apple and our customers need timely updates.
It’s also very easy to an app to hide any nefarious activity. JavaScript has an eval() function that makes it easy for code to be obfuscated and very difficult to be checked at review time. Look at this page and see if you can guess how the uppercase text was created. Then view the HTML source and see how wrong you were.
Additionally, an app that wants to collect your information can easily implement a remote switch that disables the functionality while the app is in review. App reviewers won’t stand a chance.
Changing how WebKit and UIWebView behave isn’t practical either. To prevent this keylogging technique, Apple would need to release a new version of iOS for each version that included Safari and WebKit. Do you really think they’re going to do a point release of iOS 3?
And this brings me back to protecting users with OAuth. It’s designed to avoid these problems and works well to maintain privacy. Granted, it goes against section 10.6 of the App Store Review Guidelines, but in my opinion, this is a case where user security trumps usability. Apple should change their policy for apps that use OAuth.
Another goal of this essay is to increase user awareness of the potential dangers of using an in-app browser. You should never enter any private information while you’re using an app that’s not Safari.
An in-app browser is a great tool for quickly viewing web content, especially for things like links in Twitterrific’s timeline. But if you should always open a link in Safari if you have any concern that your information might be collected. Safari is the only app on iOS that comes with Apple’s guarantee of security.
(For the record, we never collect any private information in any of the Iconfactory apps. And we never will.)