Year: 2009

An expensive lesson

One of my favorite sayings is “the best education comes from the school of hard knocks—if the tuition doesn’t kill you.” We’ve learned a very expensive lesson this week from a bug in iTunes.

The problem started when Pickin’ Time was approved on October 9th. We’ve all been busy working on other projects, so we delayed the launch until October 27th by setting the availability date in iTunes Connect. That gave us time to get the website and other promotional materials finished.

Unknowingly, it also gave us plenty of time to shoot ourselves in the foot. iTunes uses the date of approval as the “release date”, not the day you set for availability.

iTunesPickinTime

The result is that there’s no sign of Pickin’ Time in any of the “new” application lists: the casual browsers aren’t finding the app. There’s none of the momentum that’s so important in the App Store.

It’s frustrating that success, after months of hard work, can hinge on something unknown and completely out of our control. The customers who have found the product, are leaving great reviews [iTunes link]. It’s not a question of the game’s quality, it’s a problem of being lost in quantity.

Hopefully, this post will help another developer avoid this unenviable situation. We’d also appreciate any help in getting the word out about Pickin’ Time. Thanks!

Written on October 29th, 2009 for Miscellaneous

Waving a red flag

As a result of my last essay, it has come to my attention that there is a simple and effective way to get Apple’s attention for critical bug fixes. An email to appreview@apple.com that explains the critical problem and which product is affected will help speed your update through the system.

Use this email address only for issues with application reviews. If you have a problem with iTunes, getting paid or any other part of the App Store business, use the Contact Us section of iTunes Connect.

Don’t abuse this important communication channel, or we’ll lose it.

Written on July 23rd, 2009 for Advice, Business, Development

Year two

As we approach the first anniversary of selling things on the iTunes App Store, I’d like to take the opportunity to look at where we’ve been and where we’d like to go. A lot of good things have happened since last July 11th, but there’s still much room for improvement.

Note: These words may be mine, but their origin is from conversations with hundreds of iPhone developers. Saying that there was a lot of discussion about the App Store in the bars around WWDC would be the understatement of the century. The length of this essay is proof.

There is hope

As developers, we have a tendency to look at things that are “wrong” and complain. It’s also important to look at what’s been improved in the past year: it gives me hope that we’ll eventually see some of the suggestions I make in this essay.

Here are things that have helped us sell more product:

  • Reviews only from people who have purchased the app.
  • Promotion codes.
  • Better categorization.
  • A nationwide ad campaign.

The root of the problem

Building the App Store using the iTunes infrastructure was a brilliant move on Apple’s part. It allowed them to sell software to millions of users with a minimal amount of development. Look at it this way: would you rather have lost the first 6 months of revenue while waiting for Apple to perfect the system used to deliver our applications?

But now iTunes presents problems for both Apple and third party developers because of this simple fact: software is not music. I believe this is the basic issue that both parties are coming to grips with:

  • Music and software differ greatly in the number of titles. Apple is currently touting 50,000 applications in the App Store. While an impressive number in this nascent category, it pales in comparison to 10 million songs.
  • As developers, we may be small fish in this sea of titles, but we’re more important to the health of the iPhone platform than musicians are to the health of iTunes. People like Walt Mossberg don’t talk about the importance of music when reviewing the competition.
  • iTunes customers do not have functional problems with songs. People don’t have to learn how to listen to music. Anyone who’s done customer support can tell you that’s not true for software.
  • There are never new versions of a song. Yes, there are remixes and other adaptations, but these are labeled, marketed and sold (via SKU) as unique products. Software, on the other hand, usually gets a new version every few months.
  • Music can’t damage the device. Malicious software can.
  • Music doesn’t depend on the network while its playing. Our software can break or change behavior because of this dependency on external data.
  • Many musicians sell their content through third parties (media companies), most developers do not.

Finally, the complexity of system with 10 million database rows backed by a worldwide content distribution mechanism is not lost on us. Changes to infrastructure of this scale won’t happen overnight.

Approvals

Apple needs to face the fact that the current review process is not scaling well. When the App Store launched, there were a thousand apps and reviews took less than a week.

Now, with 50,000 apps, we’re currently looking at three week lead times for reviews. When there are 100,000 apps are we going to be waiting a couple of months for approval? Or more?

And even with these long review periods, there is no shortage of applications being pulled from the App Store. Some developers wonder if approval is less about consumer protection than it is about protecting Apple’s interests.

This long review process has a serious impact for both developers and customers:

Maintenance releases aren’t viable

There are always bugs that pop up which can be fixed very easily. Twitterrific currently has a problem with marking tweets as a favorite (caused by a change at Twitter.) We could literally fix this bug in five minutes, but it would still take three weeks to get it onto a customer’s phone.

Patching an existing release and doing a quick update is something all developers have been able to do on other platforms (Mac or Windows, for example.) But after spending several weeks in a queue, your 1.0.1 release is likely to be out of date because in the interim you’ve been working on a 1.0.2 release. If you replace your 1.0.1 release with the new 1.0.2 release, you go to the end of the queue and start waiting again. You end up in a cycle where these small releases never see the light of day.

The net effect is that many developers are giving up on these maintenance releases. And that’s bad for customers.

Emergencies

No matter what some of us might think, developers are only human. Sometimes we do stupid things, and when we do, we need to resolve these issues with critical updates. In these cases, we need to jump to the front of the review queue. As someone who’s been the victim of stupidity more than once, I have no problem giving up my place in line for someone who’s suffering the consequences of a dumb mistake.

Solutions

There are a couple of things that could help with the review process. The first is to verify developers, not apps.

I am perfectly willing to be held accountable for my actions. So is every other developer I’ve talked to. If we do something evil, pull all our products off the App Store shelves. Start legal action. Shut down our business. Those of us who respect our customer base and value our relationship with Apple will not be affected.

We don’t mind waiting a few weeks for a major version release (e.g. a 2.0) because it’s likely something that has already taken many months to create. Use an initial or major release as a point to establish the trust between Apple and the developer. Then, incremental releases (e.g. a 2.0.1 or 2.1) can be approved quickly because of that trust. If a developer breaks that trust in a point release, penalize them.

As my company releases more products, I also have more incentive to maintain that trust level (pulling multiple products hurts more than pulling just one.) Knowing that, Apple could spend less time reviewing each new product from a existing developer.

With critical updates, we won’t need to jump to the front of the review queue often. The best thing would be to give each developer a limited number of “fast track tickets” and tell them to use them wisely (no exceptions for developers who need “just one more.”)

Upgrades

For readers who aren’t developers, let me explain the simple economics of our business: we sustain our efforts with upgrade revenue. Getting the initial sale is only the beginning of a long-term relationship with a customer.

Here’s how it works:

  1. You know the number of total customers for your product. (Some developers call this the installed base.)
  2. The percentage of users who upgrade a product is predictable. For most products, it’s normally around 40-60%.
  3. You know how much the upgrade is going to cost.

Given this information, you have a very good idea of how much money the upgrade will generate. That, in turn, tells you how much money you should spend creating that software.

Unfortunately, the App Store doesn’t allow us to generate upgrade revenue. We either do free upgrades or charge for a completely new product (SKU.) The former is bad for developers because we don’t see any incremental revenue. The latter is bad for customers because they don’t get a price break.

Many developers who have been in the App Store since its inception are facing this problem now: we’re working on our 2.0 releases and struggling with how to make money with them. Even if your product is selling well, it’s difficult to justify adding new features if there’s no extra revenue to pay for them.

Free upgrades also mean that great products (with smaller customer bases) will eventually wither and die. “There’s an app for that” will only apply to products with a mainstream customer base. Niche and vertical markets will not be served well.

For game developers, this is less of an issue: each new version can be marketed as a unique entity. But if you apply that same approach to other apps, you end up with “Twitterrific: The Awakening”, “Twitterrific: The Second Coming”, “Twitterrific: The Return of the Tweet”, etc. This will only make the App Store more confusing for customers of those applications.

Finally, major updates can’t be released on a predictable date. If you try to set an application’s availability date to some time in the future, the current product in the App Store becomes unavailable (so customers can’t buy the product while it’s being reviewed.) Apple, more than anyone, should understand the benefits of controlling your product release. There’s a lot more to it than a row in iTunes’ database.

Solutions

Why can’t apps be upgraded just like music? If you look at version 1.0 being like a song with DRM, and version 2.0 being a song without DRM, you’ve solved most of the problem. The only difference is that we need to upgrade the song more than once.

The upgrade should also have its own release date: giving developers the ability to time press releases, website launches and other supporting activities.

Better rules

We need better rules for the types of applications that are allowed in the App Store: the wording in section 3.3 of the license agreement is just too vague.

It’s likely that Apple’s legal team prefers to handle marginal cases on an ad hoc basis. But that’s a system that won’t scale and whose cost will quickly outweigh the benefits. How many lawyers will be needed when there are a million applications in the App Store?

With so much left open to interpretation, it makes it easy for a developer to wave a flag and generate bad press for Apple. As more applications are added to the App Store, there has also been an increasing number of these sensational “app rejection” pieces in the tech media. It’s event starting to go mainstream at places like CNN.

The situation is exacerbated by developers who are starting to realize that this “app rejection” attention by the media is good for sales once the controversy is resolved.

A clear set of rules also lets developers feel confident that their efforts will be rewarded with placement in the App Store. At present, it’s a crap shoot. When you roll snake eyes, you have no other channel to sell your product and you’ve lost a lot of time and money.

We also realize that there’s one rule that’s hard to define: “offensive content.” It’s completely subjective.

When you look at the music industry, you’ll see they’ve had a similar problems since 1985. After several decades of input from thousands of individuals in public forums, there are still controversies surrounding the rating of music. From a social point-of-view, it’s an impossible problem to solve. The current “solution” is to let the record companies police themselves.

As iPhone developers, we have one company making these policy decisions: Apple. The lack of transparency in their decision making process makes it impossible for us to know what’s acceptable and what is not. For all we know, the rules are being made up (and changed) as we go along.

Solutions

A clear set of rules for reviewers and developers will make the process easier for everyone.

Apple should publish what can and cannot be developed for the App Store. Provide examples of what is OK and what is not. We have Human Interface Guidelines for how our applications behave, why don’t we have similar guidelines for the functions our applications perform?

For the situations that fall outside of these rules, provide developers with a means to get pre-approval for application ideas. If running our ideas by Apple’s legal department comes at an extra cost, that’s fine. The objective is to waste less money on app ideas that will never get published.

Better experience for customers

Both developers and Apple want the best possible experience for our customers. As the seller, Apple focuses on the sale. As developers, we focus on getting people to make the purchase in the first place.

So how can we make it better?

Product evaluations

As it now stands, customers are making their purchase decisions with a few screenshots and a short product description. This is often not enough to convey the benefits of our products: especially ones that are more complex (and have a higher cost.)

Imagine the problems music companies would have if you couldn’t preview songs on iTunes. They’d have a lot of unhappy customers who bought the wrong song. And if the workaround was for customers to go to the artist’s website to listen to a sample or see a video clip, they’d lose a lot of sales because it’s a hassle and an unpredictable experience.

The current situation akin to walking into a phone store and looking at a hardware mockup that has a sticker of a screenshot and no battery. Compare that to walking into the Apple Store and using a fully functional iPhone.

Solution

The best solution would be to allow a time limited version of the application to be downloaded for free. After evaluation, the customer could purchase the application. Developers could do this now if in-app purchases were allowed on free versions of applications. (There is even precedence for this approach: AT&T is doing something similar with its Navigator app.)

Apple has stated that “free remains free.” I can see why they want to do that, it’s a way to avoid applications that try to use bait-and-switch schemes. (Although, one could argue that we’ll see the same problem with paid apps.)

So why not compromise on a solution? iTunes on the desktop and the App Store application on the device both have access to excellent multimedia capabilities. Why not use these facilities to allow a couple of minutes of video for each app? It’s not as good as trying out the application, but it’s certainly better than five screenshots. Let developers upload a QuickTime movie with iTunes Connect and both they and customers will benefit.

Respond to reviews

Again, music is not software. Music is simple, you either like the song or you don’t.

For software, customers see the reviews pages not only as a place to comment on an application’s suitability, but also as a forum to voice their problems and concerns. As developers, it drives us crazy that we have no way to address those issues.

Solution

Give us a way to contact the customer. Ideally, we would be able to post a public response in iTunes so other users with similar issues could benefit.

Again, I can see this being too much to ask of the iTunes infrastructure. I also doubt that Apple wants to provide us with customer’s email addresses (which can be considered private data.)

So why not present a form for developers to fill out within iTunes? Our comments are entered into a text field and sent to the customer’s email address by Apple. We could provide our own contact information in that message for followup conversations.

Finding apps

Songs and games work in the current “top hits” ranking. Popularity for this type of content is ephemeral. This type of content is also easily identifiable by a brand name (“Death Cab for Cutie”, “Flight Control”, etc.)

The problem is finding applications that aren’t at the top of the charts. It’s incredibly hard to find the “that” in “there’s an app for that.” Between keyword spamming and the sheer volume of choices in each category, customers can’t find what they want.

Solution

The answer to this problem is organizing search by solutions, not titles or sales rank. During this summer season, customers are interested in applications that:

  • Entertain while traveling to/from their vacation destination.
  • Guide them around that destination.
  • Helps them relax during the break.

The only thing you’ll find if you search the App Store for “Paris” is that there are a lot of fricken’ icons with the Eiffel Tower on them.

Macworld, with its App Guide, has figured this out. Apple should look at the success of staff favorites and expand upon that within iTunes.

Charge us more money

The $99 developer fee is great for making the App Store all inclusive. But the needs of larger, full-time developers are much different than those of those doing it part-time or as a hobby.

Solution

Charge $999 for premium service. For professional developers, this cost is not prohibitive and would allow Apple to provide additional services such as:

  • Shorter review times. The trust level between Apple and the developer is higher when there’s more money on the table.
  • More “fast track tickets” as mentioned above. Larger development shops have more staff doing stupid things, so give us more freedom to deal with it.
  • Discounts for test devices. With each product release, it becomes more difficult for developers to support the new hardware. The primary factor here is that an AT&T contract is required to get the test device. Something similar to the Mac hardware discounts for ADC members would help significantly.
  • Background artwork for our products in iTunes. Getting a nice background means a lot to us: it makes our products more attractive to potential customers. Let us pay for the time it takes to put it up.
  • Approval of ideas prior to development. Developers have crazy ideas. Before spending months of development on that wacky thought, it would be nice to get pre-approval from Apple on the concept.
  • More devices in the Program Portal. The current limit of 100 devices is a joke for developers with multiple products: if you have 5 products, you can only have 20 devices per app. That, combined with multiple hardware releases in the 12 month period, means we’re effectively prevented from testing our applications until July 12th.
  • A contact in Apple Developer Relations. Sometimes we need to talk directly to someone at Apple.

Pricing

I’ve talked about pricing before, but not much has changed. The addition of in-app purchase is great for certain kinds of applications: our upcoming title, Ramp Champ, makes great use of it. But most apps cannot take advantage of this new feature and are left struggling at the ringtone price point.

These low prices are making decisions for us. If you gave an iPhone developer $50,000 to spend anyway they choose, they’re going to make 5 ringtone apps at $10,000 each instead of one killer app for $50,000. In a gold rush, it’s better to have five small claims than a single large one.

The problem is that the long-term success of the iPhone platform will be defined by killer apps, not throwaways that you replace on your home screen after a few months.

Unlike other sections in this essay, I don’t have specific suggestions to make. I do think that the solutions presented above will, in aggregate, help us get some upward movement in pricing. And that would be very good news for those of us making a living from the App Store.

Conclusion

There’s no denying it: the last year has been a good one for both Apple and third-party iPhone developers. That’s not to say there isn’t room for improvement, especially at the point-of-sale.

The suggestions I’ve presented above are intended to help us grow this business and keep the ecosystem healthy. Every developer’s fear is that Apple doesn’t want an open a dialog regarding the App Store. It scared the shit out of me when our questions weren’t answered at WWDC.

A successful partnership is one where both parties work to the benefit of the other. If our needs are ignored, it will only lead to disenchantment. Working with the developers that are driving this new platform is Apple’s best long-term business strategy.

Written on July 10th, 2009 for Business, Development

Brain farts

What happened?

In spite of plenty of advance warning from Twitter, we got caught by the Twitpocalypse bug.

For the 2.0.1 release, we had tested our software extensively. I actually wrote an emulation layer on top of the code that reads data from Twitter that added a large number to every ID read from Twitter. This testing uncovered several bugs which were fixed and incorporated into the MGTwitterEngine open source code.

Unfortunately, this testing didn’t take into account that the library we use to parse the data (YAJL) includes a range check that ensures signed values, which are allowed by the JSON specification, will fit into 32-bit storage. The “YAJL Error 3” is that library telling us that the range check failed.

In hindsight, I should have fed the parser some large unsigned integers. In the wonderful world of software development, we call this a brain fart.

How did we respond?

Unfortunately, the Twitpocalypse occurred on Friday evening, just as my wife and I were heading out for a high school graduation. I quickly shot off an email to Twitter asking a few questions (I had mistakenly thought that there were problems with data being returned by an authenticated connection.)

After a couple of hours of iPhone email and SMS, it was clear that we were going to need do a new release. We had a critical bug that affected thousands of users. And our fear was this submission would take longer than normal: many developers are submitting 3.0 updates.

Life officially sucked at this point, and the graduation ceremony was memorable for all the wrong reasons.

After getting a few hours of fitful rest, I opened up Xcode on Saturday morning and started looking for the problem. It took just a few minutes to find the bug and another few minutes to fix it. Our concern at this point was getting the update to users.

Apple saves the day

In spite of it being the middle of weekend after a busy week at WWDC, we were able to get in touch with Apple Developer Relations. Our contact was able to expedite the approval of the application.

To say that this was a relief would be the understatement of the century. The update for the free version started showing up in iTunes on Sunday evening. The paid version was updated on Tuesday morning.

What can we learn from this?

Developers are human. We make mistakes. It’s interesting to note that Loren Brichter and Buzz Anderson, fellow developers whose skills I hold in high regard, were also affected by the Twitpocalypse. It really does happen to the best of us.

It’s also interesting to see that Loren and Buzz reacted in the same way I did: by fixing the problem ASAP. In Loren’s case, that meant finding a hotel with WiFi in order to distribute his update. Buzz released a new beta in a matter of hours.

In my experience, these brain farts are problems with easy fixes. It’s something like checking for invalid bounds, getting a Boolean state wrong, or something else of that nature. It’s not a complex problem: it’s an oversight.

The problem, therefore, is not how fast we can react to fixing critical bugs, but how fast the App Store reviewers can react with an approval.

As a device that’s constantly connected to the Internet, the iPhone taps into a stream of data that is unpredictable. Data in “the cloud” can change at any time, and web applications naturally adapt to these changes with a quick deployment strategies. Those of us who are building client applications on top of this network infrastructure need the ability to adapt quickly, too.

Security issues are another area where a quick response is a requirement, not a luxury. If I discover something that puts a user’s private data at risk, it’s my responsibility to fix the problem as quickly as possible. Time spent in a review queue is time spent being exposed to a flaw.

At the same time, we shouldn’t blame Apple for these delays in reviewing applications. In the year that they have been selling our products, the App Store has been more successful than anyone imagined. It’s clear to me that reviewers and others involved in the approval process are overwhelmed by this success.

How can we fix it?

Fortunately, I think there’s a simple way to solve this problem for all developers selling products on the App Store. The inspiration for this solution will be obvious to anyone who’s used Apple’s Developer Technical Support (DTS.)

When you purchase an ADC membership, you are given a number of “incidents”. These DTS incidents can be used when you have a problem that can’t be solved through documentation, support forums or hours and hours of debugging. It’s for the hard stuff, and usually involves getting an engineer at Apple involved to understand and fix the issue.

As a developer, I’m very careful to use these incidents wisely: they are a last resort. There are some years where I don’t use them at all, those are the good years.

A similar system could be put in place for critical bug fixes on the App Store. If every developer was given one or two “prioritized reviews,” it would act as insurance for the brain farts. You’d have a way to raise a flag and say “I need special attention for a critical bug.”

If another developer has a critical bug, I have no problem with my review process for a feature release taking a little longer. And since prioritized reviews would be a scarce resource, they won’t be open for abuse because developers will think twice before using them.

Because it’s not a matter of if you have a brain fart that leads to a critical bug, it’s a matter of when.

Written on June 15th, 2009 for Development, Observation, Opinion

A phone by any other name would smell as sweet…

The general consensus is that there will be a new iPhone announced next week. I, like others, think it’s going to have new features and capabilities. But how is Apple going to label this new device?

iPhone 3G?

It’s entirely possible that Apple will keep the same name as the previous version. There’s precedence in the Mac product line: we’ve had MacBook Pros for several years now with various ways of distinguishing different product iterations (“Late 2008”, “Unibody”, etc.)

Given Apple’s reluctance to publish any of the device’s hardware capabilities, this would seem to make sense.

Except for one small problem. It’s likely that the new iPhone will have a faster processor and more memory. Some applications will be written to take specific advantage of these improvements. And that, of course, means that you need a way to let iTunes customers know if those applications are compatible with their device.

I don’t see Apple putting “iPhone 3G (2008)” and “iPhone 3G (2009)” anywhere in the iTunes UI. It’s just too confusing.

iPhone 4G?

What about having artificial product designations like they did with PowerPC desktop Macs: “G3”, “G4” and “G5.”

The problem here is that 3G refers to a network service, not a product generation. If this naming convention is used, customers are bound to wonder if a “4G” product works on their 3G network.

iPhone 3G Plus?

So maybe they do something like we’ve seen with the iPod product line. Using terms like “Classic”, “mini”, “nano” and “shuffle” to differentiate the products.

The problem here, of course, is that there’s only one product on sale. Unless Apple plans to keep the current product on sale (at a reduced price,) this just doesn’t make sense.

I’m not going to make any predictions on the device’s name, but I will be paying close attention. The ultimate choice is likely to offer some subtle clues regarding Apple’s plans to evolve this product in future revisions.

Written on June 4th, 2009 for Miscellaneous, Observation
Older