Why Was PerfectIt Cloud Down for So Long?
By Ed Fisher
While not everyone may have been aware of, or affected by it, we recently had an outage of PerfectIt Cloud. Now, outages are the worst. You already know that. They mean you can’t access the service you paid for. And they often hit right before a deadline. However, what you might not realize is that they’re awful for my team too.
In Customer Support, we thrive on being able to fix problems. When a cloud service goes down, we’re completely unable to help. For our engineering team, it’s just as bad. They’re under pressure to find a root cause of a problem, and work on a fix, while knowing customers are unable to use the product they built.
All of that is to say how sorry we are that this outage occurred. The misery that you’re dealing with compounds with what we’re dealing with. I mean it when I say it: outages really are the worst, and we’re genuinely sorry about this one. I want to explain the details of what happened, and why it happened too, so we can be transparent about how this all came about.
“I mean it when I say it: outages really are the worst, and we’re genuinely sorry about this one.”
Ed Fisher, Customer Success Manager at Intelligent Editing
What Actually Happened?
Both PerfectIt Cloud and our website were using an out-of-date component. We were working on upgrading those components, but it was proving to be more complicated than expected. On 31 August, Microsoft withdrew the service. We hadn’t seen the announcements about its withdrawal. So, everything went down.
With lots of outages, you can restart a server to get things going again. That solution is quick (it’s a lot like turning your computer off and then on again). However, when a foundational service disappears, you can’t do that.
Our engineering team had to build the resources to work with the replacement component. It’s new development and it usually doesn’t happen overnight! However, the team worked well into the night. By early the next morning, the logins on our website were working again. A day later, they were working on PerfectIt Cloud.
But something still wasn’t right. PerfectIt Cloud allowed logins, but was running at a snail’s pace, and frequently timing out completely. Although it was possible to use, we decided it wasn’t good enough yet so did not report any change in status.
The next four days were full of bugfixes and testing to try to determine why it was operating like that. On 7 September, a full week after the outage was reported, we identified the cause and released a fix. At that point, we let everyone know that it was back.
Why Are We Here with PerfectIt Cloud?
We know that PerfectIt Cloud has had issues for years. From slowness to time-outs, as well as the difference in features compared to PerfectIt for Word (Windows), my team is familiar with all of it. But the history begins before I was at the company. In 2018, Intelligent Editing said we were aiming for parity within six months. In 2021, the company provided a full explanation and workarounds. In 2022, we created a Cloud Only package so that Mac users pay less than everyone else. But an outage like this, and the intense frustration it has caused everyone, has prompted us to do a lot of thinking beyond even those.
In retrospect, it’s easy to see why PerfectIt Cloud is harder to remedy than any of us would like. It suffers from early adopter path dependency. The Intelligent Editing team started building PerfectIt Cloud in 2016 as soon as Microsoft announced improvements to the Javascript API for Office. We were responding to hundreds of requests that users had made for a Mac version of PerfectIt. We leapt at the chance to build that. The choices we made were good ones for 2016. But seven years is an eternity in tech!
The risk of being early to market with a new product in a new space is locking into an architecture or system that subsequently doesn't move forward as you expect. This is exactly what has happened with PerfectIt Cloud. It is built on older technology, and its legacy nature makes it hard to update and improve upon.
A physical example of a similar issue is the London Underground. The Underground was built in the Victorian era originally to certain specifications and was a marvel when it was the world’s first underground railway. Now in the modern era, new Underground trains need to be made to the specification of the small, cramped tunnels originally dug, even though other standards have improved and changed over time.
Here’s what path dependency looks like in software – certain updates and re-writes of PerfectIt Cloud “as is” end up being development of legacy technology, which in and of itself could cause further issues down the line. The real solution will be moving PerfectIt Cloud onto newer technology that is now embraced and established by Microsoft. We don’t want to keep buying Betamax, we want to switch to VHS.
But realistically, that means if we’re going to solve the issues in PerfectIt Cloud permanently, it will need significant work. We’re a small company, and it will take time to make such comprehensive changes.
So, What Are You Doing Now?
First off, I want to thank you. The response to this outage from our users has been one of the most generous and kind-hearted responses that a customer support team could imagine. I’ve honestly never seen anything like it.
When PerfectIt Cloud was down, you told us how much you felt for us. Seriously! This is what Twitter looked like, but we had similar comments on every channel:
So, thank you! Your generosity and empathy in a situation like this was extraordinary.
Still, our immediate response is to find a way to compensate users for the time lost. That’s not part of our service agreement. There’s no legal right to compensation. However, especially with all your kindness, we want to make sure we do the right thing.
We looked at compensating users individually. However, that makes no sense. PerfectIt is such a low-cost product that even with a seven-day outage, the compensation would be just a little over $1. We’d spend more in fees and staff time arranging transfers then it would possibly be worth to get $1 back. And besides, we know that the cost to you in editing time is far higher than that anyway. We also looked at offering free time. However, the nature of subscriptions makes that extraordinarily difficult.
So instead, we’re going to pool all of the compensation together and donate it to the UNHCR’s appeal for Ukraine. With no end to the awful war in sight, UNHCR is on the ground in Ukraine and neighbouring countries, providing lifesaving protection and support to ordinary people fleeing the crisis. That means Intelligent Editing will receive no money from PerfectIt Cloud for the time that it wasn’t functional. We hope knowing that the money has gone to an important cause is helpful.
We also know we need to improve our communication. Lots of you were (rightly) frustrated that we didn’t communicate the outage clearly on enough channels. We’re putting in place procedures to make sure we do better in the event of any future problems.
What’s the Long-Term Plan?
In retrospect, Intelligent Editing built PerfectIt Cloud for the Microsoft Store before the technology had sufficiently matured. However, since then, that tech has improved. It is now possible, one day, to achieve full feature parity. In fact, beyond feature parity, it’s even possible to imagine a world where the Microsoft Store version becomes the main version of PerfectIt.
That moment is a long way off. Work of this magnitude is a process of years rather than months. So it’s important to say that we understand if you decide this outage is enough and you’re not going to renew again. We still hope you’ll stay on the mailing list and revisit PerfectIt Cloud in future because there will be a time when we solve this.
In the meantime, we will keep offering the Cloud Only option of PerfectIt as a lower-cost alternative that reflects the lesser offering. We remain committed to the platform. In time, we will improve the product. Until we do, Mac users will continue to pay less than Windows users.