Lessons to be learned from #amazonfail

(Not all directed at the same set of people, and not all necessarily 100% relevant to what /actually/ happened)

Likely causes of Bad Things, in probable order:

- mistake
- accident
- stupidity
- odd concatenation of unrelated circumstances (really, you would be surprised how often I run into 'this problem wouldn't have killed the servers, except for this other problem we weren't previously aware of')
- bad design/incompetence/paranoia
- outside interference/gaming the system/variants thereon
and, right down the bottom
- deliberate and conscious malicious action by the supposed guilty party
and so far off the bottom it's not even funny
- conspiracy theory

I mean, jeez, people - get a sense of proportion. Any /sensible/ large company with half a clue about its public reputation isn't going to deliberately do things which alienate one of its most vocal user segments. And the blog/LJ/Twitter/Facebook-verse most definitely IS these days.
However, most vocal != largest source of income. And don't be surprised to discover that a very large percentage of Amazon users either don't know or care about #amazonfail.

What to do when you appear to have done a Bad Thing

- get on the case fast
- say you're on the case
- keep people informed
- hold your hand up and say you're sorry if it turns out to have been something you could have avoided
- fix it
- tell everyone you fixed it and what steps you put in place to make it not happen again

Databases 101

- update [table] [action] where [condition] is not necessarily a trivially reversible operation, depending on the precise nature of [action] and [condition]. Anyone who tells you otherwise is being desperately naive.
- just because your model of the problem is one field in one table, don't assume that's how it's implemented elsewhere.

Also

- out-of-band mechanisms you put in place to avoid major embarassments (I'm thinking gaming the 'People who liked this, also liked...' links with 'adult' items, for example), need to ring major alarm bells every time they fire.
- there is no substitute for human decision making, yet
- humans occasionally screw up.
- /most/ humans don't do so maliciously
- the Internet is, however, populated with a subset of malicious little fuckwits who should have their typing hands removed.

We have a new 3 piece suite...

...well, for values of 'new' that are 'used to belong to Anne's sister who hated it after a very short time'.

( It passes the cat test. )