Jul 02

Post mortem on a catastrophic data loss

Nerd No Comments »

I recently made a series of dumb dumb dumb mistakes that culminated in the loss of about a week’s worth of work. In order to extract some positiveness from this incident I figured it would be good to do a post-mortem on exactly where I f’d up and what I learned so that I might perhaps save others from making the same mistakes. BTW this is only going to be mildly amusing/useful if you’re a geek – if you’re a layperson stop reading now b/c your eyes will glaze over. One thing I will say is having restored things at this point I have profound empathy for the couch surfer guy’s catastrophe and what he went through (I only lost a week of work – he lost 3 yrs with his “perfect storm”). Here’s what happened:

The context

So I’m admittedly terrible when it comes to attention to detail. The fact I somehow at one point programmed Cold Fusion web applications and commerce systems from scratch that handled hundreds of thousands of dollars of people’s money still boggles my mind. The fact is I know just enough tech to be dangerous and I try to leave the hardcore IT functions to others. In this particular situation though I was essentially working rogue to get a microsite up to test a new commercialization opp for Scratch Audio around the idea of facilitating online remix competitions. Using the free microinstance tier of hosting on Amazon EC2 and the WordPress JumpBox I figured I could implement a site in a weekend, throw some quick traffic at it and determine fairly quickly if there was resonance around this idea.

I started on a Friday and did a marathon session of pulling together all the marketing working on a local VM of the WordPress JumpBox. Given that I was working out of a cabin in the woods over a crumby connection, the aspect of being able to develop against a local server was really handy. By Sunday evening I had a microsite done in WordPress about 85% of what it needed to be. I used the JumpBox backup procedure to extract the state to a local file on my desktop, shut down the VM, made a timemachine backup of my laptop and did the 2hr drive back down to Phoenix feeling pretty good about things.

The next morning I woke up in Phoenix planning to use the JumpBox migration procedure to move my dev instance of the site to a live hosted scenario using the Amazon Free offering. I lit up a new instance on EC2 in minutes using the JumpBox launch widget (spiffy!), imported the backup file and checked the site. The page content was there but all theming was lost. Here’s where I made my first error…

The failure sequence

Now I should have known this having used JumpBoxes for the past five years but the backup procedure explicitly excludes certain directories by default (and this is a sensible way for it to work). About half the work I had done in that marathon session was in making changes to the default theme. Had I installed a new theme and worked there, no problem… but alas I made all changes on the default theme which was excluded from backup. “No biggie, I’ll just grab the theme directory out of the local VM and use that.” Here comes mistake #2…

That morning I realized I had mislabeled the directory that the VM lived in with the name “June 2010 site changes” (yea i’m frequently about a year behind). I fixed the date name on the directory thinking nothing of it earlier that morning. When I went to fire up the VM to grab the theme directory it was VMware armageddon. The first message was a helpful “a needed file cannot be found” warning. “Oh, must be that I renamed the directory. I’ll just rename it back.” Enter a barrage of new errors informing me that various i’s were not dotted and t’s were not crossed. I spent the next 2hrs learning the intricacies of the .vmx file, changing various settings, sacrificing a chicken, throwing some salt over my shoulder and finally was able to recover the VM (note: the sage advice from @godber – make a backup of the entire VM dir before you do anything).

Anyways, with VM restored I was able to manually grab the excluded theme directory via SFTP and push that into the Amazon EC2 instance. Worked like a charm and the new site was live!

Aaaaand… here’s where I made mistake #3.

I configured S3 backups and breathed an unknowingly false sigh of relief thinking “everything is on Amazon now and backups are in place. Nothing can go wrong.” Of course, my backups were still inflicted with the exact same problem that had forced me to retrieve the theme dir from the VM in the first place. (yeah this is why I have no business in IT folks ;-)

I trashed the VM on my laptop in order to save 6GB of disk space (I figured worst case I still had the timemachine backup at the cabin). With site working and eager to get some immediate feedback I implemented an Adwords campaign. Over the course of that week I iterated the marketing, implemented various tracking scripts like Chartbeat, Crazy Egg, Analytics, Optimizely and Adwords conversion tracking. On Friday evening I implemented a Stumble Upon campaign thinking “okay let’s get a broad swathe of musicians looking at it and see if anything shakes out.” Closed the laptop lid, went to happy hour… bad idea. Turns out microinstances fall down under load of 9 concurrent users on WordPress (and that’s even with Hypercache running). I get a Chartbeat page about an hour later that the site had gone unavailable. No biggie, I pause the SU campaign from my phone, pause the adwords campaign and figure “I’ll just restart it in the morning and run it under a larger instance size.”

I wake up the next morning, open my AWS console and am greeted with the cheery message “you have no instances running.” “Umm yea but what about the instance I was running last night that’s now unreachable?” Nothing. Worst case at this point I thought I had the S3 automated backup from the night before so I had only lost a day’s worth of modifications. Wrong.

Upon inspection of the S3 backup I realize my automated daily backups suffered the same (obvious) problem as the one I used to restore from and validate the nickname I earned in 1st grade: “absent minded professor.” < begin head slapping > Okay okay, worst case now I’ve lost changes back to Monday but now I need to drive up to the cabin and pray that the backup file I had in the cloud on S3 would restore successfully into the VM that would hopefully restore successfully from Timemachine backup I had on the firewire drive at the cabin (it was starting to feel like my data existed at the 4th level of Inception).

What was really puzzling though was how getting a slug of traffic to a microinstance could completely wipe it off the map? I would think it would hang it but not obliterate it and outright eradicate the EBS volume with the data. Completely baffled by this and with all hope lost on retrieving the EC2 microinstance at this point I happened to check the JumpBox GUI to see if I could access it from there. Miraculously it still showed it active although clicking the “Access” button just left the browser hanging. This didn’t jibe with what my AWS console was telling me but at this point I shrugged and used the JumpBox GUI to terminate the instance. Mistake #4…

Turns out the instance was still there- it was just in the west region and the AWS console defaults you to the East region. So the data on the EBS volume was still there and retrievable right up until the nanosecond I clicked that terminate button… < commence Seppuku >

Given that the majority of the work that week had been in refining page content (which was protected by the S3 backups since it was stored in the database on the WP JumpBox) it wasn’t actually all that bad. I ended up driving back to the cabin, restoring the VM from Timemachine (which worked flawlessly), importing the latest S3 JumpBox backup into the local VM and using the WP Import/Export function plus some manual finagling to move the site. Having remembered most of the changes I had made that week it was a matter of reimplementing those and re-adding the various tracking scripts that were missing. In all, about 5hrs worth of duplication of effort to recreate everything under its new home.

I’m happy to report that remix.scratchaudio.com is now live on a server that can survive substantial traffic and we just had our first band signup yesterday.

What went right

For all that went wrong in this series of idiotic blunders on my part here are some things that went right:

  • Timemachine appears to be effing bulletproof
  • The JumpBox backups work flawlessly but with the caveat that you understand exactly what they’re backing up.

What I learned

  1. Test your backup procedures with an actual fire drill where you have to use them to restore your data. You are almost invariably guaranteed to learn something valuable from this exercise (even if it’s just the peace of mind of having done it – like changing a tire before you actually have a flat).
  2. I have no business running servers ;-)
  3. This is why services like Page.ly exist
  4. Don’t delete stuff until you absolutely have to. I had 150GB of free space on my laptop and yet I felt like I needed to get rid of this 6GB VM once I was finished with it. Dumb. Keep until you need to throw it away. There’s utilities like Disk Inventory X that make it easy to clean out the cruft eventually.
  5. Microinstances are handy, light-weight, disposable tools for dev/test but should never be used in production. They cannot handle any kind of load. Kimbro had actually told me this but it took experiencing it first-hand for it to sink in.
  6. EC2 instances never just disappear, they’re still there even when they become unreachable via the web. When something seems fishy, stop and seek alternate explanations and get a second set of eyes on it rather than trouncing forward and making the situation worse.
  7. VMware VM’s are surprisingly brittle – simply renaming the parent directory in which they reside unleashes a chain of events that makes it unusable. I’m shocked given that product’s level of maturity that they’re not more bulletproof. The good news is your data is still probably retrievable when things get moved around but you will spend the next two hrs wading through config files to manually futz with parameters in order to get it working again.

Anyways, hopefully this writeup is useful and saves even one person from making some of the errors I did in this debacle.

Jun 23

My problem with The Black Swan

Reviews 4 Comments »

A couple people have recommended I read the book The Black Swan recently. I’m only 40pgs in and already I have a serious issue with it. The picture below is a good summary of my gripe:

It’s essentially like that highway roadsign that warns you of falling rock – it’s useless advice. Since when has that sign ever altered your driving behavior? In fact it’s worse than useless, it’s presence is detrimental because it generates unnecessary worry and distraction without giving you any actionable info to be able to do something about it. It merely broadcasts, ”
Yeah, so rocks might fall on you. Sucks.”

Similarly The Black Swan appears to offer this as its core message:

Despite our best efforts to make sense of situations throughout history, inevitably a massive, random event at some point manifests and its unpredictable effects trump everything we knew previously.

Basically “rocks might someday fall on you. sucks” – a completely worthless and defeatist message. And yet somehow this book has won a bunch of praise as being insightful. I’m not sure if the author is advocating that we stop applying science to attempt to understand current situations but that is a message that one could infer.

I’ve admittedly only read a sliver of this book thus far so maybe the author eventually gets around to offering some type of prescriptive advice. But at this point it appears to be a pop psychology wankfest (and a verbose one at that). At least books like “Tipping Point” and “Blink” had concise writing and referenced interesting psychology experiments to yield conversation fodder. This one appears to be entirely devoid of both. Somebody who’s read it and found it valuable – what did you take from it that was useful and how has it changed your behavior and how you think?

Tagged with:
Jun 09

Thoughts on Microconf

Entrepreneur, Events 2 Comments »

I just got back from Microconf in Vegas and wanted to do a brain dump of some thoughts while they’e still fresh. This was an outstanding event in every way – killer speakers, high quality of attendee & flawless execution. I highly recommend (provided Rob and Mike decide to do this again next year) that any single founder or bootstrapped company attend.

Takeaways

This was a 2-day deal with some top-notch speakers. As is always the case with these types of events though the real value is in the hallway and meal conversations. I was fortunate to meet some super duper cool cats & dogs. As far as nuggets of actionable advice, I noted a couple from the various talks:

  • Ramit showed the scientific approach he takes to understanding his customers’ problems and objections via increasingly specific surveys and systematic testing on the site. His testimonials are all surgically placed to address the objections he uncovers via testing. This wasn’t earth shattering but he hammered home the value of a methodical approach to unearthing your customers’ problems and thinking in those terms vs. selling the features of your product. I’m looking forward to implementing some of his ideas on how to conduct killer surveys.
  • I finally got to meet Sean Ellis whom I’ve followed for about the last year and a half. I love his philosophy of mandating that you achieve a certain level of measurable product market fit before ramping marketing. Think of it almost like a type of “escape velocity” in that you don’t leave orbit and apply the rocket fuel until you’ve achieved this “must have” level of affinity from at least 40% of your user base. He dropped a nugget in passing that I thought was very insightful. He said “You can increase the gratification of your users without even changing the existing product simply by identifying what they perceive to be the core value and stripping all messaging down to that essence.”
  • Hiten is my hero and is the lyrical gangsta of funnel analysis and conversion optimization. He dropped some pure gold with his presentation on the various lessons he’s had in building Crazy Egg, Kiss Insights and Kiss Metrics. For people already immersed in the Customer Development movement it wasn’t anything new but it was a great orientation for the folks who weren’t familiar with that framework. He pulled together a neat bundle of resources which I plan to go through soon. He also inspired me to re-implement Kiss Metrics and get a firm grasp on exactly where we’re losing people in the funnel. If you’re not following him and Kiss Metrics on Twitter you’re doing yourself a disservice as it’s the best curated fountain of useful techniques for young startups. One audience member during Hiten’s talk shared what I thought was an ingenious cheap/elegant hack for getting early CustDev feedback on an app: post an ad in the jobs section of Craig’s List for the industry role you’re targeting and solicit input either via a survey or a physical focus group.
  • Noah blew our mind with hot sauce. If you weren’t there we’ll just have to leave it at that. But there was literally Sriracha flying.
  • The website teardowns were one of my favorite parts. They picked apart the sites volunteered by attendees and walked through what could be improved. It was hugely interesting to hear their take on the flaws and the rationale for how/why/what to change.
  • The Pluggio guy, BuySellAds guy, Rob Walling and Mike Taber all gave great presentations worth noting.

In all it was a superb event that I would highly recommend to anyone contemplating attending. The one format change I would propose is to break up the lineup of back-to-back speakers all day by interspersing some type of interactive exercise. The speakers are just the excuse to get the right people in the room but the truly valuable part is the interaction with other attendees. It would be neat to see them sub out one of the speaking slots with a problem solving exercise whereby people break into groups and work to cobble together a solution to a specific business challenge and then have an ambassador from each report back to the group at large. Anything you can do to increase the surface area for having conversations amongst attendees goes a long ways towards making the conference even more useful.

Lastly, I just wanted to share a moment I had after making the drive back from Vegas last night. I’m now living up in northern Arizona in a cabin for the summer (a whole ‘nother blog post). Anyways I went for a barefoot run on the golf course listening to this guy’s playlist and this beautiful song came on right as the sun was setting and I was running this path through an outcropping of trees. This experience happened one other time but it was an absolute wave of pure gratitude that washed over me and every cell in my body simultaneously acknowledged how lucky I am to meet all these incredible people who are laboring to change the world in their own small way. For all the doubts that swirl around when building a startup in an unproven market and an uncertain economy it’s moments like these that confirm we’re running the right path.

Huge props to and Mike Taber for toiling endlessly to pull this event together. I’ll be there next time no question.

Jun 04

What makes a great IDE?

Kudos, Ponderings No Comments »

I had a bit of an “ah-hah” moment this past Thursday evening at the Phoenix Ableton Usergroup (a free monthly group I organize for musicians who use this piece of software). Here’s the gist of what changed in my thinking: prior to Thursday if you would have asked me “what makes a great integrated development environment?” I would have said:

The toolset allows you to manifest what’s in your head with the least friction and most fidelity.

Post Thursday night, here’s my new opinion:

A tool whose design is so inevitable, irreverent, spartan or unique that the tool itself inspires creative ideas that weren’t there before its use.

Pat Metheny Pikasso 42-string guitarA friend of mine Brandon conducted an experiment called awhile back called “month of music” where he forced himself to author a new piece of original music every day for a month using new and different instruments. He found that the introduction of new and unfamiliar tools generated musical ideas which hadn’t existed before.

Ableton is a piece of music recording software that has (for the past four months since I began using it) continued to blow my mind at each turn with untold possibilities. It’s admittedly one of the most daunting interfaces I’ve ever had to learn and reminiscent of 3D Studio Max in its complexity. But it’s one of those rare software programs that comes along, shatters the traditional paradigm and opens up a world of possibility that sends your mind reeling with ideas. For me it’s been a “Don’t know what you don’t know” advancement – the equivalent of taking off a pair of dirty sunglasses you didn’t even realize you were wearing.

If you’re a musician and have plateaued with your musical inspiration via your current multi-track DAW recording software, check out Ableton Live. The learning curve is going to feel like standing at the base of El Capitan looking up so you’ll ideally want a trainer, a user group or a bunch of time to dedicate to watching the various Youtube video tutorials out there. But once you can get over the “suck threshold” with it, it’s amazing.

I’ll write more on Ableton specifically as I become more advanced with it but think about whatever tools you’re using now and how you might shake loose some new creative inspiration by going out of your way to do things using a decidedly unfamiliar instrument.

Tagged with:
May 24

Chase bank is the Hotel California of banks

Activism 3 Comments »

“You can check out anytime you like but you can never leave…”

So I get this 2pg privacy statement in the mail from Chase telling me what rights I have to limit their ability to share my info. It basically says they have the right to share my info with other institutions for joint marketing initiatives and I have no ability to opt-out. That would be fine (annoying but acceptable) if I were still a customer of theirs but it turns out they retain the right to sell and share your info indefinitely even after you leave.

I called them (here’s the audio) and politely asked that they delete my info given that I’m no longer a customer and that they already lost my data once. They said “sorry we get to keep your info forever.” This too would be understandable if it were purely for the archival purpose of preserving a copy of my records for historical reasons but they actually said I have no ability to stop them from sharing my info for marketing purposes even after we’ve discontinued our relationship. They retain the right to continue to spam you and sell your info even after you’ve checked out. Pink champagne on ice indeed…

If this were some local business the attorney general would have these guys’ heads but alas it’s a bank and apparently nowadays banks appreciate sovereign immunity from all policy that is sensible. The one consolation here is that I recently completed a successful shortsale of my house (hey Chase how does that $146k haircut feel?). In layman’s terms this is a way for the average consumer to clear his/her balance sheet and shed a distressed asset similarly to how the banks have done. IMHO this process was a convoluted and opaque obstacle course by design but I plan to document the chronology of how it unfolded for anyone else who is in a similar boat and seeking to take advantage of this mechanism. F U Chase.

UPDATE 8/28/11: Introducing our Site. Double FU Chase.

Listen to the audio below for the ridiculous conversation with their CSR and pass this along as an advisory to any friends who are considering banking with Chase. Make sure your friends know the expectation of how they’ll be treated before signing up with these guys. And make sure they’re aware that checking in means they can never check out.

Tagged with: • •
Apr 09

The unattainable is unknown at Zaarly.com!

Kudos, Nerd 5 Comments »

There’s a little startup I have a special affinity for having been in the room for its inception at the LA Startup Weekend event, standing feet away when now investor Ashton Kutcher was first pitched. I had a chance run-in with the guys weeks later in a cigar smoking circle outside their RV at SXSW right after they landed $1M in seed funding. And I then got private beta access to their app at SXSW to try and solve a last-minute challenge we had in running our launch party for Scratch Audio.

The best way to understand what Zaarly does is to go to Zombocom and substitute “Zaarlycom!” They’re basically a hyper-local, real-time want ads site with no limitation on what you can ask for. Here’s why this is so cool:

There is no limit on the potential market cap for this company. Like eBay and Craig’s List, Zaarly stands to create an enormous ecosystem that will enable countless people to earn a living (and a few smart ones to become extraordinarily wealthy) by finding ways to expand the market and make it more efficient.

eBay and Paypal are so entrenched because they became core pillars in enabling others to make money. Zaarly will do the same only it will sit at meta level above each of these specialized devices essentially being this master clearinghouse of desire. In the process of making a ton of people rich, they are also going to provide a powerful secret weapon for those who understand how to wield it. Basically it’s like having your own personal “genie in a bottle” that you can pay to solve any issue for a price you name. Provided you know the value of solving the issue, you can at all times troll for a solution at a pricepoint which is profitable to you.

  • Need to get 300 mimes in El Paso by Friday? There’s a price for that.
  • Need to create and distribute 100,000 wristbands with your company logo to every fraternity member in the US? There’s a price for that.
  • Need midget geologists to sneak into a cave in mexico and perform ground-penetrating radar analysis, have it interpreted and translated into Swahili and display the results in skywriting over Chicago? There’s a… you get the point.

Sharp college kids are going to find ways to make money on the spread between asks on Zaarly and the infinite fountain of ways they can fulfill the requests. CS students will invent scrapers that check eBay, Oodle, Craigslist, etc in realtime for items sought by Zaarly’rs and they’ll broker the transactions and capitalize on the arbitrage. Entire businesses that were heretofore not feasible will be enabled by the existence of this tool. Zaarly is one of those simple business ideas like Groupon that people a few years from now in hindsight are going to kick themselves for not having thought of.

One of their other investors Naval Ravikant is a guy whose writing and speaking I’ve followed for awhile now. Knowing how he thinks and his affinity for econ, game theory and market efficiencies, he has to be giddy over the possibilities on this one. I was admittedly lukewarm on this at Startup Weekend but now that I’ve had the chance to ruminate on the implications for all the side businesses this is going to spawn, it’s genius. Every pawn shop, scrappy college kid and CL/eBay lifer is going to be all over this app. I can’t wait to see how things unfold.

The icing that actually tipped me to write this post was hustle they’ve shown in organically generating buzz the way Noah Kagan would advocate. A guy on their team (having no idea that I knew them already) reached out to me having found me via a tech-focused meetup I run in Phoenix. This is how you create a “root system” of buzz: focus on the pressure points and have one conversation at a time.

The UI for their app is gorgeous. The business potential is limited by imagination only. The team is legit guys. Their marketing plan is enlightened. And if/when this works its success will be a shining trophy and an amazing advert for the Startup Weekend event itself. I am long on both Zaarly and Startup Weekend.

Sign up for their beta – I understand they’re targeting a launch for early-mid May. This is definitely going to be something to investigate when it comes out.
Full Disclosure: I have zero financial ties w/ these guys. I just think they rule.

Previous Entries Next Entries
preload preload preload