Feb 02

I’m seeking to improve my chess game. I’ve played casually for years now and recently have been more deliberate doing 1-2 games per day against the Squareoff AI. This is a physical chess board which communicates via Bluetooth with an app on your phone and allows you to play other people online or play the computer. What’s nice is it generates the chess notation file at the end of the game which you can then upload to Lichess.org for Stockfish to analyze your mistakes and to run whatif scenarios of how the game might have gone otherwise with different decisions. This coupled with the daily chess quizzes on Lichess and occasional study of the theory via their large lesson library is the best way I’ve found so far to improve.

What I ideally want though is to find someone who I mesh with who can weekly review my games and then do a weekly 30min zoom with me to walk through the games using those as an opportunity to correct my mistakes and teach strategy & tactics. It’s nice to have the computer tell you where errors occurred but it would be far more helpful to have someone talk me through the “why” behind the mistake and what I should have been thinking in that situation.

I’ve posted below my last few games from Evernote. If you know anyone who is both adept at chess but more importantly, a good teacher who can analyze where I’m at and help accelerate my progress, please send him/her my way. I would welcome any intros and am happy to pay their hourly rate to test them out and convert to a monthly retainer assuming it’s a good fit. Thanks in advance.

Game Date
Lichess URL
Post-play Summary
White. L. Was down significant material early on but came back at the end. Similar to last game, very interested if I had a chance to win and botched it with the end game. Ultimately tried to bring my Queen to the back line and pile drive with the rook but that didn’t work out.
White. L. Was down early but made a nice comeback attempt. Got into a near-checkmate and was able to check him then ran his King all the way around. Very curious about the close quarters end game on the right side of the board and if there was an opp at any point to pull it out or if I was destined for L at that point.
White. L. I made mistake early on when I let him get bishop buried deep on my side on same line as Queen and then I brought my King forward instead of taking with B2 pawn.
White. W. He blundered early and lost Queen then I lost mine. I was up a few pieces all game and blew a couple checkmate opps. Eventually promoted a pawn and trapped him with Queen/Rook
White. W. lost the Queen early in a dumb move then tried to trap his Queen while he disemboweled me. Kept it fairly close all things considered until end where I made a dumb move and guaranteed checkmate by trapping my king close to his side.

And yes, my interest in chess far pre-dated Queen’s Gambit. But that Netflix series is, I believe, doing for chess what Karate Kid did for martial arts and what Hunger Games did for archery in terms of getting youth excited about it. Anything that steals back young minds from Tiktok is a win iMO. cheers

Jul 08

Three months ago became the newest customer of Infusionsoft. I’ve learned a ton about this product during this time and wanted to offer a brain dump of thoughts. These are directed both to potential users of the software as well as Infusionsoft the company.

What it is: CRM + automation fabric

Describing this product concisely is difficult because it’s more of a “fabric” than an application geared towards a specific use case. As far as I know it’s the first system to bring the capabilities of something like Eloqua and Marketo within reach of the small business, only better because it combines CRM. Think of it like a pliable Silly Putty for marketing & CRM automation that allows you to set your marketing material up like one of those “Choose your own Adventure” books from back in the day. It’s tempting to call this system “a nerve center of customer communications for a SMB” but that too would be pigeon-holing it to customer communications when it can be equally valuable for vendors, partners & resellers. I like the “moldable marketing fabric” analogy because it captures the right essence of the flexibility of it. My friend Eric who works there calls it “the OS of small business” and I think that’s an interesting and valid way to put it.

Another way to think of it is via the phrase coined by one of their star customers, Jermaine Griggs. He did a stellar webinar about a week ago that is a must-watch for people getting started with Infusionsoft. He cites the main benefit as enabling the “scaling of personal attention” – basically gaining the ability to render customer experiences you’d expect of a small mom & pop shop only on web scale supporting hundreds of thousands of customers with the same level of intimacy. This guy has a truly incredible story having come up from nothing – he’s worth listening to just for his humility alone but he also has a brilliant mindset of how he approaches marketing and relationship building online.

How this can benefit SMB’s

Again it’s tough to reduce the benefit to a sound byte here because it really depends on how one uses the fabric. What would you do if you could imbue your selling knowledge into your web app so it intelligently did things the way you would do if you were standing there in person talking with the prospective customers? How might that change both their experience as well as your insight into what they want/need? I find it find Dave McClure’s AARRR pirate metrics framework useful when examining online businesses – Infusionsoft can basically help in 4/5 phases (activation, retention, revenue and referral).

We “hired” Infusionsoft for two main reasons: 1) on the back end there’s simply no way for us as two people to stay on top of all the deals handled by our network. We had tried using Google Docs, Longjump, a custom WordPress app and Basecamp and while those each sufficed on the CRM front, none gave us a way to intervene and actively and intelligently babysit transactions. 2) On the front end our leadgen apparatus was very static. We had a single linear funnel for on-ramping new customers. While Mailchimp auto responders are better than nothing, you’re still highly constrained by how adaptive you can make your marketing and it’s not unified with CRM so there’s no straightforward way to pull up contacts and get a clear picture where everyone stands.

What it’s lacking

Infusionsoft has been great at addressing these issues so far but I feel like maybe I’m fawning over this product a bit much; it’s definitely not perfect. The difficulty of setup is its Achilles’ heel right now impeding mainstream usage. It took us roughly two months to implement and we are moderately-technical and highly motivated. While they’ve made strides with their new visual campaign builder, it’s still missing the mark in a couple fundamental ways. There are IMO two separate shortcomings with their current onboarding process:

  1. They ask customers to make the leap to wiring up a solution before mapping the current landscape. The campaign builder put some sanity to an extremely unintuitive implementation process that involved jumping amongst many screens to wire up templates, action sets, rules, web forms and follow-up sequences. But it doesn’t solve the intermediary step needed wherein the customer visually maps out their existing processes, identifies the opportunities to fix the funnel and thereby establishes the plan. For those with a background in software development, think of it like parachuting into a programming scenario and trying to leap to writing code to solve a problem before you’ve mapped out the current process and hashed out the user stories for what you’re trying to build.
  2. I think even if they were to add this intermediary step, it’s still going to a far stretch to ask the average SMB owner to complete this process on his/her own. There’s real artistry and skill involved in identifying the biggest opportunities and then taking the knowledge of the software and translating the business objectives into implementation. I believe their model needs to shift from a DIY solution for the end consumer to incorporate an agent as the intermediary. These “marketing architects” would have a thorough understanding of both online marketing best practices as well as detailed understanding of Infusionsoft and be bridge builders to close the current gap. In fairness, they do pair you up with a “success coach” employee who helps get you through the initial onboarding process (big shout out to ours, Brett was instrumental in our implementation). But IMO this is a band-aid fix to symptoms which would go away if the consultant ecosystem were made to be a more integral role.

It’s impressive how many people they have using the system in spite of these impediments. It’s a testament to how useful it is that folks want it badly enough to figure out how to bushwhack their way through it all to get it working.

What I would do if I were Infusionsoft

So I hate criticizing things and not offering solution suggestions. Here’s what I would be doing if I were Infusionsoft to remedy these issues:

  1. Create an “assembly exchange” They’ve already got what appears to be the equivalent of an AppExchange for vendors to market plugins to users but what I propose is actually making it so I as a user can turn my configured instance into a template I can sell other people. Think the way AWS allows EC2 users to convert a running instance into an AMI that can be made available to others – it opens up a whole new ecosystem. A bicycle store owner in Tampa who has developed a successful marketing framework in Infusonsoft should be able to productize his/her instance and sell it to bicycle store owners elsewhere. This would do some miraculous things, namely a) create a new revenue stream for that Tampa bicycle store owner without cannibalizing his/her local business b) help the new Infusionsoft user expedite her bicycle store marketing implementation and get a proven system unique to her vertical c) reduce customer attrition for Infusionsoft by reducing frustration and failed on boarding d) potentially spawn a whole new class of companies like how WPMU and Thesis grew around WordPress.
  2. Disentangle planning from implementation. Right now the “architecture” and “brick laying” are intermingled when setting up a new instance. It would be beneficial to have a separation that allows one to articulate the current business processes, then map out the ideal new flow after Infusionsoft. This document could then be handed off to a commodity implementation resource via oDesk or Elance and the tedious grunt work of implementing stuff could be offloaded. I haven’t formed a strong picture of how this looks but my hunch is they don’t need to reinvent the wheel here – this format already exists either as a specialized sequence diagram in UML or via BPM with an open source tool like Intalio. The sharp consultants are probably using a homegrown tool of their own to achieve this step – I would start by asking them how they’re doing it.
  3. Consultant ecosystem of brick layers v. architects I realize there’s already a consultant community for Infusionsoft but as best I can tell there’s no commonly accepted format for mapping out the vision with one “marketing architect” and then taking portable definition to a more commodity “infusionsoft implementor.” In my mind that first piece is what they should give away free: a way for any visitor to come and sketch out their funnel and current vs. proposed marketing interactions. Then simply charge for translating the proposed version into reality. Once the architecture can be partitioned from the implementation it opens it up to be able to commoditize the brick laying aspect and drives the architectural role to be more about formulating valuable constructs for the funnel rather than worrying about nuts & bolts of implementation.
  4. Sandboxed dev v. staging v. prod environments I would love to outsource some of the grunt work I’m doing now but I’m also very hesitant to give access to our system over to a stranger in the Philippines. The way we do this with our site now is by cloning our WordPress instance with BackupBuddy and spinning up a duplicate instance on EC2 and giving the contractor access to that system. They can break it all they want and there’s no damage. What’s better, I can spin up 5 instances and compare 5 different contractors apples-to-apples in parallel for two hours of trial work to see who gets the furthest and does the best work. It would be great if there were a way to clone our Infusionsoft environment and run sandboxed versions that could be promoted to staging and ultimately production. If the above separation of “architects vs. brick layers” is to be achieved this will be an important capability.
  5. An offline IDE This is perhaps too demanding to ask for this now but I will anyways- it’d be great to have a non-web-based way of building stuff. In the same way that Quickbooks online is cumbersome to accountants who are used to flying through screens on the desktop product, when you’re building a bunch of sequences and emails and whatnot, the latency of the web even on a fast connection is annoying. I totally get why it’s all web-based now (I would do the same) but perhaps there’s opportunity to do an Eclipse-based IDE that would generate a file which could be uploaded to Infusionsoft and provide an easier way for building a system?

Other random thoughts

So this is my random catch all for other things I wanted to mention that would be useful:

  • Bizspark equivalent the $2-3k initial startup cost is going to be preclusive for most startups even though they stand to benefit dramatically from this software. If Infusionsoft can stomach giving away some super-cheap accounts they should strive to offer a no-support, self-serve, feature-dilluted option for startups and take a longer-arc view harboring people who can’t afford it today but represent prospective future customers. Those folks are also typically more vocal startuppy types so there is likely some marketing benefit to serving them even though they’re free accounts today.
  • The gmail core plugin rocks This is missing functionality in the core Infusionsoft app but I’m happy to see that someone nailed the plugin that provides it. This plugin makes it so Infusionsoft will automatically monitor your Gmail and append communications that happen there to their corresponding contacts in IS. The benefit is multiple people can be working how they normally work and a centralized record of communications is retained in Infusionsoft for all to see. I would love to see the option in Gmail Automation Core to do attachments via filebox as well as the ability to do a one-time retroactive pull of past communications from a gmail account.
  • Global & shared dashboards There’s no way I can see for me to create a dashboard which gets shared amongst other members of my team. It’d be neat if they made it possible to have tabbed dashboard pages and possible for one to mark his/her page as “shared” so others could use it.
  • Past emails should be snapshotted by value rather than by reference There’s a glaring issue IMO right now with how past automated emails are preserved: currently it stores the variable names instead of the actual communication to the client. This needs to be a snapshot of what exactly was sent at that point in time. Think of changing prices, new sales reps, etc. There’s no good reason to store it by reference with variable names- it really needs to be snapshotted.
  • Tags and follow-ups listed in search results I would imagine they’ll add this soon but it’d be useful to be able to see sortable search results with the tags appearing in columns separated by tag category. Likewise it’d be nice to scan a bunch of results and see what follow-up sequences they’re in.
  • Calculated fields This is the one thing I really miss from Longjump: the ability to have fields on the contact record that are derived from other fields. Consider adding the ability to make a field that is simply a formula builder consisting of other fields on the record.
  • Lead origin tracking We’re likely going to do a bit of a tap dance on our site to get what I want in terms of tracking all the data on how customers originated (ie. which ad channel, campaign version, landing page, keywords, etc). This is fundamental and important enough that I believe this belongs in the core product. For now we’re planning to roll a cookie with all this info and then pass it via hidden fields on form submissions.
  • Snooze button on follow-up sequences It’d sure be nice to be able to click on the date of a follow-up sequence and just override it for that contact record. We end up doing a hoaky process now whereby we remove the sequence, create a scheduled task to remind ourselves to start that sequence over again at a later date. It’d be much preferable to be able to have a “snooze button” that would let you manually override the date of the follow-up sequence and push the whole thing out X days.
  • File-type form field to upload attachments The form builder tool is very good but it’s missing an obvious field type: there’s on ability to do file attachments. This one shouldn’t be terribly difficult to implement but there should be a new type of form field for attachments and they should be sent to the filebox for that contact.
  • No split testing I’ve seen the recommendations for how to implement split testing in Infusionsoft and unfortunately it’s pretty ghetto. I understand people are working on IS plugins that will lend this functionality but A/B testing is so integral to online marketing now that I would argue it belongs in the core product. If this is too peripheral to be in the core product today I would suggest making a one-click option for integrating something like Optimizely or Visual Website Optimizer. We’ve used Optimizely for awhile now and it’s excellent.
  • No funnel visualization There’s no way I’ve seen to visualize the various funnel interactions and run cohort analysis. Likewise this is pretty core to online marketing and if it’s not something they want to bake into the product itself I’d lobby for making it easy to enable KissMetrics or MixPanel by just entering in your acct ID for those products. I’ve used KissMetrics for about two years and it’s great for determining which section of your funnel demands attention.

Anyways, in summary, Infusionsoft is a really promising product and I’m excited to finally have our v.1 implementation in place. I’ll report back here on what I learn as we go. It’s great to see a local AZ company delivering a service that’s represents so much promise and has customers around the world. I look forward to seeing it evolve. If you’re interested in potentially using Infusionsoft for your business request a demo or check out some of their past client case studies.

Dec 22

I wanted to leave you 5yrs ago when you morphed your checkout process into a misleading gauntlet of extra screens and pre-filled menus that tried to trick the customer into purchasing unnecessary stuff.

I wanted to leave you 3yrs ago when Bob Parsons turned the entire web site into a soapbox for his chauvinistic machismo patriotism-for-profit blather and insulted the women of the tech community with sexist super bowl commercials.

I wanted to leave you 2yrs ago when I learned how you fail to protect your customers and cave to bullies who threaten DMCA suits in pursuit of stealing customers’ legitimate domains.

I wanted to leave you a year ago when you turned your technical support people in to annoying sales people that tried to sell me crap when I called because your service was broken.

I committed to leaving you when I learned that you use shady methods to swoop up customer domains and hold them hostage.

I began leaving you and moving my domains to another registrar one by one when Parsons pulled that stunt glamorizing his slaughter of a wild elephant in Zimbabwe as if it were some kind of heroic act.

I am now officially peaced out with this SOPA nonsense.

Your stance on SOPA is inexcusable political back scratching to curry favor and undoubtedly advances Parson’s financial interest in some sketchy under-the-table way. I don’t even know where to begin responding to the points made in your response. I am a native Phoenician and an active member of our startup community. The fact that you are regularly cited as a success on the basis of your revenues is an abomination as it neglects to account for the nasty underbelly of how your company operates. You guys are an utter embarrassment to our tech scene. I actively encourage all my colleagues to boycott your service and spread the word of how evil you truly are as a company. I have escalated the exodus of my few remaining domains on your system. good riddance.


For anyone else who is currently on Go Daddy, there are a ton of hosting providers offering coupons to switch off their service given their recent shenanigans. Check this Reddit thread. If you haven’t been following the SOPA developments, search Twitter & Google to get informed and involved to help fight this insanity they’re proposing.

Aug 31

I attended Jason Baer’s Facebook Marketing seminar at NACET last week. Usually these social media marketing seminars are fluff or so remedial they don’t yield any new insight. But Jason was extremely knowledgeable and I learned a couple interesting things from his talk and discovered a few sites and services that appear very useful. In all it made me realize we’re under-utilizing Facebook for our startups and need to implement a FB strategy going forward. Below are my notes and links to the relevant sources:

Five Different strategies to take with FB

Awareness - new Facebook page that actually collects emails - "win the news feed" x-factors - high target interaction weight - edgerank formula seems similar to HN "gravity" algorithm - Pagelever for better FB metrics (Edgerank checker = low end) Increase Sales - ShopTab.net from Phx do this well (like CartFly?) - Direct response = higher end Market research & insight generation - Liking page = expression of support (equiv digital bumper sticking). - intersperse q's to keep interactive and prevent feed fatigue ** play with polls more to activate people - Bulbstorm / idea challenges- good eg. ^ <- nother Phx co! ** investigate ExactTarget Customer Service - listen proactively, worst case = have a page and not attend - deleting neg comments is hugely detrimental, only if hate speech etc **** Kurrently realtime Twitter & Facebook monitoring - use discussions tab as a place to take discussions off the Wall. - key to establish hours of operation just like phone CSR's. ground rules for posting Fifth ^missed it - maybe someone else who was there can comment?
Tagged with:
Aug 07

July 7th I cancelled the account I’ve had with AT&T for the past twelve years and embarked on an experiment to see if it was possible to go carrierless and retain a somewhat normal existence. I’m happy to report after 30 days that this situation has worked surprisingly well. This post will summarize the key things I learned over the last month and offer some helpful nuggets of advice for anyone contemplating doing the same.


The goal in cutting my mobile phone service was very simply to eliminate a non-critical expenditure in the effort of getting extremely lean financially while working on our startup. I had used skype on my iPhone prior to the switch and knew that it handled calls well. My concerns were primarily whether running daily phone calls over Skype via iPhone would be annoyingly cumbersome.

My setup

I have an oldschool iPhone 3GS, a Verizon 3G Mifi mobile hotspot (grandfathered in under the unlimited bandwidth plan they used to offer) and the Google Voice and Skype apps on my iPhone.

The Switch Process

I ported my AT&T number to Google Voice for $20. Doing so automatically triggers the cancellation process w/ AT&T. You’ll want to check what your early termination fee is if you’re still under contract – mine was $70 so I more than covered it in the past month. It took a day for the number port to complete during which time all SMS messages sent to my phone blackholed. SMS apparently doesn’t work like email where it will keep retrying to send until it goes through. It took about 3 days before I began receiving SMS after the port. Once completed I spent another $20 to keep my previous GV # as an alternate phone (it was printed on all the business cards I had recently ordered and I like the idea of keeping separate business & personal #’s).

Unfortunately the GV app alone on your iPhone won’t enable you to send and receive calls. You’ll need to setup a Skype account if you don’t already have one and get an online number to be able to forward your GV # to. Skype offers a $3/mo option that gets you unlimited calls in the US. With the Skype number configured as the primary forwarding number in GV I was now able to receive and send calls from my iPhone.

The Good

  • The obvious benefit is that this eliminated a $130/mo phone bill.
  • There is an undeniable psychological win in saying FU to AT&T.
  • I’m navigationally-challenged and rely heavily on the iPhone for directions. The good news is the GPS is a true GPS and works via satellites (the cell tower triangulation is apparently the fallback method when it can’t acquire a GPS signal). This means the location-based features continue to work in spite of not having a cell carrier.
  • An indirect result of going carrierless was that I picked up all the benefits of using Google Voice. Transcribed voicemails is an awesome feature that eliminates listening to long messages.

The Bad

Now, the not-so-good:

  • You lose the ability to send picture messages over SMS. This isn’t a huge deal and I’m sure there are apps out there that give this capability. I haven’t really missed this – it’s easy to just email photos but if you rely upon sending picture texts for some reason, this is something to consider.
  • I have no 911 emergency dial. This is an acceptable risk for me but if you have dependents or frequently commute through dark allies, losing this might be unacceptable.UPDATE: apparently 911 does still work – thanks Dave for the clarification. Now I can continue to live dangerously!
  • Call quality is flaky over slow connections. Skype degrades the call quality when you have poor connectivity. It hasn’t been a problem yet but there have been a few instances when I’m working over a shady connection that I have to relocate to continue a call. The Verizon Mifi has been surprisingly good for calls with the exception of about 10% of the time it just seems to be unreliable. If you have a 4G mifi card I’m sure this isn’t even an issue. The answer when making important calls is to plan ahead to be working over a solid ground-based wifi connection for the call.
  • Unfortunately there’s no way to disable voicemail on Skype – this is a known shortcoming and a lot of people have complained about it. So to ensure that GV handles your voicemail the workaround is to enable call screening on GV (formerly known as “call presentation”). It’s only mildly annoying but means you’ll have to answer calls and click the “1” button to patch the person through.


So here are some random tidbits of advice based on what I learned:

  • Use Google Talk when in front of computer and save on outbound calls. I went through $9 in calls over the past 30 days on the pay-as-you-go plan. If you have the unlimited plan then this is irrelevant but Google Talk is free for US calls and you can offload some of your paid outbound calls to GT while you’re in front of the computer.
  • You need to setup Google Contacts for caller ID to work. This is simple/obvious but you need to have all your contacts loaded in GV for it to be able to associate names w/ calls and text messages. Skype will automatically pick it up from your Contacts in the iPhone so inbound calls should still register with a name.
  • If you’re calling someone who happens to have Skype the call quality is way better if you can just use Skype instead of Skype->phone.
  • Get the Chrome extension for google voice. It’s pretty slick – it gives you a toolbar button that lets you quickly make calls and check messages.
  • You need to remember to launch Skype when you restart your iPhone to keep it running in the background. Otherwise your phone won’t ring when you receive a call- you’ll just get the missed call alert via GV. I have it running on both my phone and computer and will typically just answer via the computer when I’m on it. What’s neat is the iPhone earbuds w/ the mic will work for talking on the computer and actually perform pretty well even in a noisy coffee shop.
  • The battery life on the Verizon Mifi is about 5hrs which is not enough to last all day. You should turn it on when in transit and switch over to a local wifi hotspot when you get where you’re going. If you forget and leave the Mifi on all day you’ll kill the battery. It has a USB input so you can get a standard cigarette lighter USB charger if you do a lot of driving.


The bottomline is this setup has proven totally adequate for my situation. There is some admitted flakiness over slow connections and annoyances like lack of ability to send SMS picture messages, but it’s 90% of the service for literally 6% of the price. Ultimately the factors you’ll need to consider in determining whether it’ll work for you are 1) criticality of 911 emergency calling 2) tolerance for occasional call flakiness 3) guilt level for not supporting your phone carrier. In tight economic times, saving $130/mo is significant – I would be re-examining my core business if I were a cell phone service provider. If anyone else has cut the cord and successfully run a similar setup I’d love to hear what your experience has been and if you have any hacks you made that improved it.

Jul 02

I recently made a series of dumb dumb dumb mistakes that culminated in the loss of about a week’s worth of work. In order to extract some positiveness from this incident I figured it would be good to do a post-mortem on exactly where I f’d up and what I learned so that I might perhaps save others from making the same mistakes. BTW this is only going to be mildly amusing/useful if you’re a geek – if you’re a layperson stop reading now b/c your eyes will glaze over. One thing I will say is having restored things at this point I have profound empathy for the couch surfer guy’s catastrophe and what he went through (I only lost a week of work – he lost 3 yrs with his “perfect storm”). Here’s what happened:

The context

So I’m admittedly terrible when it comes to attention to detail. The fact I somehow at one point programmed Cold Fusion web applications and commerce systems from scratch that handled hundreds of thousands of dollars of people’s money still boggles my mind. The fact is I know just enough tech to be dangerous and I try to leave the hardcore IT functions to others. In this particular situation though I was essentially working rogue to get a microsite up to test a new commercialization opp for Scratch Audio around the idea of facilitating online remix competitions. Using the free microinstance tier of hosting on Amazon EC2 and the WordPress JumpBox I figured I could implement a site in a weekend, throw some quick traffic at it and determine fairly quickly if there was resonance around this idea.

I started on a Friday and did a marathon session of pulling together all the marketing working on a local VM of the WordPress JumpBox. Given that I was working out of a cabin in the woods over a crumby connection, the aspect of being able to develop against a local server was really handy. By Sunday evening I had a microsite done in WordPress about 85% of what it needed to be. I used the JumpBox backup procedure to extract the state to a local file on my desktop, shut down the VM, made a timemachine backup of my laptop and did the 2hr drive back down to Phoenix feeling pretty good about things.

The next morning I woke up in Phoenix planning to use the JumpBox migration procedure to move my dev instance of the site to a live hosted scenario using the Amazon Free offering. I lit up a new instance on EC2 in minutes using the JumpBox launch widget (spiffy!), imported the backup file and checked the site. The page content was there but all theming was lost. Here’s where I made my first error

The failure sequence

Now I should have known this having used JumpBoxes for the past five years but the backup procedure explicitly excludes certain directories by default (and this is a sensible way for it to work). About half the work I had done in that marathon session was in making changes to the default theme. Had I installed a new theme and worked there, no problem… but alas I made all changes on the default theme which was excluded from backup. “No biggie, I’ll just grab the theme directory out of the local VM and use that.” Here comes mistake #2

That morning I realized I had mislabeled the directory that the VM lived in with the name “June 2010 site changes” (yea i’m frequently about a year behind). I fixed the date name on the directory thinking nothing of it earlier that morning. When I went to fire up the VM to grab the theme directory it was VMware armageddon. The first message was a helpful “a needed file cannot be found” warning. “Oh, must be that I renamed the directory. I’ll just rename it back.” Enter a barrage of new errors informing me that various i’s were not dotted and t’s were not crossed. I spent the next 2hrs learning the intricacies of the .vmx file, changing various settings, sacrificing a chicken, throwing some salt over my shoulder and finally was able to recover the VM (note: the sage advice from @godber – make a backup of the entire VM dir before you do anything).

Anyways, with VM restored I was able to manually grab the excluded theme directory via SFTP and push that into the Amazon EC2 instance. Worked like a charm and the new site was live!

Aaaaand… here’s where I made mistake #3.

I configured S3 backups and breathed an unknowingly false sigh of relief thinking “everything is on Amazon now and backups are in place. Nothing can go wrong.” Of course, my backups were still inflicted with the exact same problem that had forced me to retrieve the theme dir from the VM in the first place. (yeah this is why I have no business in IT folks ;-)

I trashed the VM on my laptop in order to save 6GB of disk space (I figured worst case I still had the timemachine backup at the cabin). With site working and eager to get some immediate feedback I implemented an Adwords campaign. Over the course of that week I iterated the marketing, implemented various tracking scripts like Chartbeat, Crazy Egg, Analytics, Optimizely and Adwords conversion tracking. On Friday evening I implemented a Stumble Upon campaign thinking “okay let’s get a broad swathe of musicians looking at it and see if anything shakes out.” Closed the laptop lid, went to happy hour… bad idea. Turns out microinstances fall down under load of 9 concurrent users on WordPress (and that’s even with Hypercache running). I get a Chartbeat page about an hour later that the site had gone unavailable. No biggie, I pause the SU campaign from my phone, pause the adwords campaign and figure “I’ll just restart it in the morning and run it under a larger instance size.”

I wake up the next morning, open my AWS console and am greeted with the cheery message “you have no instances running.” “Umm yea but what about the instance I was running last night that’s now unreachable?” Nothing. Worst case at this point I thought I had the S3 automated backup from the night before so I had only lost a day’s worth of modifications. Wrong.

Upon inspection of the S3 backup I realize my automated daily backups suffered the same (obvious) problem as the one I used to restore from and validate the nickname I earned in 1st grade: “absent minded professor.” < begin head slapping > Okay okay, worst case now I’ve lost changes back to Monday but now I need to drive up to the cabin and pray that the backup file I had in the cloud on S3 would restore successfully into the VM that would hopefully restore successfully from Timemachine backup I had on the firewire drive at the cabin (it was starting to feel like my data existed at the 4th level of Inception).

What was really puzzling though was how getting a slug of traffic to a microinstance could completely wipe it off the map? I would think it would hang it but not obliterate it and outright eradicate the EBS volume with the data. Completely baffled by this and with all hope lost on retrieving the EC2 microinstance at this point I happened to check the JumpBox GUI to see if I could access it from there. Miraculously it still showed it active although clicking the “Access” button just left the browser hanging. This didn’t jibe with what my AWS console was telling me but at this point I shrugged and used the JumpBox GUI to terminate the instance. Mistake #4

Turns out the instance was still there- it was just in the west region and the AWS console defaults you to the East region. So the data on the EBS volume was still there and retrievable right up until the nanosecond I clicked that terminate button… < commence Seppuku >

Given that the majority of the work that week had been in refining page content (which was protected by the S3 backups since it was stored in the database on the WP JumpBox) it wasn’t actually all that bad. I ended up driving back to the cabin, restoring the VM from Timemachine (which worked flawlessly), importing the latest S3 JumpBox backup into the local VM and using the WP Import/Export function plus some manual finagling to move the site. Having remembered most of the changes I had made that week it was a matter of reimplementing those and re-adding the various tracking scripts that were missing. In all, about 5hrs worth of duplication of effort to recreate everything under its new home.

I’m happy to report that remix.scratchaudio.com is now live on a server that can survive substantial traffic and we just had our first band signup yesterday.

What went right

For all that went wrong in this series of idiotic blunders on my part here are some things that went right:

  • Timemachine appears to be effing bulletproof
  • The JumpBox backups work flawlessly but with the caveat that you understand exactly what they’re backing up.

What I learned

  1. Test your backup procedures with an actual fire drill where you have to use them to restore your data. You are almost invariably guaranteed to learn something valuable from this exercise (even if it’s just the peace of mind of having done it – like changing a tire before you actually have a flat).
  2. I have no business running servers ;-)
  3. This is why services like Page.ly exist
  4. Don’t delete stuff until you absolutely have to. I had 150GB of free space on my laptop and yet I felt like I needed to get rid of this 6GB VM once I was finished with it. Dumb. Keep until you need to throw it away. There’s utilities like Disk Inventory X that make it easy to clean out the cruft eventually.
  5. Microinstances are handy, light-weight, disposable tools for dev/test but should never be used in production. They cannot handle any kind of load. Kimbro had actually told me this but it took experiencing it first-hand for it to sink in.
  6. EC2 instances never just disappear, they’re still there even when they become unreachable via the web. When something seems fishy, stop and seek alternate explanations and get a second set of eyes on it rather than trouncing forward and making the situation worse.
  7. VMware VM’s are surprisingly brittle – simply renaming the parent directory in which they reside unleashes a chain of events that makes it unusable. I’m shocked given that product’s level of maturity that they’re not more bulletproof. The good news is your data is still probably retrievable when things get moved around but you will spend the next two hrs wading through config files to manually futz with parameters in order to get it working again.

Anyways, hopefully this writeup is useful and saves even one person from making some of the errors I did in this debacle.

preload preload preload