Jul 24

So I’m reviving a tradition I had many years ago on here called “Kernel Dump.” It’s a periodic brain dump of everything I’ve been up to. It’s basically like declaring blogging bankruptcy for all the little things I’ve meant to write posts on and a way of clearing the buffer with a single catch-all update. This will be admittedly tl;dr for strangers and is more intended for friends and family that want to check in on me and see what I’ve been up to. Here goes as close to chronological order as possible starting from back in April timeframe:

A new startup

After almost five years at a company I helped start, I was providing little new value and it was time to move on. My cofounder continues to run it and I remain on the board but have since transitioned out of the day to day to pursue a new music-centric startup. We just closed our first licensing deal last week for the Scratch Audio technology with a company out of LA and are in process of proving out a niche application musical artists that utilizes our own platform and lets them enable fans to online.

Shortsale

Facing a growing amount of debt on my home in Scottsdale and a disappearing income source, I investigated the options of foreclosure and shortsale. I was fortunate to meet a solid dude in Phoenix via my friend Bill Ramsey who successfully navigated the short sale waters on my behalf and achieved a favorable outcome. I’ve been meaning to post a summary of what all was involved in that process. Before this past Christmas I had never heard the term before and in the span of one evening bouncing amongst three holiday parties, it came up in conversations at every party. I investigated it and determined it made a ton of sense given my situation. It’s an insanely useful option if you’re heavily under water and I believe if more people understood how it worked, more would take advantage of it.

Cabin move

With summer approaching and 110 deg days on the horizon I convinced my folks to let me move what few possessions I have up to their cabin south of Flagstaff. I’ve been living there since and have been lucky to have a rent-free temporary setup. My buddy Bryan who got me onto the Paelo lifestyle has been rooming there as well so I’m not a complete hermit at this point ;-) This is a guy I had met years ago who also ran a party bus business at the time. We share similar mentalities around CustDev and I’m 100% sold on the Paleo lifestyle at this point having lost 27lbs and 4 belt sizes. I could write for days on the back-to-basics life changes we’ve implemented with barefoot running, composting, chemical-free existence, organic-only diet and now with beginning to hunt our own food (Zuckerberg barely beat us to the punch).

Public Speaking

I was lucky enough on May 20th to have opportunity to speak for Ignite Phoenix as well as at the Arizona Disruptors Pitch Day. We ended up being extended an offer from Disruptors but graciously passed and opted instead to align ourselves with NACET, a technology incubator in northern Arizona with a great track record for helping companies commercialize their technologies. We still have huge respect for what Hamid and his crew are doing in Scottsdale (so great to see someone step up to the plate and incubate startups) but NACET was a better fit and has been a nice homebase in the pines. The Daily Sun had a pretty good summary the other week of what I’m doing there. I’m speaking tomorrow to a group of high school sophomores as a part of their YES week intro to entrepreneurship.

Microconf

Last month I got a windfall free ticket to attend MIcroconf in Vegas at the very last minute from the host . It was a conference I desperately wanted to attend but was precluded purely by cashflow. So of course I jumped on the opportunity and made the 4hr drive on I-40 from Flagstaff. Glad I did because I met a few incredibly badass people there. Some people are just bright stars, there’s no other way to put it. I wrote up some thoughts on Microconf afterwards. If you’re trying to strike out on your own with a small business in the web realm it’s absolutely something to put on your calendar for next year.

Going carrierless

So this is something I’ve so wanted to do for awhile and finally just pulled the trigger on last week. I killed my AT&T account and am now running Skype mobile on my iPhone over Verizon EVDO. I have the Verizon Mifi mobile hotspot which gives me mobile connectivity for my laptop and phone (I’m grandfathered in under an unlimited data plan for $60/mo so it’s tough to beat). I ported my cell number to Google Voice and have it so it rings my Skype on both my phone and computer. I now have a phone bill of only $3/mo. Calls are flaky at times but it’s 90% of the phone service I had before for about 2% the cost. I do important conference calls via the landlines at the NACET facility. And Verizon coverage is definitely superior to AT&T – before on the I-17 drive down to Phx my iPhone would cut out at various spots along the way. Over the Verizon card it has continuous service the entire way. Extremely happy with this setup. I plan to write-up a post exclusively on this topic once I hit the 30 day mark so I have enough context to comment intelligently on how adequate it is and what to know before switching.

Useful apps I’ve discovered

I moved my todo list to Things and have been very happy with it for GTD. It has native iPhone and Mac apps with a beautiful UI on each. My only complaint is it lacks OTA sync – it has sync over wifi so it’s not a huge deal but would be great if they offered it. Tungle has been a delight for scheduling calls. Those guys were recently acquired by RIM so I’m not the only one who feels this way. I was introduced to the iPhone app Fast Customer when I met its creator, Paul Singh while staying at Noah’s place for SXSW. Without question for $2 this app has the best ROI of any app I’ve ever purchased. Actually now it looks like it’s free so it technically has infinite ROI. It’s basically like having an assistant call a big company on your behalf and ring you once they have the CSR on the line. GroupMe is something I discovered at SXSW and it’s basically like an adhoc listserv that runs over SMS. It’s similar to Beluga only it doesn’t require that users have a smartphone since it uses SMS as the interface. Super useful for coordinating an evening with a loose-knit group of strangers. Glympse is another neat one. Almost like a more frictionless, temporary Loopt this app lets you publish your location via private Google map to select people. Similar to GroupMe it’s a useful tool for coordinating amongst groups moving around in an area.

Can of Awesome interviews

I made a New Years’ resolution this year to start podcasting interviews with people I admire. I did the first four starting out with the most influential mentors in my life prioritized in order of influence. I haven’t done one in awhile because I’m trying to keep with this trend and doing so mandates that I interview Jason Barney next. I hope to have that interview soon and may soon after open it up so it’s not impeded by this constraint. My intention in doing these interviews is to share wisdom from the coolest people I know. This is maybe a morbid thought but I want to pay tribute to incredible people while they’re alive and not wait to say nice things about them after they’re dead. Anyways, these are worth a listen. I know podcasts are an investment time-wise and I promise to not waste anyone’s time with these- there are already some incredibly interesting/valuable lessons shared by the guys I interviewed.

My eBook: Cart Drawn Horses

This is a work still in its infancy but I’m pulling together a small eBook that will contain everything I’ve learned about practical how-to for CustDev, rapid prototyping & validated learning. CustDev and Paleo are probably the two most life-changing things I’ve ever learned. There are great conceptual resources about CustDev but there seems to be a major void right now in terms of scarcity of practical guides “this is how you actually implement it.” I’m giving away the first 100 copies so if this is the type of thing that interests you be sure to sign up for a rough cut.

Music

I haven’t written much of any new music lately but a buddy and I and I have been playing a bit with our cover band Conjugal Visit. I attended a concert by The Glitch Mob in SF about this time last year and was blown away by what they were doing musically. That led me to start tinkering with a piece of software called Ableton Live. I now run the Ableton User Group in Phoenix and have met a bunch of people way smarter than I am who have been teaching me all kinds of stuff. I’m blown away by this software- it’s rare that something comes along and rewrites the rules for a field but this app is single-handledly changing what’s possible in a live performance with one or two people.

Travel

Other than Vegas I haven’t been traveling at all – been in cash conservation mode until I get some steady income from the various projects I’m working on. Two startup buddies of mine Chase and Josh just completed a month in San Francisco and I had really hoped to be a part of that but sadly it just wasn’t in the cards from a cashflow perspective. I’m hoping this latest deal we did for Scratch Audio will give me an excuse to make it out to LA next month. I met a ton of great folks last time I was there for Startup Weekend and have kept in touch with some. If you haven’t yet played with Namesake this is an interesting “Quora meets IRC” community. It seems to be still pretty heavily LA-centric but they opened it up to the public recently and it’s been getting a decent amount of growth.

Books

I’m about 2/3rds through Black Swan and have revised my opinion that it does actually have some value. Initially I was surprised why it got so much good press given that it was the equivalent of the falling rock sign for unpredictable catastrophic events. I read a new agey one recently called “Power of Now” – it’s a bit on the crystal-hugger side for me but it did one simple idea that has altered my thinking. The premise is that too often we live in the past or present (by hours, days or even years). There’s nothing wrong with planning or reminiscing in moderation but living in either means we’re missing an immediate opportunity. Life takes place in the present only. It’s such a cliched and simple concept but something about this book hammered this idea through and I’ve been consciously pulling my attention back into the immediate presence since reading it.

I got to review an advance copy of Derek Sivers’ book which is now out and it’s definitely worth reading (he even put me in the credits!). This is a guy who created a company from scratch and sold it for $20MM but then, more incredibly, put all the money in a trust for musicians. It reminded me a lot of Rework by 37 Signals- it’s a fast read packed with wisdom from this guy. I’m a week into resuming a course on speed reading that I had done ten years ago. This thing works if you stick with it. I believe being able to digest written information either 5x more or 5x faster than your competition is a huge advantage and this is the best resource I’ve found for increasing one’s reading speed. The other business one I’m reading is the 2nd edition of Solution Selling. I’m a horrible sales person and admittedly need to get better. Of all the books I’ve read on sales the first edition of this one had the most useful advice in terms of a framework and I’m looking forward to reading the updated version that has revised practices for software companies.

Fun Gadgets

I haven’t bought many toys lately but I’ve been getting a huge amount of enjoyment from this slingshot that we converted to shoot arrows. We haven’t used it to hunt yet but it’s absolutely capable of bringing down small game and given that compound bows start at $600 this $20 weapon is an awesome value. Chris Dumler in LA turned me onto this gravity-fed tea strainer and I use it daily. If you’re into loose leaf tea it makes it so much easier to make. My best friend gave me a mini RC helicopter for Christmas last yr and you just have to try these things to understand how cool they are. They’re about $25 on Amazon and gyroscopically stabilized so they’re super easy to fly. Lastly at the suggestion of this CNET review I got Klipsch noise canceling in-ear headphones. They make working so much better when you need to tune out background noise and zone out (great also for reducing travel fatigue on plane flights). BTW if you use Rdio for music, I have a great mix for zone out work music.

Learning

So aside from speed reading, I’ve committed to learning Rails and some basic development stuff. I figure at this point I’ll likely be involved with various web applications and dev teams for the foreseeable future and there’s huge value to understanding development from a low-level perspective. I met the Ming.ly guys in LA and Tyler pretty much sold me on Github, Rails, Hipchat, PivotalTracker & Heroku. After waffling on Python vs. Rails I’ve decided in favor of learning Ruby & Rails. I committed my first project to GitHub the other night. It’s a stupid simple Coldfusion app I wrote to be able to read Hacker News the way I like. My plan is to rewrite it in Rails and use that as a practical excuse to learn Rails. The Heroku deployment sounds awesome and I’ve already been tinkering w/ Pivotal Tracker which is very cool for agile project management. Lately I’ve also been tinkering w/ Agile Zen and actually did a makeshift Solution Selling pipeline using their free proj mgmt tool.

Plan for Q3

So as far as what I hope to accomplish this quarter:

  • I’d like to see us land another licensing deal for Scratch and ideally get it to a level of revenue that supports at least myself for now (as Bill Ramsey would say “a one-man boat”). Eric, my co-founder starts a new full-time job tomorrow and the ultimate goal will be to get revenue to a level that we can afford to have him full-time on it.
  • I’m very excited about testing this MixFork concept with musicians and proving that businesses can be built ontop of our technology platform.
  • We’re taking Paleo to a new level and I hope to have more to report on this front soon.
  • Really looking forward to incorporating Ableton in our live performances and writing some new original music.
  • On the side I’m pulling together a . This is the resource I wish had existed when I was going through the process and my goal is to give people a solid resource to educate themselves on what’s involved and take advantage of it as an alternative to foreclosure if it makes sense.
  • Lastly, you can’t time this type of thing but I’m hoping to meet someone. The solitary cabin existence has been super useful for getting a bunch done but it would be killer to meet another great girl at some point. I went on my friend Lyndsie’s show Valley Catch a few weeks back but no bites so far ;-)

Anyways, that’s the TL;DR update from Sean. If you’ve read this far and we haven’t chatted in awhile hit me up and let’s catch up. Santa Monica peeps: I hope to be back out there in September, if you’re around at that time let’s hang out.

Jul 02

I recently made a series of dumb dumb dumb mistakes that culminated in the loss of about a week’s worth of work. In order to extract some positiveness from this incident I figured it would be good to do a post-mortem on exactly where I f’d up and what I learned so that I might perhaps save others from making the same mistakes. BTW this is only going to be mildly amusing/useful if you’re a geek – if you’re a layperson stop reading now b/c your eyes will glaze over. One thing I will say is having restored things at this point I have profound empathy for the couch surfer guy’s catastrophe and what he went through (I only lost a week of work – he lost 3 yrs with his “perfect storm”). Here’s what happened:

The context

So I’m admittedly terrible when it comes to attention to detail. The fact I somehow at one point programmed Cold Fusion web applications and commerce systems from scratch that handled hundreds of thousands of dollars of people’s money still boggles my mind. The fact is I know just enough tech to be dangerous and I try to leave the hardcore IT functions to others. In this particular situation though I was essentially working rogue to get a microsite up to test a new commercialization opp for Scratch Audio around the idea of facilitating online remix competitions. Using the free microinstance tier of hosting on Amazon EC2 and the WordPress JumpBox I figured I could implement a site in a weekend, throw some quick traffic at it and determine fairly quickly if there was resonance around this idea.

I started on a Friday and did a marathon session of pulling together all the marketing working on a local VM of the WordPress JumpBox. Given that I was working out of a cabin in the woods over a crumby connection, the aspect of being able to develop against a local server was really handy. By Sunday evening I had a microsite done in WordPress about 85% of what it needed to be. I used the JumpBox backup procedure to extract the state to a local file on my desktop, shut down the VM, made a timemachine backup of my laptop and did the 2hr drive back down to Phoenix feeling pretty good about things.

The next morning I woke up in Phoenix planning to use the JumpBox migration procedure to move my dev instance of the site to a live hosted scenario using the Amazon Free offering. I lit up a new instance on EC2 in minutes using the JumpBox launch widget (spiffy!), imported the backup file and checked the site. The page content was there but all theming was lost. Here’s where I made my first error

The failure sequence

Now I should have known this having used JumpBoxes for the past five years but the backup procedure explicitly excludes certain directories by default (and this is a sensible way for it to work). About half the work I had done in that marathon session was in making changes to the default theme. Had I installed a new theme and worked there, no problem… but alas I made all changes on the default theme which was excluded from backup. “No biggie, I’ll just grab the theme directory out of the local VM and use that.” Here comes mistake #2

That morning I realized I had mislabeled the directory that the VM lived in with the name “June 2010 site changes” (yea i’m frequently about a year behind). I fixed the date name on the directory thinking nothing of it earlier that morning. When I went to fire up the VM to grab the theme directory it was VMware armageddon. The first message was a helpful “a needed file cannot be found” warning. “Oh, must be that I renamed the directory. I’ll just rename it back.” Enter a barrage of new errors informing me that various i’s were not dotted and t’s were not crossed. I spent the next 2hrs learning the intricacies of the .vmx file, changing various settings, sacrificing a chicken, throwing some salt over my shoulder and finally was able to recover the VM (note: the sage advice from @godber – make a backup of the entire VM dir before you do anything).

Anyways, with VM restored I was able to manually grab the excluded theme directory via SFTP and push that into the Amazon EC2 instance. Worked like a charm and the new site was live!

Aaaaand… here’s where I made mistake #3.

I configured S3 backups and breathed an unknowingly false sigh of relief thinking “everything is on Amazon now and backups are in place. Nothing can go wrong.” Of course, my backups were still inflicted with the exact same problem that had forced me to retrieve the theme dir from the VM in the first place. (yeah this is why I have no business in IT folks ;-)

I trashed the VM on my laptop in order to save 6GB of disk space (I figured worst case I still had the timemachine backup at the cabin). With site working and eager to get some immediate feedback I implemented an Adwords campaign. Over the course of that week I iterated the marketing, implemented various tracking scripts like Chartbeat, Crazy Egg, Analytics, Optimizely and Adwords conversion tracking. On Friday evening I implemented a Stumble Upon campaign thinking “okay let’s get a broad swathe of musicians looking at it and see if anything shakes out.” Closed the laptop lid, went to happy hour… bad idea. Turns out microinstances fall down under load of 9 concurrent users on WordPress (and that’s even with Hypercache running). I get a Chartbeat page about an hour later that the site had gone unavailable. No biggie, I pause the SU campaign from my phone, pause the adwords campaign and figure “I’ll just restart it in the morning and run it under a larger instance size.”

I wake up the next morning, open my AWS console and am greeted with the cheery message “you have no instances running.” “Umm yea but what about the instance I was running last night that’s now unreachable?” Nothing. Worst case at this point I thought I had the S3 automated backup from the night before so I had only lost a day’s worth of modifications. Wrong.

Upon inspection of the S3 backup I realize my automated daily backups suffered the same (obvious) problem as the one I used to restore from and validate the nickname I earned in 1st grade: “absent minded professor.” < begin head slapping > Okay okay, worst case now I’ve lost changes back to Monday but now I need to drive up to the cabin and pray that the backup file I had in the cloud on S3 would restore successfully into the VM that would hopefully restore successfully from Timemachine backup I had on the firewire drive at the cabin (it was starting to feel like my data existed at the 4th level of Inception).

What was really puzzling though was how getting a slug of traffic to a microinstance could completely wipe it off the map? I would think it would hang it but not obliterate it and outright eradicate the EBS volume with the data. Completely baffled by this and with all hope lost on retrieving the EC2 microinstance at this point I happened to check the JumpBox GUI to see if I could access it from there. Miraculously it still showed it active although clicking the “Access” button just left the browser hanging. This didn’t jibe with what my AWS console was telling me but at this point I shrugged and used the JumpBox GUI to terminate the instance. Mistake #4

Turns out the instance was still there- it was just in the west region and the AWS console defaults you to the East region. So the data on the EBS volume was still there and retrievable right up until the nanosecond I clicked that terminate button… < commence Seppuku >

Given that the majority of the work that week had been in refining page content (which was protected by the S3 backups since it was stored in the database on the WP JumpBox) it wasn’t actually all that bad. I ended up driving back to the cabin, restoring the VM from Timemachine (which worked flawlessly), importing the latest S3 JumpBox backup into the local VM and using the WP Import/Export function plus some manual finagling to move the site. Having remembered most of the changes I had made that week it was a matter of reimplementing those and re-adding the various tracking scripts that were missing. In all, about 5hrs worth of duplication of effort to recreate everything under its new home.

I’m happy to report that remix.scratchaudio.com is now live on a server that can survive substantial traffic and we just had our first band signup yesterday.

What went right

For all that went wrong in this series of idiotic blunders on my part here are some things that went right:

  • Timemachine appears to be effing bulletproof
  • The JumpBox backups work flawlessly but with the caveat that you understand exactly what they’re backing up.

What I learned

  1. Test your backup procedures with an actual fire drill where you have to use them to restore your data. You are almost invariably guaranteed to learn something valuable from this exercise (even if it’s just the peace of mind of having done it – like changing a tire before you actually have a flat).
  2. I have no business running servers ;-)
  3. This is why services like Page.ly exist
  4. Don’t delete stuff until you absolutely have to. I had 150GB of free space on my laptop and yet I felt like I needed to get rid of this 6GB VM once I was finished with it. Dumb. Keep until you need to throw it away. There’s utilities like Disk Inventory X that make it easy to clean out the cruft eventually.
  5. Microinstances are handy, light-weight, disposable tools for dev/test but should never be used in production. They cannot handle any kind of load. Kimbro had actually told me this but it took experiencing it first-hand for it to sink in.
  6. EC2 instances never just disappear, they’re still there even when they become unreachable via the web. When something seems fishy, stop and seek alternate explanations and get a second set of eyes on it rather than trouncing forward and making the situation worse.
  7. VMware VM’s are surprisingly brittle – simply renaming the parent directory in which they reside unleashes a chain of events that makes it unusable. I’m shocked given that product’s level of maturity that they’re not more bulletproof. The good news is your data is still probably retrievable when things get moved around but you will spend the next two hrs wading through config files to manually futz with parameters in order to get it working again.

Anyways, hopefully this writeup is useful and saves even one person from making some of the errors I did in this debacle.

preload preload preload