Dec 09
Not really. But these two services when used in conjunction with one another give you the data-mashing powers of Chuck Norris and a roll of digital duct tape that would make MacGyver jealous. Below is a video screencast tutorial on how to get started with the Yahoo Pipes and Dapper services. Dapper lets you essentially construct an API for any web site while Yahoo Pipes lets you consume that API and perform operations on the data to turn it into something more useful.
The problem we’ll solve in the next 18 min: there’s currently no easy way to subscribe to the 200+ local bloggers listed on Read Phoenix (short of visiting each blog and sub’ing the RSS feeds individually). In this tutorial we’ll build an app from start to finish that spiders the list of bloggers on that site, grabs the latest posts from each blog and provides a single, chronologically-sorted master feed of the most recent posts and filtering out auto-generated bookmark posts. Here’s the tutorial:
The assets for this tutorial can be found here:
What kind of useful mashups could you imagine creating with these tools?
Hey Sean, thanks for the run through – I’ve never actually used Dapper before, and even though it was a basic usage, it was neat to know it’s a scraper. Cool pipes usage also.
Nice technical post – kudos.
Very cool! Thanks much for this video.
Worth exploring to see if filters can be added so that you only see posts with select keywords.
Sean, AWESOME!! I’m going to subscribe to that feed right now. Thanks!
@chuck @brian- thanks!
@james – it could definitely be made to do that. A cool thing about Pipes is that once they’re published it’s like open source – anyone can clone what you made and extend or alter it. It’s about the best way to learn by browsing the library and seeing how people did stuff. You could definitely add a user input block and wire it up to the filter i made and have it only allow posts that contain a certain keyword in the body or title. good suggestion – click on the pipe link above and go make it happen ;-)
sean
[…] How to launch rockets with Dapper and Yahoo Pipes | Scrollin’ on Dubs – The problem we’ll solve in the next 18 min: there’s currently no easy way to subscribe to the 200+ local bloggers listed on Read Phoenix (short of visiting each blog and sub’ing the RSS feeds individually). In this tutorial we’ll build an app from start to finish that spiders the list of bloggers on that site, grabs the latest posts from each blog and provides a single, chronologically-sorted master feed of the most recent posts and filtering out auto-generated bookmark posts. […]
[…] While this particular app may not be all that relevant to you, it should get you thinking about how you could use this tool. Ponder for a minute all the mini-challenges you encounter in IT when you have these isolated systems with bits of data that need to be brought together in one place. What could you do with a tool that lets you wrangle records out of a MySQL database, scrape data off a web page, mash it up with data via a public web service and transform it into a web page or an RSS feed or a csv file or an email or… This is an extremely interesting JumpBox and definitely worth playing around with. If you want to tinker, skim the video to get the basics and then launch your own private instance using the orange button on the widget below. And visit SnapLogic.com to learn more about their offering. If you’re really into this stuff, take a look at Yahoo Pipes and compare how the two systems work. Happy tinkering! […]
[…] intelligence systems are too heavy for most early-stage startups. But free online tools like yahoo pipes & dapper feeding a google spreadsheet can give you much of what you need. Of course if you’re slightly […]