// Internet Duct Tape

The Programmable Web – Yahoo Pipes

Posted in IDT Labs Software Development, Perl, Technology, Yahoo Pipes by engtech on May 30, 2007

Duct tape is a great tool because it is so shiny and sticky. You can use it to glue so many things together, even if they end up looking like Frankenstein by the end of it. All you need is a camera, a cellphone, an MP3 player and a piece of duct tape to get yourself the only mobile convergence device worth having. The programming language Perl has quite often been called “duct tape for the internet” because it lets you easily transform text and interact with web sites.

(photo by philgarlic)

Here is a simple Perl script that downloads an RSS feed and bookmarks each entry to del.icio.us:

my $delicious = Net::Delicious->new(
my $feed = XML::FeedPP::RSS->new($rss_url);
$feed->normalize();         # Sort by pubDate and remove non-unique
foreach my $item ($feed->get_item()) {
  my $description = $item->description();
  $description =~ s/<.*?>//g; # remove HTML
  my %args = ('url'=>$item->link(),
  my $retval = $delicious->add_post(%args);

The problem with Perl is that you have to either run it on your own machine, or buy web hosting that lets you run your own Perl scripts (or Python/Ruby). This is a real pain in the butt.

Enter Yahoo Pipes

Yahoo created one of the most innovative web tools I’ve ever seen. Yahoo Pipes lets you do all kinds of conversions and filtering on the web without requiring a web host to host your programs. If you want to convert XML/RSS data to other XML/RSS then look no further.

There’s still room for improvement:

  • Scraping web sites that do not have information in XML/RSS. There are other companies that let you do this, but they’re even harder to use than Pipes (IE: dapper.net).
  • Notifying you when your pipes don’t work. That makes finding existing pipes and mashing up multiple pipes tricky as best. It’s hard to use a tool when things constantly change underneath you.
  • It would also be nice if allowed HTML in the Pipes descriptions as it is hard to describe how to use them sometimes.
  • Better debugging messages when developing your own Pipes

Yahoo Pipes is targeted towards programmers, not casual users, but there is still a million and one things you can do with it. Here are some of my pipes that are free for other people to use.


Social Sites


Got Pipes?

Are there any specific RSS feeds mashups you’re looking for but don’t have the Yahoo Pipes expertise to create? Leave a comment on this post and I’ll see what I can come up with.

See the full list of free software I have created.

You can get frequent updates about all of my new software, tools or blog themes by subscribing to IDT Labs by RSS or by email. Or you could just subscribe to my main blog, Internet Duct Tape.

Subscribe to feed

12 Responses

Subscribe to comments with RSS.

  1. prof kienstra said, on July 29, 2007 at 1:03 pm

    thanks for the tip!

  2. Yusuke said, on August 02, 2007 at 9:38 am

    my $description = $item->description();

    is missing. Thank you.

  3. engtech said, on August 02, 2007 at 10:26 am

    @yusuke: good find, thanks.

  4. […] own server — so why are you using WordPress.com?) Thankfully, Yahoo provides a service called Yahoo Pipes that let you merge, mash, filter, and combine other RSS feeds into a new feed. It even has the […]

  5. KC Lau said, on September 29, 2007 at 11:03 pm

    Hi, I am always amazed when you publish a long list of the best of feed. I am sure that you use some kind of method (Yahoo pipes maybe) to create those posts instead of editing them manually. Would you share about your method?

    I have a lot of post that I like to share with readers but find it a very tedious work to edit the post. Sometimes it takes more than a few hours to complete the best of feed post.

  6. engtech said, on September 30, 2007 at 10:20 pm

    @KC Lau:

    I use a custom program I wrote to do it, but the easiest way to do it is with Yahoo Pipes.

    Step #1: Get an account on del.icio.us and use that for bookmarking links

    Step #2: Install this Greasemonkey script that makes it easy to cut-and-paste from Yahoo Pipes

    Step #3: Find the RSS feed for your del.icio.us account, IE:
    http://del.icio.us/rss/engtech/linkblog — an example of the RSS feed for a specific tag

    Step #4: Go to this Yahoo Pipe and cut-and-paste your delicious RSS feed instead

    Step #5: Click Run and cut-and-paste the result into a blog post

    Wish I’d thought of doing it that way before I spent the time writing that program :)

  7. KC Lau said, on October 01, 2007 at 6:28 am

    Thank you very much! You are my tech guru :)

  8. […] List of my Yahoo Pipes for bloggers […]

  9. […] Provide content for your blog […]

  10. Peter Kelley said, on February 19, 2008 at 6:28 pm

    The pipe that I want is one that takes a del.icio.us feed and morphs the dc:subject tag into a bunch of category tags, one for each space separated tag in dc:subject. I’m assuming string tokenizer but I just can’t make the jump from a bunch of strings in an array variable to new tags on a feed item.

    Any ideas?

  11. Faggot83 said, on October 22, 2009 at 6:04 pm

    Biodiesel Handling and Use Guidelines. ,

Comments are closed.

%d bloggers like this: