The Programmable Web – Yahoo Pipes

Duct tape is a great tool because it is so shiny and sticky. You can use it to glue so many things together, even if they end up looking like Frankenstein by the end of it. All you need is a camera, a cellphone, an MP3 player and a piece of duct tape to get yourself the only mobile convergence device worth having. The programming language Perl has quite often been called “duct tape for the internet” because it lets you easily transform text and interact with web sites.

(photo by philgarlic)

Here is a simple Perl script that downloads an RSS feed and bookmarks each entry to del.icio.us:

my $delicious = Net::Delicious->new(
				    {'user'=>$user,
				     'pswd'=>$password},
				    'updates'=>'.',
				    'xml_parser'=>'simple',
				    'debug'=>1);
my $feed = XML::FeedPP::RSS->new($rss_url);
$feed->normalize();         # Sort by pubDate and remove non-unique
foreach my $item ($feed->get_item()) {
  my $description = $item->description();
  $description =~ s/<.*?>//g; # remove HTML
  my %args = ('url'=>$item->link(),
	      'description'=>$item->title(),
	      'extended'=>$description,
	      'tags'=>"from_feed",
	      'replace'=>'no');
  my $retval = $delicious->add_post(%args);
}

The problem with Perl is that you have to either run it on your own machine, or buy web hosting that lets you run your own Perl scripts (or Python/Ruby). This is a real pain in the butt.

Enter Yahoo Pipes

Yahoo created one of the most innovative web tools I’ve ever seen. Yahoo Pipes lets you do all kinds of conversions and filtering on the web without requiring a web host to host your programs. If you want to convert XML/RSS data to other XML/RSS then look no further.

There’s still room for improvement:

• Scraping web sites that do not have information in XML/RSS. There are other companies that let you do this, but they’re even harder to use than Pipes (IE: dapper.net).
• Notifying you when your pipes don’t work. That makes finding existing pipes and mashing up multiple pipes tricky as best. It’s hard to use a tool when things constantly change underneath you.
• It would also be nice if allowed HTML in the Pipes descriptions as it is hard to describe how to use them sometimes.
• Better debugging messages when developing your own Pipes

Yahoo Pipes is targeted towards programmers, not casual users, but there is still a million and one things you can do with it. Here are some of my pipes that are free for other people to use.

Simple

• Switch title and description in any RSS feed
• Remove older results from a feed
• Remove del.icio.us daily link post from any blog feed

Social Sites

• Get your Technorati Rank as an RSS feed with a link to Add to Favorites
  • You will need your Technorati API key
• Keep track of when your site is submitted to Digg or Reddit
  • Displays number of Diggs in the title, and links to the Reddit voting/comment page instead of directly to the article
• Convert any Reddit feed to link to the comments page instead of the article it’s voting on
• Get a list of post titles and links (no description) from a Google Reader shared items feed

Personal

• Posts from my blog in the past 10 days
• Technorati Rank for engtech.wordpress.com

Got Pipes?

Are there any specific RSS feeds mashups you’re looking for but don’t have the Yahoo Pipes expertise to create? Leave a comment on this post and I’ll see what I can come up with.

See the full list of free software I have created.

You can get frequent updates about all of my new software, tools or blog themes by subscribing to IDT Labs by RSS or by email. Or you could just subscribe to my main blog, Internet Duct Tape.

Show Your Geek Love with Code Poetry (in Perl)

I’ve seen a few posts going around the net about geek Valentine’s Day ideas. If you really want to geek out why not go back a few years and use some Perl poetry?

Reprinted without permission.

# open HEART by Eric Windisch, Feb 15, 1999

   open (HEART, ">for_me");
   for ($this-valentines-day; $you and $me; $together++) {
     $you = "My special one";
     $me = "Your darling";
   }

   %time = ($you => $me, $together => "forever");
   while ($you = push(@me, @away)) {
     foreach (@second) {
       die a_bit to my $death;
       goto hell;
     }

     pack $my_bags, @and_leave;
     package my_love;
     unless(!$i_see_you) {
       write YOU_SOON;
     }
   }

   reverse keys %time;
   bless me;
   for (last; kill $me;) {
     if ($you) {
       die;
     }
   }

# rush by Sharon Hopkins, June 26, 1991

    'love was'

      && 'love will be' if
	(I, ever-faithful),
	  do wait, patiently;

    "negative", "worldly", values disappear,

      @last, 'love triumphs';

    join (hands, checkbooks),
      pop champagne-corks,

	"live happily-ever-after".

	  "not so" ?
	    tell me: "I listen",

	      (do-not-hear);

    push (rush, hurry) && die lonely if not-careful;

    "I will wait."

      &wait

# Perl Haiku by Mike Badger

    my $Life;			#Defineing my life
    $Life=$me || $you		#A Union of Me and you
      if ($ourLove);		#if our love is true.

# Perl Haiku by Bob Meyers

    use strict; my @love;
    my $wounds = open(FLAME, "of_passion");
    foreach (<FLAME>) {
      push @love "fully";
    }

More Perl Poetry Links

• Code Poetry at runme.org
• Perl Poetry Contest #1
• Perl Poetry at PerlMonks
• Camels and Needles – Computer Poetry Meets the Perl Programming Language (translation of PostScript)
• The History of Perl Poetry
• Magnetic Poetry for Perl (software app)

Related Posts

• 76 Romantic Movies for Guys and Girls
• The Secret to Finding the Perfect Valentine’s Day Gift for the Person You Love
• Ode On a Silicon Yearn
• Geek Valentine’s Day Links

Blog Housekeeping: Cleaning Up Dead Links on Your Wordpress.com Blog

Recently Lorelle was doing a one year retrospective of the various articles she’s written on blogging. She brought up the topic of “blog housekeeping“, which is very important.

Blog maintenance can be broken into different categories:

• Frequent maintenance
  1. Posting
  2. Linking back to older posts (“deep archives”)
  3. Answering comments
  4. Banning comment spam
• Occassional maintenance
  1. Backing up the blog
  2. Generating buzz
• Infrequent maintenance
  1. Pruning dead links
  2. Changing themes
  3. Upgrading software (scripts, plugins, etc)

I haven’t checked my site for dead links yet, so I’ll do it for the first time. Here’s how:

Using Perl on Windows Part 1: Installing ActiveState Perl

When I want to get something done with Windows XP, I have a habit of using Perl because I’m very familiar with it (I’m a Unix geek).

Luckily, Perl is available as a Windows executable thanks to the people at ActiveState. This guide will show you how to install ActiveState Perl on Windows.

Installing Perl isn’t hard, but I plan to make some scripts available for people unfamiliar with Perl. This guide is intended for them.

Perl Oneliner for date calculation

Perl is great for writing oneliner scripts to do simple task. I’m often confounded that there are no inherent unix utilities for doing date arithmetic / calculations, since so many unix utilities require the date as an option. Thanks to the Perl Date::Calc module, that isn’t an issue.

Here’s a one liner to find out what the date was 28 days (4 weeks) ago:


perl -e 'use Date::Calc qw(Today Add_Delta_Days); my ($y, $m, $d) = Add_Delta_Days(Today(), -28); print "$y/$m/$d\n";'


Note the value of -28 to tell it to subtract 28 days from the current date. Because it is using Date::Calc, it will correctly handle holidays, different months, leap years, etc (as opposed simply subtracting one from the month which will net you something between 28-31 days).

Note the sexy ISO-8601 return value.

Presentation Tutorials on Regular Expressions and Perl

Turing Labs have some good tutorials on system administration, but what stuck out for me were some excellent presentation slides for regular expressions and perl. These would be great for someone who wants to do quick refresh of something they already know, but it would be even better for putting together a quick lunch-time learning presentation but didn't want to invest the time (or more realistically, doesn't have the time to invest).

• Regular Expressions (August 14th, 2004)
• Introduction to Perl (July 30th, 2005)

Turing Labs – GNU/Linux and BSD System Integration : Managed Servers and CRM : Network Security Solutions : Training

Doing it Better – Avoiding backticking `pwd`, `find .`, and `mkdir -p` in Perl

When writing perl scripts, it can seem very natural to incorporate shell commands using backticks. Unfortunately, this is almost always the wrong thing to do because you are introducing a dependency on an external program without adding any of the necessary logic that might be required for portability. This isn't a very big issue for a one-off throw-away script, but in a work environment, scripts have a bad habit of being extended and reused well past your initial intentions.

This article covers some basics:

Get current working dir/Avoiding `pwd` – use Cwd
Get all of the files/subdirs in the current dir/Avoiding `find .` – use File::Find
Creating a new subdirectory / Avoiding `mkdir -p` – use File::Path

Perl – Doing it Better – Avoiding backticking `pwd`, `find .`, and `mkdir -p`

Verilog-Perl – Perl modules for interacting with Verilog code

 Verilog-Perl is a collection of perl modules for interacting with Verilog code. I haven't given it much of a look as of yet, but if "vpm" and "vppp" work as the synopsis advertises then it may have some utility. Especially if it can already correctly parse SystemVerilog.

• Verilog::Getopt parses command line options similar to C++ and Vcs.
• Verilog::Language knows the language keywords and parses numbers.
• Verilog::Netlist builds netlists out of Verilog files. This allows easy scripts to determine things such as the hiearchy of modules.
• Verilog::Parser invokes callbacks for language tokens.
• Verilog::Preproc preprocesses the language, and allows reading post-processed files right from Perl without temporary files.
• Vpm preprocesses warnings and assertions for any simulator.
• vppp preprocesses the complete Verilog 2001 and SystemVerilog language.
• Vrename renames and cross-refrences Verilog symbols.

Verilog-Perl

Safe Scripting in Perl — don’t trust the user

This covers some interesting thoughts about using Perl from a security perspective where you don't trust the user.

• Use taint to make sure that tainted variables (that came from user input or environment variables) never make it into system/exec calls.
• When using an optimized regular expression, use the /Q /E metacharacter to prevent an eval statement from running arbitrary code.
• Tips and techniques for running a script as a specific user instead of "nobody".

WWW Security FAQ: Safe Scripting in Perl

symbolic link checker in Perl and script to find biggest files

These are a couple of perl utilities I've found useful.

linkcheck.pl is for finding broken symbolic links. biggest.sh is for finding the largest file on a file system (ie: when you've run out of diskspace and you need to find the culprit)

linkcheck.pl looks for all links on the local file system and tests each link's integrity–to determine whether the link still valid, or if it points to an object that no longer exists. Only local filesystems are searched and this is ensured by testing each filesystem argument via "df -l" command.

Listing broken links is helpful, and you can certainly write a much shorter Perl program to do that, but linkcheck.pl comes with other useful options:

-a Display All links
-H Detailed documentation
-h Usage brief
-l Long list (e.g., ls -al)
-r Remove broken links (use with caution)
-v Verbose output
fs Required filesystem for search (multiple filesystems may be specified)

How many times has a file system filled up causing you to spend more time than you liked finding that hernia file or files? I'll wager more times than you care to remember. After I lived with that aggravation for way too long, I put together biggest.sh.

Biggest.sh has run successfully on Sun Solaris and Linux. It is written in Bourne shell (instead of Korn or Bash) for one reason only – portability. The script runs under Bourne, Korn, and Bash environments without modification.

   Usage: biggest.sh -fHh -l <nn> -v -t <dir> -s <nnn> fs     -f - Follow links
     -H - Displays detailed documentation
     -h - Provides usage brief
     -l - Displays <nn> lines (default is 500)
     -s - Minimum file size is <nnn> (default is 500K)
     -t - Use <dir> as temp/work directory (default is /var/tmp)
     -v - Edit (vi) file list
     fs - Required file system argument

Bob Orlando's Unix Tools

Perl Doxygen Filter — Automatically Document Your Perl Code

About Perl Doxygen Filter

Of course, Perl developers are used to use POD rather than some other code documentation tools. However, most developers actually are not restricted to using one single language. Instead of using multiple code documentation systems one tends to use one tool for all – Doxygen is quite a powerful code documentation system that already has built-in support for multiple programming languages.

Unfortunately, Doxygen does not directly support Perl. Thus, Doxygen Filter has been written in order to be able to use Doxygen for generating code documentation for Perl projects, too.

Linklint – fast html link checker in Perl

Linklint – fast html link checker

I've used this tool for checking intranet HTML documentation I took over that was poorly maintained. It will generate a summary like the following as well generating HTML results that show all of the missing links, etc.

Linklint found 196 files in 3 directories and checked 136 html files. There were 41 missing files. 27 files had broken links.
119 errors, 5 warnings.

found 3 directories with files
found 1 default index
found 135 html files
found 51 image files
found 9 other files
found 9 http links
found 8 mailto links
found 1 file link
found 377 named anchors
—– 1 action skipped
—– 88 orphans
warn 5 warnings
ERROR 37 missing html files
ERROR 4 missing other files
ERROR 78 missing named anchors

Perl one-liner for replacing text in a file.

perl -w -i -p -e's/string/string/g;' $*;