Search results via RSS now available

Techmeme Search now provides results through RSS. What does this enable? A lot of things. You can now subscribe to search results in iGoogle, Netvibes, or your favorite RSS reader. Developers can hook up other kinds of applications to Techmeme Search as well. Wherever background information on a name or topic in technology can provide useful context, Techmeme Search can help. You can see this on CrunchBase's company profile pages (here's one for Twitter) where Techmeme headlines in the right margin provide expanded news coverage.

Obtaining the RSS feed URL: On any results page, simply click the RSS button in the browser's location bar, usually an orange or blue square icon like the one above, but tinier. In Safari, the icon simply contains the letters "RSS". A feed URL link also appears at the bottom of any results page. Look (again) for an orange RSS icon. Subscribing involves copying this URL to your aggregator of choice.

"About" vs. "any mention": Remember that Techmeme Search runs in either of two overall modes. Suppose you're searching for "eBay". In the default mode, it tries to find results about eBay by searching only the title and first few sentences for "eBay" (example results). To get results that mention eBay anywhere in the post, uncheck "Search title & summary only" (example result). The associated RSS feeds for these two alternatives preserve this searching behavior.

About Techmeme Search: Techmeme Search was introduced May 20, and demands for RSS feeds arrived (of course) on May 20. Bloggers soon came through with other helpful information. Among them was Amit Agarwal, whose post provides a link that instantly adds Techmeme Search to the Firefox or IE toolbar. Another was Louis Gray, who uses Techmeme to examine who broke (or otherwise gave early notice of) stories of enduring interest.

Techmeme listens: 32 months in the making, search arrives!

Hours after Techmeme launched in 2005, search uberblogger Danny Sullivan remarked "there's no keyword search facility that I can see. I want that, and soon!" Nobody wants to let Danny down, so I got right to work and 32 months later, a search box now sits atop the site.

Well, actually, work on search was rather seriously delayed. For years, I blew it off since it always seemed a secondary concern. Techmeme was focused primarily on surfacing the newsworthy, not providing Yet Another Blog/News Search. Even Danny came to agree with me on that point.

So why now? As an increasing number of users became increasingly dependent on Techmeme, one common use case emerged: people wanted simply to recall things they'd seen on the site early. Another use is best illustrated in the "Techmeme Posts" listed in the right column of CrunchBase's Facebook company profile. These are posts about Facebook that surfaced recently on Techmeme. Listing them in reverse chronological order offers a concise list of the company's latest milestones. This shows that another great use for search is in producing a timeline of major events for any topic.

Posted headlines only, newest first: Techmeme Search returns items that have appeared as full headlines on Techmeme, in reverse chronological order. Headlines appearing only in "Discussion" are excluded. This way, Techmeme Search aims to surface only the most notable results for a query. For more exhaustive results, services like Ask Blog Search, Google News, and Topix already do a good job of returning everything that they've indexed.

Though a search box sits atop Techmeme's main page, you can also find one in the obligatory bare bones search-only start page at techmeme.com/search.

Full text search isn't default: There are two overall modes of searching, depending on how "close" a result is desired. The default mode only returns matches occurring in the title or the first couple of sentences. Searching for "Yahoo" in this mode typically return stories about Yahoo. Unchecking "Search title & summary only" on the result page (or the bare bones page) enables search of the full article text. In this mode, any article simply mentioning "Yahoo" will appear.

Narrowing results by source url, author, date, and other attributes is also supported. For instance, "sourceurl:http://searchengineland.com/" returns posts only from the blog Search Engine Land (as seen here). A concise list of all the search operators involved is available through the "Advanced" link on a search results page (or again, on the bare bones page).

What's next? Search for Techmeme's sister sites are planned but not active as of today. Also on the way are RSS feeds for search results, the simplest kind of search "API".

Final note: I'm happy to report that Techmeme Search was implemented not by myself but by Techmeme's first hire, Omer Horvitz. So I can now say "we" when referring to Techmeme. I'm still trying to get used to the sound of that.

Update: Some coverage around the web: TechCrunch broke the news here. Techmeme Search is bub.blicio.us and even "kills" Technorati for Mashable's Stan Shroeder. Charlie Anzman is pleased with "TechMeMe" search. Amit Agarwal provides a link that instantly adds Techmeme Search to Firefox's or IE's toolbar. Download Squad says Techmeme is now "useful to the general public". Amit Chowdhry says it's helpful for "media researchers such as myself". Susan Mernit wants RSS by tomorrow. Techmeme Search lands at Search Engine Land, and appears inside Outside the Lines. Nick Bradbury has enabled Techmeme searches inside FeedDemon. Search Engine Journal and Rev2.org weighed in. The "My Blog Posts" blog measures mentions of Indian cities with Techmeme Search. Blogcosm offers a general roundup.

New Twitter and RSS "Firehose" feeds include all headlines

High-volume, "complete" RSS feeds are available now for Techmeme, WeSmirch, memeorandum, and Ballbug. For years, the RSS feed for each site has included the day's most important 20-30 headlines, typically those placed "above the fold". Going beyond 30 headlines would overwhelm most subscribers. This selectivity has a downside: headlines don't appear in the RSS feed until they've amassed enough "importance", so the feed seems "slow". Another, more obvious downside: certain news junkies want to see every last headline that appears.

The new "Firehose" feeds introduced today do just that. Any story attaining headline status on Techmeme (which exclude "Discussion" links) will appear in the feed within 5 minutes. This means well over 100 headlines per day for Techmeme Firehose, so casual news readers will want to use the Main feed (i.e. the previous, standard feed) instead.

Twitter accounts for both Main and Firehose feeds are available as well. Users of Twitter, even more than users of RSS readers, tend to favor a more immediate view of events around them, and some have complained about the "slowness" of the existing Techmeme Twitter account (which is based on the Main feed). Following Techmeme Firehose on Twitter will provide the real-time experience these users crave.

The following table summarizes all the relevant URLs:

Firehose RSS	Firehose Twitter	Main RSS	Main Twitter
Techmeme	Techmeme	Techmeme	Techmeme
WeSmirch	WeSmirch	WeSmirch	WeSmirch
memeorandum		memeorandum
Ballbug		Ballbug

Guess who won a Crunchie!

Friday's Crunchies award ceremony was a blast for many reasons, not least of which was Techmeme's win in the Best Boostrapped Start-up category! It was nice to see Techmeme sponsor Zoho take one home as well.

Much thanks to everyone making this possible: Techmeme's voters, readers, and sponsors, plus the publishers who create the stuff Techmeme links to. Thanks also to the event sponsors for making the ceremony so enjoyable, among them Techmeme sponsors Adobe and Microsoft. I'm looking forward to next year's event, no matter who gets nominated.

Mobile browsing of Leaderboards, via Mowser

The announcement of the Techmeme Leaderboard last October made only a passing reference to its OPML feed. I didn't know what would come of the feed, and mostly forgot it was there.

So of course, a nice application did appear: a web app for mobile browsing of the Techmeme Leaderboard. The great thing about it is how clicks on sources naturally give way to post summaries, which give way to original posts, all of it rendered for mobiles.

It's a very smart combination of the capabilities of Mowser, a mobile transcoding service and the data contained in the Leaderboard's OPML feed. For more on this see this post by Russell Beattie, Mowser's founder, and his followup where he introduces a companion service for memeorandum's Leaderboard.

Mowser's Leaderboard services are a special case of "OPML grazing", for which general apps have appeared in recent years. Last fall, Ross Mayfield embedded a grazing widget powered by Grazr in this post which similarly uses the Techmeme Leaderboard's OPML as a starting point. You can customize your own Grazr Leaderboard widget starting at this page. They also offer iPhone-optimized grazing. Several RSS aggregators have similar capabilities built-in. Blogbridge's Pito Salas, for instance, explained how the OPML file could enable a Techmeme Leaderboard "Reading List" subscription.

If you know of any more applications for the Leaderboard OPML feeds, please let me know.

Time-lapse Techmeme!

Along with surfacing stories and putting them in context, one of Techmeme's chief functions is providing a kind of visualization for news. Headline size, cluster size, and "Discussion" length all signal the structure of the buzz surrounding a story.

I've always wanted to make a cool, time-lapse view of Techmeme to take this to the next level. So I was very happy to find Amit Agarwal arrive at the idea, and most importantly, pull it off.

Fifty hours in the recent life of Techmeme unfold in a mere 50 seconds in the video available at this link. It's also embedded below, but a little hard to read due to compression, so use the previous link if you're on broadband.

What are memeorandum's top 100 sources?

The ascent of blogs and other independent publishers has expanded and revitalized political dialogue in the United Status. While the "mainstream" media remains the primary source of factual reporting, they no longer own the agenda. It follows then that tapping a diversity of media sites is essential to produce memeorandum's autosummaries of the web's most buzzed-about political items.

What exactly are these sources? The memeorandum Leaderboard, launched today, identifies 100 of them, ranking sources simply by how much they've appeared on memeorandum in the past month. It updates every 20 minutes and offers archives of past days. This follows the introduction of a similar list for Techmeme ennumerating leaders in tech reporting.

The memeorandum Leaderboard doesn't tell the whole story of course. For instance, influential curators of opinion like Instapundit.com don't figure highly given memeorandum's preference for longer articles. Yet it remains a handy portal to many of the sources with the greatest role in framing and shaping the national debate.

Check out WeSmirch's top 100 sources

Celebrity news sites, like the stars they cover, do not draw equal attention. Some serve millions of visitors while others are still working for their "first million".

WeSmirch, the real-time celeb news tracker, has its own favorites, those sites it relies on most for headlines. And the WeSmirch Leaderboard, introduced today, lists the top 100 such sources. Every 20 minutes it ranks them by how much they appear on WeSmirch. A similar list for Techmeme went live last week.

Since WeSmirch favors original news reports, well-funded news operations tend to lead the list. Yet filling out much of the Leaderboard are the smaller indie blogs adding a dose of opinion and snark to the day's news.

The list aims to provide a varied tour of the web sites currently dominating the celeb beat. I hope you enjoy it.

More sponsors: Graphing Social Patterns, Assembla, and Socialtext

Two great sponsors have arrived on Techmeme, joining four wonderful existing sponsors. But first, some notes on a September sponsor not introduced earlier:

Socialtext and Assembla are the new additions (though Socialtext happily appeared in 2006 as well):

You're hopefully familiar with Techmeme's current sponsors:

Occasionally blog posts rounding up headlines from Techmeme's sponsors will appear in Techmeme's RSS feed. Please be sure look them over — these are innovative companies who understand and seek to reach Techmeme's readership. They write good stuff! I'm happy to have them all as sponsors this autumn.

Techmeme Leaderboard is live

The Techmeme Leaderboard, that much-leaked list of Techmeme's top 100 sources, including blogs, non-blogs, and everything in between, is now up. It ranks sites, every 20 minutes, simply by the amount of headline space they've occupied on Techmeme over the past month.

Why now?  I suppose this is long overdue. For two years, I've been urged to publish such a list. Why? Techmeme, in surfacing the latest tech news, also identifies leaders in tech reporting. As a friend who works in PR recently told me "I gauge hot new blogs via Techmeme". Yet I hesitated, perhaps given the potential for misunderstanding. Any ranking invites attacks on the methodology, attacks over "objectivity". I hope to address this now; in short, Techmeme is biased (more below).

A more mundane reason for launching now: I've heard that people were even constructing and circulating unofficial lists. Might as well make the official one.

Methodology:  A source's presence is the probability that a random Techmeme headline at a random time over the past month was published by that source. The Leaderboard ranks sources by presence. What is a source? Sidestepping knotty issues of ownership and affiliation, sources are simply identified by the branding a publisher chooses. So blogs are generally distinct sources from their parent site. Thus, Saul Hansell writes for two different sources: Bits (the NYT blog), and the New York Times proper, even though the New York Times Company publishes both. The same goes for CrunchGear and TechCrunch and other blogs contained in blog "networks".

Because presence is additive, anyone can construct their own "supersources" from the table and rank accordingly. So summing presence for ZDNet blogs such as Between the Lines and All About Microsoft along with plain old ZDNet yields a total indicating a ranking much higher than the individual sources.

Is it biased?  I wish it were obvious, but there's no such thing as an unbiased automated news site (or search engine for that matter). Automation doesn't remove bias, it merely obscures it. The configurations that make Techmeme a tech news site embody some of that bias. Beyond that, headlines are also skewed by Techmeme's emphasis of business news over areas like video game reviews, developer news, gadget arcana, and green tech. Finally, influencers that communicate mainly in links don't figure prominently on Techmeme. Slashdot is widely read, yet absent from the top 100.

OPML and archives:  Obtain Leaderboards from earlier dates by typing the date in the "History" box on the right. Archives begins on September 30, 2007, so there's little to see as of yet. An OPML file is also available, enabling external mashups using the Leaderboard data.

How to use:  Since the Techmeme Leaderboard reflects the reality that both blog-driven sites and traditional sites define today's news, use it to discover new sources, recommend sites to others, or illustrate where tech news breaks. I hope you find it useful, and if you have a stake in tech reporting, not too infuriating.

On AIR, Zoho, Channels 9&10, and Compete.com sponsor Techmeme

Beginning today, new sponsors "on AIR" (Adobe) and Compete.com join previous sponsors Zoho and Channels 9&10 (Microsoft) in supporting Techmeme this summer.

On Air is a brand new blog covering the Adobe Integrated Runtime (AIR). So you saw their first post, written by Ryan Stewart, on Techmeme! The remaining three have all been making news on Techmeme through their excellent blogging. And that's no exaggeration (example). Take a moment to check out what they have to say and subscribe for more.

For more on Techmeme's sponsorship program, please see this earlier post.

Appreciation for Port 25, Channels 9&10 and Zoho

In May Techmeme was again sponsored by Port 25, Channels 9&10, and Zoho. If you haven't already, please check out what they have to say:

For more on Techmeme's sponsorship program, please see my earlier post.

Direct links now in "New Item Finder"

A small interface change: you can now reach articles and blog posts listed in "New Item Finder" (aka "Latest News Finder") by simply clicking on the title. No longer is a two-step process (click "Find", click headline) necessary. These new direct links aren't underlined, but are highlighted if you hover over them.

For those unclear on the original point of "Find": it uncovers additional context for a new item, e.g.: Are there "Related" articles? What "Discussion" items are attached? Is it near the top of the page? Still, popular demand made a direct shortcut imperative.