FeedRank® - Real-Time Search

Facebook News Feed Patent, Wordpress PubSubHubBub Support, and Google real-time indexing, what is FeedRank role here?

FeedRank Admin — Mon, 15 Mar 2010 21:52:29 GMT

Last month some important news came out which were related to feeds, real-time web publishing and indexing. The first news was about a controversial patent that was awarded to Facebook for “Dynamically providing a news feed about a user of a social network”. The patent is huge for many reasons, but most importantly signifies the role of feed as the preferred publishing method on social networks and once again shows how feed is turning into the very fabrics of the web and how businesses are rushing to have more footprints on this very fabric.

In terms of real-time data distribution, PubSubHubBub protocol is now more streamlined with Wordpress and Google itself and more likely others will join this frenzy by implementing PSHB hubs. It seems likely that this trend will change the web and how we publish and consume the data, and as far as apps running on the web, feed will play an increasingly important role for us.

And finally it seems likely that publishers, subscribers, apps, search engines will work more off of RSS and Atom feeds rather than HTML pages and our mission is more demanding and critical at the time while providing a mechanism for ranking RSS and Atom feeds.

Links:

Facebook News Feed Patent:
http://www.allfacebook.com/2010/02/facebook-feed-patent/

Wordpress PubSubHubBub Support:
http://wordpress.org/extend/plugins/pubsubhubbub/

Google Real-Time Indexing:
http://www.readwriteweb.com/archives/google_developing_real_time_index.php

FeedRank, the technology of ranking real-time data on the web

FeedRank Admin — Wed, 24 Feb 2010 20:44:35 GMT

FeedRank role in real-time search
It has been over 3 years since RSSMicro started to analyze the content published on RSS feeds, our primary goal was to identify top publishers and measure the quality of the content as well as their update frequency. The significance of working with RSS or Atom feeds is the access to the most fresh content on the web with some acceptable latency. Today the same concept is being adopted by Google and others to distribute real-time data on the web using RSS and Atom feeds available on content publishers.

RSSMicro quickly took advantage of PubSubHubBub and rssCloud protocols and merged the existing ranking system it developed over the years known as FeedRank with real-time data by implementing these two real-time protocols on millions of RSS and Atom feeds already in its index. The result was significant and promising, for the first time RSSMicro achieved a high level of relevancy on real-time data published on RSS feeds.

FeedRank which is the RSSMicro technology behind ranking RSS and Atom feeds is now taking real-time data to a new level offering a solution to the noisy and irrelevant content published in real-time.

Twitter and Real-Time Protocols
Twitter is known as the leader of the real-time search, it has real-time distribution deals with Google and Microsoft. So many other companies like OneRiot basically follow Twitter to create their own real-time search engines. Unlike others RSSMicro offers a global solution which does not only rely on one or two sources and instead follows the recent developments in real-time technologies and solutions first offered by Google. RSSMicro offers relevant and rich content along images and videos as they are being published in real-time. Currently identified many Twitter and FriendFeeds accounts, news publishers, blogs and many other sources on millions of RSS feeds which have been closely monitored over the years. In theory, any system which relies on real-time protocols should surpass Twitter in volume and FeedRank will be able to help to add relevancy and accuracy to the equation. The trend in which people implement these protocols is going up as we see this on WordPress and some other blog platforms and here we need to scale up the FeedRank processes so that we can follow the growth we anticipate in the near future.

Twitter: Real-Time Content vs Public Opinion
As we see the growing trend in using real-time data distribution technologies by many news publishers, blog platforms and businesses, one question remains unanswered and that is what would be the role of Twitter in maybe 2 or 3 years from now?

Twitter is a 140 characters messaging service which has limitation in posting images, videos, HTML content and more, it seems likely that if people stay loyal and use Twitter and enjoy tweeting about their status and some other stories that find interesting, Twitter will become the source of public opinions and trends on the web rather than real-time content unless it takes a sharp turn and re-defines its messaging and posting service and creates a full-featured content publishing tool, at that point it would be more like another blog or messaging platform that compete with other tools already available on the market. However the role of being the source of public opinions and trends will remain the greatest asset of Twitter as millions of loyal users tweet on regular basis about current events and stories.

Find featured RSS feeds on RSSMicro here:
http://www.rssmicro.com/featured_rss_feeds.web

FeedRank on Twitter:
http://twitter.com/FeedRank

About FeedRank:
http://www.rssmicro.com/FeedRank.web

FeedRank Team.

Why OneRiot is an unsuccessful business model in real-time search market.

FeedRank Admin — Fri, 19 Feb 2010 18:36:31 GMT

One of my friends pointed out to the recent news from OneRiot, saying that their business is gaining ground and that they are becoming a big player in real-time search market, here is the original news on TechCrunch:

http://techcrunch.com/2010/02/18/oneriots-new-realtime-search-api-served-up-with-a-side-of-revenue/

Here are some thoughts on this story:

It may seem strange to some people but I do not see any momentum on the OneRiot side, 97% of their search (momentum) coming from API, and here is the problem: free API search alone is not a good indication of a healthy business model, OneRiot is making high volume free API search calls as the last resort because:

Consumers don't see much value to their search comparing to Twitter search results.

OneRiot traffic for the past 6 months is downward, despite a good coverage by TechCrunch and venture funding.

Lack of innovation, sticking to very limited sources, I am assuming 90% are Tweets, 8% are Diggs, 2% other posts, no real-time protocols available for Twitter and many more...

Businesses use free API when they see lack of consumer demand for their product in an attempt to win some traffic and some market share. OneRiot lack of consumer demand shows that they are basically an unsuccessful business model using the last shot in an attempt to make their product appealing to some market leaders. A successful business first creates a unique and competitive product, builds solid consumer demand on top of it and as the demand goes up and to address the exponential growth creates the API for the partners not VISE VERSA.

OneRiot is mostly a duplicate work of Twitter trying to pretend a large/important real-time player ready to be bought. Frankly, I do not see OneRiot (as its current state) to be a real-time search engine. Their real-time search functionality is Twitter based without any chance to expose recent developments in real-time search technology simply because Twitter doesn't support PubSubHubBub or rssCloud protocols. Despite all that RSSMicro is a working example of recent developments in real-time search first introduced by Google and other marker leaders using PubSubHubBub and rssCloud, RSSMicro opens completely new possibilities to a decentralized real-time search, showcasing relevancy and accuracy of the search results by using its proprietary FeedRank algorithm.

Cheers!

Google subscription to RSSMicro PubSubHubBub enabled feeds

FeedRank Admin — Sun, 07 Feb 2010 21:00:59 GMT

All RSSMicro internal feeds are PubSubHubBub enabled, subscribers can take advantage of RSSMicro hub and receive RSSMicro feed updates in real-time. Checking the server logs we have noticed a relatively new PubSubHubBub subscription service started at https://pshb-service.appspot.com/ we couldn't find any information on the web regarding the nature or the source of this new service, however we have all the reasons to believe that this is in fact Google subscription service to PubSubHubBub enabled feeds. The service has implemented HTTPS protocol using Hub.secret parameter for secure server to server connection.

RSSMicro hub has been able to verify all the secure subscription requests coming from the source and pushing content to the subscriber's address above.

I will update this post once I find more information about this particular subscriber.

FeedRank work space grows as Google Reader adds custom feeds to the sites without a feed.

FeedRank Admin — Wed, 27 Jan 2010 01:41:02 GMT

Google announced a new feature on Google Reader, this time website owners without a feed can use Google to generate a feed for their sites. They can add the feed to the site and let users enjoy updates by subscribing to the feed. This work can be done in a couple of minutes.

http://googlereader.blogspot.com/2010/01/follow-changes-to-any-website.html

It is important to note that the significance of this work is not the reader ability to notify you with the site updates but is the ability to create a feed using Google infrastructure. The feed can be added to the site and being updated outside of Google reader. You can see a sample Google generated feed for a site without a feed here:

http://www.google.com/notificationservice/webchanges/webfeeds/1451785850960242912

This would expand FeedRank to potentially all content on the web. More and more content would be FeedRank enabled as feed become the standard mechanism for distributing site updates on the web. In other words, if someone wants to deal with updates on the web they better use a feed. This has a huge impact on the web as we know it.

For real-time search players this could also be a turning point, feed will have the most recent updates on any website and it is the best candidate for real-time search engines, there is no better mechanism that can support real-time data distribution while keeping the decentralized nature of the web.

This means more work for us to do here in RSSMicro, and if RSSMicro can show how FeedRank works on 4 Million feeds, it sure can do it for 400 Million feeds and more.

Cheers!

RSSMicro to introduce Real-Time Image and Video Search

FeedRank Admin — Mon, 25 Jan 2010 19:49:25 GMT

The real-time data distribution and search has come to a new frontier in RSSMicro, for the first time it introduces real-time image search and real-time video search along real-time search in articles and posts. All search capabilities will soon be available through an API for all search partners.

On RSSMicro revamped home page users will be able to search for images and videos as they are being published or distributed on millions of RSS feeds. This creates a tremendous opportunity for news editors, bloggers and consumers to search and find relevant images and videos pertaining to current news and events.

You can search RSSMicro for real-time images using the sample link below:
http://www.rssmicro.com/?q=Haiti&f=1

Real-time video search on RSSMicro:
http://www.rssmicro.com/?q=Haiti&f=2

You can send your comments to info[at]rssmicro.com

RSSMicro to launch PubSubHub server, joins real-time data infrastructure

FeedRank Admin — Mon, 04 Jan 2010 21:06:30 GMT

RSSMicro is happy to announce its PubSubHub server at: http://pubsubhub.rssmicro.com

RSSMicro real-time PubSubHub along FeedRank technology brings RSSMicro to the forefront spot in real-time data delivery and search systems with the most relevant content available in its index.

Feel free to ping the hub if you are a publisher or subscribe to get the most relevant content in real-time.

More about PubSubHub protocol here.

RSSMicro completes the integration of rssCloud and PubSubHubBub real-time protocols into FeedRank

FeedRank Admin — Thu, 24 Dec 2009 01:00:43 GMT

Today RSSMicro completes the integration of rssCloud and PubSubHubBub two real-time data distribution protocols into its proprietary ranking system for RSS and Atom feeds.

RSSMicro is the leading RSS feed search engine, since 2006 it has been working on RSS feed discovery as well as ranking and indexing millions of feeds in its database using FeedRank algorithm. Today by integrating two leading real-time data distribution protocols it has achieved a milestone in indexing relevant information in real-time. RSSMicro API partners as well as individuals can enjoy relevant articles, images and videos in real-time. For API information contact RSSMicro at info[at]rssmicro.com

Follow real-time news on Obama here:
http://www.rssmicro.com/search/Obama

Follow RSSMicro on Twitter:
http://twitter.com/RSSMicro

ActivityStream extension and real-time data formats.

FeedRank Admin — Mon, 21 Dec 2009 19:02:32 GMT

Came across this article:

http://www.readwriteweb.com/archives/cliqset_activity_streams_api.php

Here are some thoughts:

ActivityStream extension to RSS and Atom format (mistakenly referred to as a format in the article) and the notion behind standardizing or normalizing "activities" is a good idea however there are two important things to think about:

1- How much interest is there for every single activity on social networks? people sometimes show little interest in the post itself yet how they might be interested in a feed entry for adding an image, a feed entry for changing background color, a feed entry probably for adding new friend. The real-time search is already saturated with content that adding all these data to the stream doesn't make sense, at least for now. These type of activities can be used for some customized end user applications that serve a special purpose not real-time search to my view. I can think of a celebrity that is also active across multiple social networks (which is very rare), and people want to know every single activity from him or her in real-time (still very rare).

2- Data formats have already been standardized, that was their point in the article (super highway coast to coast), the standard format is Atom or RSS. Other data can be attached as an extension (a new namespace) to these two formats to make it richer. This has been happening since the creation of RSS and Atom. What Cliqset is doing is a repetitive action by duplicating feeds that already carry ActivityStream extension in their feeds (if there is any). Those social networks who have such API for activities and willing to make it public already have ActivityStream attached to their feeds, not sure what the purpose of Cliqset is in here and what value it brings to the table.

Bottom line, some of these ideas come from lack of understanding of the existing technologies, maybe it is just the author.

Now the question is which format is more capable to adopt new extensions, to my view it is the RSS that is even richer than Atom, RSS feeds can add Atom namespace and use its elements while Atom feeds can't use RSS elements as a namespace because RSS deosn't have a namespace.

Yahoo joins the real-time search parade

FeedRank Admin — Thu, 10 Dec 2009 19:48:23 GMT

The good news is it is again Twitter not OneRiot, it would be a bad decision if people use OneRiot while Twitter public API is available to everyone with no restriction. Another good news is that it seems everyone's best shot is at Twitter, they are short in ideas when it comes to real-time search (at least for now).

http://news.cnet.com/8301-30684_3-10412712-265.html

Google and PageRank

FeedRank Admin — Wed, 28 Oct 2009 23:27:09 GMT

A note on Google and PageRank:

Google is biased at least partially, few years ago they found out and admitted on their blog that PageRank no longer is effective. It could have exhausted even their resources to calculate PageRank regularly for trillions of pages along many quadrillion page intersections, they probably didn't put into account or expect the web exponential growth in recent years. This cumbersome process along the extensive work of SEOs made them think twice and gradually changed the algorithms to something extremely non-scientific and experimental which surprisingly worked as good as the original idea. However in this process some websites happily accepted the results their business thrived and they prosper, some eliminated or got an unfair treatment and never came back. Since then no one knows what exactly Google is doing and that is the concern since nobody knows what portion of the web is missing or hidden from users.

Twitter is a successful business model not for the complex algorithms or UI or anything in that nature, it is successful because "everything" chronologically appears on the search with almost no manipulation. I believe it was the most important factor that made Twitter so popular and let it capitalize on Google manipulative system.

Good to note that the PageRank original concept is completely in contrary to the requirements of the real-time search, older an article is, more likely to accumulate external links, therefore PageRank tends to give higher scores to older pages and vise versa new pages have almost no rank.

News Aggregators

FeedRank Admin — Wed, 28 Oct 2009 22:37:52 GMT

Few comments about traditional news aggregators including Google News:

1- They almost all have their own RSS feeds which means the updates are normally available to RSSMicro.
2- The timing is based on the indexing not the publishing time, there is sometime a big latency between the time article has been published and indexed.
3- They are traditionally limited to few thousands of news sources, they do not count in the content from social medias.
4- Since the competition is high and the technology is limited they tend to compete on UI or other services irrelevant to real-time data indexing and delivery which makes other services like Twitter very popular.

On another note: wondering why some websites are well treated by Google for maximum exposure:

http://www.google.com/search?hl=en&source=hp&q=site%3Adaylife.com&aq=f&oq=&aqi=

Wondering why Google indexes so many pages on DayLife while it is not an original content site? is Google biased?

Twitter vs RSSMicro in real-time search

FeedRank Admin — Mon, 26 Oct 2009 20:16:13 GMT

A note:

Technically if a system can index news stories faster than those people who tend to gradually share on Twitter, your system will always be one step ahead of the crowd. The real-time search on Twitter is the real-time chatter around certain stories not the actual content, that means there will always be a latency. It looks like Twitter will serve better as a real-time public opinion.

Here is a sample among many:
(the search query is picked as a sample trending topic at the time of writing this note)

22 minutes time period for latest 15 tweets:
http://search.twitter.com/search?q=Ivanka+Trump

3 hours time period for latest 20 "controlled" articles comprised of images, summaries and "two-dimensional related keywords":
http://www.rssmicro.com/search/Ivanka-Trump

RSSMicro real-time search is not yet fully implemented and considering the resources available to both Twitter and RSSMicro, it seems reasonable to believe that Twitter does not have a significantly higher edge over other content distribution methods and is not a final solution to real-time search.

A Real Real-Time Search Engine

FeedRank Admin — Thu, 15 Oct 2009 23:24:41 GMT

I came across these two articles:

http://www.trendwatching.com/briefing/

http://www.readwriteweb.com/archives/top_50_real-time_web_companies.php

As you can see there are so many real-time search engines out there, but tell me if I am wrong, seems almost all are new services working off Twitter. It is obvious, no one can claim such a functionality unless they own the data (lots of data) or index so much fresh content at any given time that can simulate a real-time search. Even Google doesn't claim to have produced such a critical functionality.

PubSub RSS extension is a publisher/subscriber protocol that will address this critical functionality on RSS feeds and here in RSSMicro we started collecting these information on RSS feeds. If implemented in a large scale (requires all publishers to adopt the technology) it will produce results that if not surpass Twitter it will effectively compete with Twitter in real-time search:

http://code.google.com/p/pubsubhubbub/

Twitter Real-Time Search Deal With Google And Microsoft

FeedRank Admin — Fri, 09 Oct 2009 18:43:05 GMT

This just gets better and better, there are few things we can learn here:

The market for real-time search is very strong and is very demanding.
Even the big players are struggling in this area as their current processes do not meet the requirements of the real-time search.
Although it may address the real-time search in short term but relying on one source (no matter how big is the source) is in contrary to the concept of the real-time search.
They have to have these deals with FriendFeed, eBay, Craiglist, Amazon, AOL, Typepad, Livejournal, Digg, Delicious, all web forums, discussion boards... and many more who publish user generated content to meet the real-time search demand in global scale.
Real-time data and corresponding search capabilities on the web can not be dominated or owned by one or two sources, it is against the non-centric nature of the web.
Other mechanisms and standards that are publicly available and follow the non-centric nature of the web will eventually replace the centralized, privately owned and operated real-time search processes.

Those who can successfully demonstrate a non-centric, comprehensive real-time search capability that is built on top of existing technologies and standards are all-time winners.

Original Post:
http://paidcontent.org/article/419-twitter-in-talks-with-google-and-microsoft-about-search-deal/