FeedRank® - RSSMicro

Google buys PostRank, a social analytics service.

FeedRank Admin — Sat, 04 Jun 2011 17:21:08 GMT

Google acquisition of PostRank is a good news for RSSMicro and its FeedRank® offering. We briefly discussed about the PostRank in 2009 here. Although PostRank targeting social media such as Twitter and Facebook has become very valuable for Google but RSSMicro FeedRank® goes beyond social media analytics and has a far more reaching capabilities that targets the entire web.

Congrats to PostRank team.

RSSMicro Feed Submission Stats - May 2 to May 9 2011

FeedRank Admin — Wed, 11 May 2011 05:55:47 GMT

RSSMicro feed submission service has been a phenomenon which has certainly exceeded our expectations. In recent weeks we have seen a growing demand for RSSMicro feed submission and FeedRank calculation. Below is a quick look at the feed submission stats on RSSMicro website for the week starting May 2 to May 9 2011:

Sources submitted to RSSMicro: 1470 unique sources.

Following top 30 sources account for 85.32% of all the submissions:

1....webs.com.................%8.08333
2....wikispaces.com...........%7.14327
3....wetpaint.com.............%6.85153
4....diigo.com................%6.17669
5....delicious.com............%5.55195
6....insanejournal.com........%5.02445
7....thoughts.com.............%4.72092
8....blog.com.................%4.21111
9....xanga.com................%3.62468
10...blurpalicious.com........%3.61289
11...ordpress.com.............%3.14139
12...gather.com...............%3.11192
13...livejournal.com..........%2.59326
14...bravejournal.com.........%2.53433
15...onsugar.com..............%2.46065
16...url.org..................%2.4135
17...wikidot.com..............%1.96263
18...blogspot.com.............%1.79466
19...hubpages.com.............%1.40566
20...links2rss.com............%1.25832
21...shutterfly.com...........%1.23474
22...oyax.com.................%1.10508
23...rankbuilder.com..........%1.10213
24...bukmark.net..............%0.8045
25...feedburner.com...........%0.78092
26...sosblog.com..............%0.7102
27...html2rss.com.............%0.5216
28...rssmix.com...............%0.49213
29...foxfire-security.com.....%0.46266
30...experienceproject.com....%0.4273
31...OTHER....................%14.68

Have a nice submission...

RSSMicro Team.

HTML5 Server-Sent Events and Worker Processes to paralyze search engines

FeedRank Admin — Sun, 24 Oct 2010 23:16:45 GMT

We published articles here before that talks about AJAX programming and the impact it has on search engines. AJAX allows applications on the web to provide a better and more interactive user experience; it does not require a user to refresh a page or follow a link to see more content, yet the content seamlessly become available by the actions user take on the site. This has a huge impact on the search engine. Although there are some controversial fixes the content seems mainly to remain inaccessible to the search engine crawlers.

Although AJAX adds interactive features to a site, it still follows the traditional response/request architecture of the web. For the content to become available an action by the user is required, this action such as a “mouse click” sends an HTTPRequest and receives an HTTPResponse from the server. In addition to the user action, continues polling or frequent and timely requests can make this possible but the technique is not efficient and consumes a lot of bandwidth and resource on the server.

To fix the problems with the application responsiveness and interactivity, HTML5 offers Server-Sent event and Worker Processes. The Server-Sent event has a profound effect on the web and changes the way we access information on the web. This truly makes the web upside-down with no request/response architecture to follow. To learn more about the HTML Server-Sent event click here. HTML5 Worker Process likewise employs user’s computer power instead of the consuming resources on the server to calculate complex mathematical operations or performs data analysis; the results of these processes would not be available to search engines too.

AJAX programming along HTML5 Server-Sent and Worker Process are search engines nightmares, as we noted before search engines have to use other ways to crawl and index content on the web, it will become increasingly more difficult for search engines to crawl next generation apps that are comprised of AJAX or HTML5 components.

Feed in its RSS or Atom format more likely to become the preferred method for site content distribution and increasingly being crawled by search engines to find new content on the web.

FeedRank Team

USPTO trademark registration completed for FeedRank®

FeedRank Admin — Sat, 02 Oct 2010 18:14:33 GMT

FeedRank^® is now a registered trademark by United States Patent and Trademark Office (USPTO), here is the trademark description that we are very excited about:

Providing online non-downloadable search engine computer software for search of XML-based feeds in the nature of RSS (Really Simple Syndication) and ATOM feeds; Providing online non-downloadable computer software utilizing algorithms for ranking XML-based feeds in the nature of RSS (Really Simple Syndication) and ATOM feeds, based on importance, relevance, quality, and update frequency.

For more information search USPTO website here:
http://www.uspto.gov/trademarks/index.jsp

Visit RSSMicro to calculate your site FeedRank^®
http://www.rssmicro.com/feedsubmit.web

FeedRank, the technology of ranking real-time data on the web

FeedRank Admin — Wed, 24 Feb 2010 20:44:35 GMT

FeedRank role in real-time search
It has been over 3 years since RSSMicro started to analyze the content published on RSS feeds, our primary goal was to identify top publishers and measure the quality of the content as well as their update frequency. The significance of working with RSS or Atom feeds is the access to the most fresh content on the web with some acceptable latency. Today the same concept is being adopted by Google and others to distribute real-time data on the web using RSS and Atom feeds available on content publishers.

RSSMicro quickly took advantage of PubSubHubBub and rssCloud protocols and merged the existing ranking system it developed over the years known as FeedRank with real-time data by implementing these two real-time protocols on millions of RSS and Atom feeds already in its index. The result was significant and promising, for the first time RSSMicro achieved a high level of relevancy on real-time data published on RSS feeds.

FeedRank which is the RSSMicro technology behind ranking RSS and Atom feeds is now taking real-time data to a new level offering a solution to the noisy and irrelevant content published in real-time.

Twitter and Real-Time Protocols
Twitter is known as the leader of the real-time search, it has real-time distribution deals with Google and Microsoft. So many other companies like OneRiot basically follow Twitter to create their own real-time search engines. Unlike others RSSMicro offers a global solution which does not only rely on one or two sources and instead follows the recent developments in real-time technologies and solutions first offered by Google. RSSMicro offers relevant and rich content along images and videos as they are being published in real-time. Currently identified many Twitter and FriendFeeds accounts, news publishers, blogs and many other sources on millions of RSS feeds which have been closely monitored over the years. In theory, any system which relies on real-time protocols should surpass Twitter in volume and FeedRank will be able to help to add relevancy and accuracy to the equation. The trend in which people implement these protocols is going up as we see this on WordPress and some other blog platforms and here we need to scale up the FeedRank processes so that we can follow the growth we anticipate in the near future.

Twitter: Real-Time Content vs Public Opinion
As we see the growing trend in using real-time data distribution technologies by many news publishers, blog platforms and businesses, one question remains unanswered and that is what would be the role of Twitter in maybe 2 or 3 years from now?

Twitter is a 140 characters messaging service which has limitation in posting images, videos, HTML content and more, it seems likely that if people stay loyal and use Twitter and enjoy tweeting about their status and some other stories that find interesting, Twitter will become the source of public opinions and trends on the web rather than real-time content unless it takes a sharp turn and re-defines its messaging and posting service and creates a full-featured content publishing tool, at that point it would be more like another blog or messaging platform that compete with other tools already available on the market. However the role of being the source of public opinions and trends will remain the greatest asset of Twitter as millions of loyal users tweet on regular basis about current events and stories.

Find featured RSS feeds on RSSMicro here:
http://www.rssmicro.com/featured_rss_feeds.web

FeedRank on Twitter:
http://twitter.com/FeedRank

About FeedRank:
http://www.rssmicro.com/FeedRank.web

FeedRank Team.

Why OneRiot is an unsuccessful business model in real-time search market.

FeedRank Admin — Fri, 19 Feb 2010 18:36:31 GMT

One of my friends pointed out to the recent news from OneRiot, saying that their business is gaining ground and that they are becoming a big player in real-time search market, here is the original news on TechCrunch:

http://techcrunch.com/2010/02/18/oneriots-new-realtime-search-api-served-up-with-a-side-of-revenue/

Here are some thoughts on this story:

It may seem strange to some people but I do not see any momentum on the OneRiot side, 97% of their search (momentum) coming from API, and here is the problem: free API search alone is not a good indication of a healthy business model, OneRiot is making high volume free API search calls as the last resort because:

Consumers don't see much value to their search comparing to Twitter search results.

OneRiot traffic for the past 6 months is downward, despite a good coverage by TechCrunch and venture funding.

Lack of innovation, sticking to very limited sources, I am assuming 90% are Tweets, 8% are Diggs, 2% other posts, no real-time protocols available for Twitter and many more...

Businesses use free API when they see lack of consumer demand for their product in an attempt to win some traffic and some market share. OneRiot lack of consumer demand shows that they are basically an unsuccessful business model using the last shot in an attempt to make their product appealing to some market leaders. A successful business first creates a unique and competitive product, builds solid consumer demand on top of it and as the demand goes up and to address the exponential growth creates the API for the partners not VISE VERSA.

OneRiot is mostly a duplicate work of Twitter trying to pretend a large/important real-time player ready to be bought. Frankly, I do not see OneRiot (as its current state) to be a real-time search engine. Their real-time search functionality is Twitter based without any chance to expose recent developments in real-time search technology simply because Twitter doesn't support PubSubHubBub or rssCloud protocols. Despite all that RSSMicro is a working example of recent developments in real-time search first introduced by Google and other marker leaders using PubSubHubBub and rssCloud, RSSMicro opens completely new possibilities to a decentralized real-time search, showcasing relevancy and accuracy of the search results by using its proprietary FeedRank algorithm.

Cheers!

Google subscription to RSSMicro PubSubHubBub enabled feeds

FeedRank Admin — Sun, 07 Feb 2010 21:00:59 GMT

All RSSMicro internal feeds are PubSubHubBub enabled, subscribers can take advantage of RSSMicro hub and receive RSSMicro feed updates in real-time. Checking the server logs we have noticed a relatively new PubSubHubBub subscription service started at https://pshb-service.appspot.com/ we couldn't find any information on the web regarding the nature or the source of this new service, however we have all the reasons to believe that this is in fact Google subscription service to PubSubHubBub enabled feeds. The service has implemented HTTPS protocol using Hub.secret parameter for secure server to server connection.

RSSMicro hub has been able to verify all the secure subscription requests coming from the source and pushing content to the subscriber's address above.

I will update this post once I find more information about this particular subscriber.

FeedRank work space grows as Google Reader adds custom feeds to the sites without a feed.

FeedRank Admin — Wed, 27 Jan 2010 01:41:02 GMT

Google announced a new feature on Google Reader, this time website owners without a feed can use Google to generate a feed for their sites. They can add the feed to the site and let users enjoy updates by subscribing to the feed. This work can be done in a couple of minutes.

http://googlereader.blogspot.com/2010/01/follow-changes-to-any-website.html

It is important to note that the significance of this work is not the reader ability to notify you with the site updates but is the ability to create a feed using Google infrastructure. The feed can be added to the site and being updated outside of Google reader. You can see a sample Google generated feed for a site without a feed here:

http://www.google.com/notificationservice/webchanges/webfeeds/1451785850960242912

This would expand FeedRank to potentially all content on the web. More and more content would be FeedRank enabled as feed become the standard mechanism for distributing site updates on the web. In other words, if someone wants to deal with updates on the web they better use a feed. This has a huge impact on the web as we know it.

For real-time search players this could also be a turning point, feed will have the most recent updates on any website and it is the best candidate for real-time search engines, there is no better mechanism that can support real-time data distribution while keeping the decentralized nature of the web.

This means more work for us to do here in RSSMicro, and if RSSMicro can show how FeedRank works on 4 Million feeds, it sure can do it for 400 Million feeds and more.

Cheers!

RSSMicro to introduce Real-Time Image and Video Search

FeedRank Admin — Mon, 25 Jan 2010 19:49:25 GMT

The real-time data distribution and search has come to a new frontier in RSSMicro, for the first time it introduces real-time image search and real-time video search along real-time search in articles and posts. All search capabilities will soon be available through an API for all search partners.

On RSSMicro revamped home page users will be able to search for images and videos as they are being published or distributed on millions of RSS feeds. This creates a tremendous opportunity for news editors, bloggers and consumers to search and find relevant images and videos pertaining to current news and events.

You can search RSSMicro for real-time images using the sample link below:
http://www.rssmicro.com/?q=Haiti&f=1

Real-time video search on RSSMicro:
http://www.rssmicro.com/?q=Haiti&f=2

You can send your comments to info[at]rssmicro.com

RSSMicro to launch PubSubHub server, joins real-time data infrastructure

FeedRank Admin — Mon, 04 Jan 2010 21:06:30 GMT

RSSMicro is happy to announce its PubSubHub server at: http://pubsubhub.rssmicro.com

RSSMicro real-time PubSubHub along FeedRank technology brings RSSMicro to the forefront spot in real-time data delivery and search systems with the most relevant content available in its index.

Feel free to ping the hub if you are a publisher or subscribe to get the most relevant content in real-time.

More about PubSubHub protocol here.

RSSMicro completes the integration of rssCloud and PubSubHubBub real-time protocols into FeedRank

FeedRank Admin — Thu, 24 Dec 2009 01:00:43 GMT

Today RSSMicro completes the integration of rssCloud and PubSubHubBub two real-time data distribution protocols into its proprietary ranking system for RSS and Atom feeds.

RSSMicro is the leading RSS feed search engine, since 2006 it has been working on RSS feed discovery as well as ranking and indexing millions of feeds in its database using FeedRank algorithm. Today by integrating two leading real-time data distribution protocols it has achieved a milestone in indexing relevant information in real-time. RSSMicro API partners as well as individuals can enjoy relevant articles, images and videos in real-time. For API information contact RSSMicro at info[at]rssmicro.com

Follow real-time news on Obama here:
http://www.rssmicro.com/search/Obama

Follow RSSMicro on Twitter:
http://twitter.com/RSSMicro

Google starts to crawl RSS feeds, exploring PubSub protocol

FeedRank Admin — Mon, 02 Nov 2009 18:46:51 GMT

RSSMicro has been working on crawling, ranking and indexing RSS feeds since 2006, the idea of real-time data discovery and data distribution has been the primary goal of RSSMicro and now it took 3 years for Google to realize the power of RSS feeds. It apparently starts to crawl RSS feeds while exploring the use of PubSub protocol.

This is a big news for RSSMicro since it reiterates that we are on the right track and as the leader in RSS feed crawling and indexing along innovative FeedRank technology, RSSMicro has a huge potential to be a very competitive player in this area:

http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html

Congrats RSSMicro on this news.........

News Aggregators

FeedRank Admin — Wed, 28 Oct 2009 22:37:52 GMT

Few comments about traditional news aggregators including Google News:

1- They almost all have their own RSS feeds which means the updates are normally available to RSSMicro.
2- The timing is based on the indexing not the publishing time, there is sometime a big latency between the time article has been published and indexed.
3- They are traditionally limited to few thousands of news sources, they do not count in the content from social medias.
4- Since the competition is high and the technology is limited they tend to compete on UI or other services irrelevant to real-time data indexing and delivery which makes other services like Twitter very popular.

On another note: wondering why some websites are well treated by Google for maximum exposure:

http://www.google.com/search?hl=en&source=hp&q=site%3Adaylife.com&aq=f&oq=&aqi=

Wondering why Google indexes so many pages on DayLife while it is not an original content site? is Google biased?

Twitter vs RSSMicro in real-time search

FeedRank Admin — Mon, 26 Oct 2009 20:16:13 GMT

A note:

Technically if a system can index news stories faster than those people who tend to gradually share on Twitter, your system will always be one step ahead of the crowd. The real-time search on Twitter is the real-time chatter around certain stories not the actual content, that means there will always be a latency. It looks like Twitter will serve better as a real-time public opinion.

Here is a sample among many:
(the search query is picked as a sample trending topic at the time of writing this note)

22 minutes time period for latest 15 tweets:
http://search.twitter.com/search?q=Ivanka+Trump

3 hours time period for latest 20 "controlled" articles comprised of images, summaries and "two-dimensional related keywords":
http://www.rssmicro.com/search/Ivanka-Trump

RSSMicro real-time search is not yet fully implemented and considering the resources available to both Twitter and RSSMicro, it seems reasonable to believe that Twitter does not have a significantly higher edge over other content distribution methods and is not a final solution to real-time search.

RSSMicro Feed Submission Stats: Oct 16 - Oct 23

FeedRank Admin — Fri, 23 Oct 2009 20:41:55 GMT

Here is the RSSMicro feed submission stats on Oct. 16 thru Oct. 23

Submitting Countries:

United States
Philippines
India
United Kingdom
Australia
Canada
Indonesia
Singapore
France
Malaysia
Slovenia
Ukraine
Spain
Thailand
Netherlands
New Zealand
Serbia
Germany
Italy
Argentina
Belgium
South Africa
China
Mauritius
Russian Federation
Egypt
Romania
Saudi Arabia
Panama
Cyprus
Pakistan
Japan
Peru
Mexico
Saint Vincent and the Grenadines
Kazakstan
Brazil
Nigeria
Vietnam
Sweden
Mongolia
Turkey
Czech Republic
Ireland
Iran, Islamic Republic of
Israel
Hong Kong
Palestinian Territory, Occupied
Greece
Poland
Cambodia
Portugal
Latvia
Syrian Arab Republic
Lithuania
Croatia
Norway
Kuwait
Macedonia
Bangladesh
Lebanon
Tunisia
Morocco
Jamaica
Bahrain
Denmark
Uzbekistan
Algeria
Iraq
United Arab Emirates
Belarus
Yemen
Costa Rica
Switzerland
Jordan

3092
731
710
348
184
180
109
73
66
58
55
54
47
46
40
36
32
31
27
25
24
19
19
18
16
15
14
13
13
12
11
11
9
9
9
9
8
7
7
7
6
6
6
6
5
5
5
4
4
4
3
3
3
3
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1

Submitting Hosts:

html2rss.com
rssmix.com
delicious.com
wordpress.com
blogspot.com
feedburner.com
feedage.com
vox.com
bloglines.com
soulcast.com
livejournal.com
ning.com
onsugar.com
xanga.com
cityeventscalendar.org
blog.com
zimbio.com
webs.com
nexo.com
easyjournal.com
insanejournal.com
blogsome.com
sosblog.com
zoomshare.com
jimdo.com
tumblr.com
bookmarktracker.com
squidoo.com
wetpaint.com
mister-wong.com
hubpages.com
atwiki.com
mixx.com
tripod.com
ezinearticles.com
feedity.com
jumptags.com
terapad.com
zoomblog.com
blogger.com
multiply.com
igooi.com
wikispaces.com
blog.friendster.com
blogetery.com
edublogs.org
blogsarena.com
oyax.com
ya.ru
propeller.com
twitter.com
xfruits.com
blog.ru
page2rss.com
posterous.com
leisurearts.com
blogs-blogs.com
shoutem.com
bravejournal.com
identi.ca
easydigitalsales.com
frozenshoulderblog.com
cleancolonoxypowderdetox.com
streetmavens.com
badrss.com
catechnologiesllc.com
homesolarenergypanelkits.com
adhesivecapsulitisblog.info
yahoo.com
news-floor.com
google.com
unrestrictedminds.com
goarticles.com
bigcontact.com
friendfeed.com
blogfreehere.com
hawaiianhealinghooponopono.com
remedythecauseofbadbreath.com
users.api.live.net
youtube.com
crochetsoiree.com
aquarist-classifieds.co.uk
youare.com
adhesivecapsulitisblog.com
3steps.com
naturalhemorrhoidsremedyrelief.com
3womenbodyscrub.com
koornk.com
org.uk
fireplaceaccessorieshome.com
payday-loan-guides.com
review2bonus.com
bloggista.net
youareamazing.com
diffbot.com
pilzestore.com
feedkiller.com
rapidfeeds.com
personalinsurancetips.com
walkingshoesonline.com

662
396
269
226
215
200
200
144
130
122
122
114
106
101
100
97
96
89
87
82
80
76
74
72
70
70
65
62
58
57
52
51
51
50
48
47
46
46
43
43
42
36
36
32
31
30
28
28
28
20
13
12
11
10
10
10
10
10
10
9
8
8
8
8
7
7
7
7
7
7
7
6
6
6
6
6
6
6
6
6
6
5
5
5
5
5
5
5
5
5
5
4
4
4
4
4
4
4
4
4

FeedRank vs PageRank

FeedRank Admin — Tue, 13 Oct 2009 00:19:19 GMT

Few words regarding FeedRank on RSSMicro:

FeedRank is neither an indication of link popularity nor a traffic or usage report but a computational algorithm that is patented or trademarked for quality control on RSS feeds. RSSMicro analytic and ranking services could be prioritized or calibrated for paid customers. Same as what quality control services do in other industries. I can imagine a time in which people buy and sell blogs, bloggers and other "publishing points". People in the business of buying and selling these also would be very interested in FeedRank detailed history for the publisher. There are very interesting criteria that could be added to the FeedRank algorithm for maximum accuracy.

PageRank does not indicate the freshness of the articles, in fact the basics of the PageRank algorithm results in a better rank for the older articles since older articles tend to accumulate more inbound links over time. FeedRank has its roots in the likelihood of the source to publish fresh content.

PageRank only applies to individual pages while FeedRank has a broader scope that applies to the entire site makes it a better indication of the worthiness of the site.

I will discuss more on this topic......

RSSMicro Feed Submission Stats

FeedRank Admin — Sat, 19 Sep 2009 00:33:24 GMT

Feed submission and FeedRank calculation on RSSMicro has been a growing trend in the last three months, we are seeing many submissions from all around the globe. This unique system of feed validation and instant ranking results tailored to individual feeds proved to be very helpful for bloggers, marketers, individuals and businesses. Later in this blog I will discuss steps we take before adding a new feed to the RSSMicro index.

Here is a break down of various RSS feed sources that have been submitting to RSSMicro for FeedRank results within the last week (Sep. 11 - Sep. 18 2009):

Submitting Countries:

United States
Philippines
United Kingdom
India
Canada
Australia
Malaysia
Thailand
New Zealand
Singapore
China
Germany
Morocco
Ireland
Romania
Indonesia
Spain
Netherlands
Saudi Arabia
Slovenia
France
Argentina
Egypt
Norway
Bulgaria
Russian Federation
South Africa
Portugal
Italy
Ukraine
Chile
Mongolia
Israel
Pakistan
Serbia
Jordan
Palestinian Territory, Occupied
Mexico
Sweden
Belgium
Uruguay
Anonymous Proxy
Japan
Hong Kong
Tunisia
Kazakstan
Vietnam
Switzerland
Syrian Arab Republic
Turkey
Iraq
Venezuela
Luxembourg
Kuwait
Austria
Brunei Darussalam
Denmark

Top 100 Submitting Site:

html2rss.com
blogspot.com
feedburner.com
delicious.com
xanga.com
vox.com
wetpaint.com
livejournal.com
wikispaces.com
blogger.com
ning.com
bloglines.com
webs.com
blog.friendster.com
tripod.com
hubpages.com
squidoo.com
soulcast.com
tumblr.com
mixx.com
diigo.com
rssmix.com
feedity.com
jumptags.com
oyax.com
goforstore.com
feedage.com
multiply.com
mister-wong.com
propeller.com
holidayfrancedirect.co.uk
unrestrictedminds.com
users.api.live.net
feedraider.com
api.ebay.com
yahoo.com
newsusa.com
igooi.com
blogsarena.com
ponyfish.com
googlekeywordsoftware.com
3steps.com
bravejournal.com
zimbio.com
google.com
ourbottlelabels.com
cdlabelall.com
li8ht.net
xfruits.com
bookmarktracker.com
mylabelmakers.com
goodweddinglabels.com
saleshoes.org
blog-newz.net
friendfeed.com
twitter.com
journalhub.com
insanity-workout.com
catastrophicinjury.com
quizilla.com
winelabelsmaker.com
agencymarketing.net
goarticles.com
articlesbase.com
blogsome.com
blogdrive.com
weebly.com
searchles.com
iblogme.com
pannipa.com
youtube.com
blogtalkradio.com
connotea.org
articleshmarticle.com
scribd.com
blogs-blogs.com
opera.com
wordpress.com
tischler-im-netz.de
gle.cc
ask.com
page2rss.com
inewblog.com
lawyers.com
badrss.com
thebossbusterjv.com
ideamarketers.com
tviexpresstravel.info
yolasite.com
hmztwsl.com
stumbleupon.com
reddit.com
ernestocortazar.net
mybloglog.com
digg.com
bissellsteamcleaner.net
articlealley.com
simpy.com
bloogs.eu

1804
509
257
255
175
146
112
87
69
48
48
44
35
32
26
26
25
23
22
21
17
17
15
15
14
13
11
10
9
9
8
7
6
6
5
5
5
4
4
4
3
3
3
3
2
2
2
2
2
2
2
2
2
2
2
2
2

431
399
223
208
181
157
114
107
102
100
89
88
80
79
79
73
72
68
53
46
43
39
38
37
35
32
30
29
22
22
17
17
16
15
15
14
13
12
9
9
9
9
9
9
9
8
8
8
8
8
8
8
8
7
7
7
7
6
6
6
6
6
5
5
5
5
4
4
4
4
4
4
4
4
4
4
4
3
3
3
3
3
3
3
3
3
3
3
3
3
3
2
2
2
2
2
2
2
2

RSS Feeds, Twitter and Real-Time Search

FeedRank Admin — Mon, 14 Sep 2009 23:15:23 GMT

Twitter has been a growing and very successful business not only as a social networking destination for millions of people but also as an effective solution for real-time search. Recently many have realized the importance and the power of Twitter as a real-time search destination which couldn't be possible before using traditional search engines like Google. The first lesson we learn from this phenomenon is that the real-time search demand is strong and is a growing trend on the web and tools, apps that systematically offer real-time search capabilities are extremely important in terms of capturing a bigger share of search business in the future.

There are three major challenges in real-time search applications:

1- Accessing a live stream of published articles, posts as it happens.
2- Real-time indexing power.
3- Real-time search technology.

The second and third items are in fact technological challenges which can be addressed with appropriate system designs, however the first one is a systematic challenge which requires methods in which one can access the data in real-time. Tweet accumulation in real-time makes it possible for Twitter to create a stream of data that is needed for real-time search. The data stream is primarily available to Twitter and that is a huge advantage for this micro-blogging site as the source of the stream. The indexing power and real-time search technologies differ from one platform to the other but for the most part the custom built applications can be successfully implemented in order to achieve real-time search and by using available database technologies.

Although Twitter real-time search capabilities are unprecedented, it is also limited to number of active users and how diversify the contributing sources are and if they are willing to contribute on daily or probably hourly basis in addition to regular site updates. Currently without any established ranking system the quality of search results on Twitter is also questionable.

Other than self-originated data streams such as Tweets, there are publicly available external data sources that might work the same or in many cases even better than Twitter.

We found RSS feed an excellent candidate as a data source for real-time search systems, it offers more diversified stream of data especially when you realize that all updates on Twitter are also available as RSS feeds. The accuracy of real-time data stream offered by RSS feeds is guaranteed by its well-formatted XML structure. The efficiency of fresh content indexing is huge since the feed for the most part has the freshest content on the web compare to other web entities.

Our estimates show that if a system supports adaptive indexing, RSS feed will produces about 50% fresh content on every unit of indexing. In addition RSS feed indexing is a green technology, that means for indexing same amount of fresh content it requires less computational power and consumes less bandwidth and therefore less energy. There are many other advantages in indexing RSS feeds which I will discuss later in this blog.

It is important to note that all external data streams that are publicly available on the web such as RSS feeds impose a disadvantage because the indexing system is not automatically aware of the updates on the source so it cannot achieve a true real-time indexing and search capability. Although critical, the disadvantage could be minimized on RSS feeds by the fact that RSS feed as a data distribution source are widely adopted and used on the web and are extremely diversified and data available through such streams are many times larger than any private, self-originated streams combined.

In case of RSS feeds it is good to note that a publisher/subscriber protocol has been proposed by some organizations that promises real-time push subscription for RSS feeds, which also works as a basis for real-time search. The missing link in RSS feeds real-time indexing and search is not a far distance reality since many technology companies and organizations feel the urge to standardize a pubsub protocol for RSS feeds.

Finally, real-time indexing and search will not be possible if an effective ranking system is not in place, normally so many junk data comes through data streams that could easily compromise the quality of the content and therefore the real-time search system entirely. FeedRank along adaptive RSS feed indexing system developed by RSSMicro are promising technologies that make the real-time search a reality using one of the most widely used data formats and publicly available data stream on the web.

RSSMicro, an RSS feed search engine

FeedRank Admin — Tue, 01 Sep 2009 20:44:16 GMT

Lets start with a brief history of RSSMicro search engine. The idea of RSSMicro an RSS feed search engine formed in 2005 when AJAX programming techniques started to pick up. The technology was available way before 2005 but the demand to develop applications with interactive features started to grow between 2005 and 2006. AJAX components and code samples were easily available to developers and that helped them to get more friendly and be more creative with the technology, after all users enjoyed these new interactive features and the fact that they do not need to refresh the page or navigate from one page to the other changed user expectations along the entire web development industry. It was a lame and unavoidable functionality of Web 1.0 apps that required a user to refresh the page or navigate form one page to the other to see the updated results.

As expected, the transition to new AJAX programming or Web 2.0 friendly applications didn't happen over night, in fact it is still happening. However, knowing the technology is available and the desire to create more sophisticated user interfaces on the web and the fact that today's web increasingly comprise of more applications rather than static pages changed the entire face of the web, as we know it back in 90's.

What next? Well, every change we make especially on this scale has consequences, some well-established legacy systems won't work this way and as I predicted, these legacy systems might slow down the process but cannot change the course.

What is one the most important legacy systems that won't work properly in Web 2.0? Well, that would be web crawler agents, a legacy web crawler that is mostly used by search engines is the one that can not function properly if it faces an interactive web application in which the content is being generated base on the user activities. the question is do webmasters or site owners prefer to make all content available on multiple pages for the sake of the search engines or to create one app on one page that conveniently interact with user to generate updated content based on user input? No one can answer this with certainty at this time but I do not see any reason to believe Web 1.0 application infrastructure would stop us from adopting a technology that is more advanced in terms of application usability and that offers much better and more enjoyable user experience.

What is the role of RSSMicro here then? Well, it was this very flaw or disadvantage in Web 1.0 content discovery technique and crawling agent that formed the idea of RSSMicro, an RSS feed search engine. If we are going to have problem crawling the Web 2.0 and if we continuously face more and more online apps rather than static pages then on what foundation or by using what standards we can start crawling the Web 2.0 again?

In seems likely that future apps will tend to distribute content through structured XML channels called feeds in RSS format, the most widely used format.

Well, as we know, it is the dynamic nature of the web that millions of people go online on a daily basis to find fresh content and RSS feed is the pinnacle of all the fresh content, it is where you will find most updated content chronologically on many Web 2.0 apps such as blogs, micro-blogs, news publishers and many more.

RSS feed is a Web 2.0 phenomenon that seems likely to dominate Web 2.0 crawling and content discovery process.

So I started RSSMicro to address this very demand for RSS feed search. That also initiated the need to develop an algorithm for ranking, which applies to RSS feeds globally, is accurate, and is based on RSS or Atom feed format specs. The idea of FeedRank formed in 2007.

I will discuss some elements of FeedRank algorithm later in this blog.

You will find more information about FeedRank here:

http://www.rssmicro.com/FeedRank.web

Google Trends	Twitter Trending Topics
chicago bears 2010 schedule	Re-Tweet This If
i believe i can fly lyrics	Eleven
doorman strike nyc 2010	Woo-hoo
what is 420 day	RESPECT IS EARNED
gang starr guru dead	Google Escalates The
marvel vs capcom 3	AYOOOOO RT
dirty thunderstorm	Kylie
sly stone	Eeeh
Timothy mcveigh execution video	Parliament
april 19th in history	Apple Wants Its
high stakes poker season 6 episode 10	I Hate My Face