by
6 Comments

Your Own Computer Can Spider the Web, Thanks to MyLiveSearch

posted on May 29, 2007


Oh boy, with the buzz MyLiveSsearch is getting today, expect to see the company snapped up by Google, Yahoo or Microsoft in the next few months. Why?

MyLiveSearch claims to have finally figured out how to tap into the “hidden web” – all the stuff that is too fresh or too deeply stored for other search engines to discover.

It works through a small browser plug-in. The search terms are put through Google, or other indexed search databases, but those results are treated as “starting points” alongside the user’s bookmarks and other popular web hubs.

From there, the live search takes over, crawling through hundreds of web pages connected to those starting points in search of more information relevant to the search.

My first question was, can indexing the live web scale? But then I realized, it doesn’t have to. MyLiveSearch is not looking to index all of the hidden web, just the parts that an individual user may have an interest in. With faster computers and fatter broadband connections, MyLiveSearch is entering the space at just the right time – you couldn’t have done this just a few years ago.

While Google wouldn’t comment on reports that it has met twice with MyLiveSearch creators, Rob and Mark Gabriel, that didn’t stop Rob Gabriel from adding fuel to the fire.

“This technology could be snapped onto any of the major search engines and improve them.”

If it does exactly what is says on the can, then you should fully expect one of the big engines to come knocking.


Related Posts

Zemanta
  • http://www.brianchappell.com Brian Chappell

    What something new and interesting in the search space? Thats about as common as paris hilton sitting in jail next to you.

    Im with you andy, this kind of platform will get swiped up quick by one of the big three.

  • http://www.u-g-h.com Owen Cutajar

    Wow, can you imagine the power of this thing? I mean, if it were to report back to a central server as to what it found and what path the user took through the reported results (to guage relevance), this could take search to a whole new level!

  • http://www.markbarrera.com Mark Barrera

    I am somewhat skeptical of this offering if they are still relying on the large hub sites and the search engine results.

    I do think it will be interesting though and hope to get a peek at their service sometime soon.

  • http://charisma18.com aaron

    This service hasn’t even launched a private beta yet. Why is it being covered?

  • http://www.marketingpilgrim.com Andy Beal

    @aaron – probably because Google has visited the company twice – that’ll get people talking every time. ;-)

  • Darren

    I was one of the first software engineers to work on the project. For 2 years now, up until a couple of weeks ago the mylivesearch test website was running. In a nutshell, Robs browser acts like a ’spider’ so when you search it searches live and matches any keywords that you have typed. Any matches that the browser finds is then automatically updated into the mylivesearch database so other people who go to the mylivesearch website can access the same info for the same keyword. Just think if millions of people are using the browser and searching, within a very short time the whole net could be mapped. Also the mylivesearch database has its own spiders constantly crawling the net updating all returned keyword searches. As soon as any mapped page is updated, it is also updated within the mysearch website.
    The smart thing about the search engine is that it can map a whole website. At this time ’spiders’ start at the index page and follow links to other pages. This search software scans the whole website and maps every single file. Even files that do not have any links to them through other pages. So that means if you store files on your server, ie; like a video file, that is not linked to any page, the MyLiveSearch will still find it and map it if it is relevent with the key word you are searching for. ie; you have a video on your server called Paris_Hilton_Sex.wmv which is not linked to any page and using the MyLiveSearch Browser search for Paris Hilton, it will scan the website, find the title which relates to the keyword and map the link to the MyLivesearch database for everyone to see.

    I actually have a program which uses this facility that is the fore runner to the search browser which we used in the early days to scan other websites. For example, ninemsn stores a lot of their video content from their channel nine television shows on their server. Normally you would not know that they are there because there is no link. They probaly used it for temp storage until programmers needed it for their shows. We use to watch them before they even went to air. You would be amazed with what stuff people store on their servers! This software can find anything, pictures,video,text,flash, you name it. The CIA and FBI would love this software!

Filed Under: General