I want a have spider modified or built which ever is easier. You can use existing opensource libraries or anything, it doesn't matter as long as it acheives the tasks.


I want to be able to run the spider as an applet and from the command line so that i can be execute as a cron job.

The spider must be able to accept command line arguments eg.


main(String args[]) { String var = args[0]}

and the applet should have a simple gui.

The spider should be able to take in the domain name and crawl that domain only unless the option is choosen for the spider to leave the domain. It must have the option to re-index if html page has changed.

It should check header status of a page and does not index unless the page is available, so status 200 etc.

--------------- Specs ---------------------------

Spider gets full html page contents


if the html tag i want to check for (eg <object></object>) is found then


Parse all html tags


get : array of tags i specify


example String getTags[]={"title","keyword"}

if keyword is empty/missing and description or title not empty then


split description at every word


return array of keywords limit to 250


else if title empty


attempt to extract keywords from html body up 250 words


if the html tag i checked for is not found then do not parse the page just get all the links from the page and continue crawling.

the crawler need to be able to return the values of html tags and their attributes that i specify.

I'd like the values returned to be in an associative array/map so that

myObject['title'] will contain the title


myObject['keyword'] will contain an array of keywords


myObject['tagName']['Attribute'] will get the attribute value of the html tag example


myObject['embed']['src']

Lastly i want the data to be inserted/indexed in my mySQL database but only if the html tag i checked for was found.

Please make sure you read and udnerstand the reqiurments. This will be integrated into one of my projects and it needs to be built according to my specs.

The spider can be a modification to the one found here


http://www.developer.com/java/other/article.php/1573761/Programming-a-Spider-in-Java.htm

or here


http://www.javaworld.com/javaworld/jw-11-2004/jw-1101-spider.html

or anything from the net


http://www.google.com/search?hl=en&source=hp&q=java +web+spider&aq=0&oq=java+web+spid&aqi=g2

or if you already have a class or library that does this.

It doesn't matter i just want a spider customized to do the above.

Escrow payment only... No automated bids please.