|
|
URL Parser
It's a valuable tool to collect online
URLs from any website, such as Yahoo. When the root URL of a website
is given, the tool will access the website, read it's HTML code and
retrieve all the URLs. Then for each URL starting with the root URL
(such as www.yahoo.com/software.htm starting with www.yahoo.com) it
repeats the action to retrieve all URLs recursively from the page. All the
retrieved URLs and related information will be saved into a database.
This tool was initially developed using
www.steellinks.com as target
site, which is with nearly 4000 URLs. It took about half an hour, for example, to download all
URLs of a the website
www.steellinks.com into a local database.
|