opensourceinfo

Getting http/ftp(s)

Fetching Http products: curl - http://curl.haxx.se/ wget - http://www.gnu.org/software/wget/wget.html snarf - http://www.xach.com/snarf/ pavuk - http://www.idata.sk/~ondrej/pavuk/ fget - http://www.feep.net/fget/ fetch - CURL: cURL is having an executable and also library for integration in to many platforms/languages. Stumbled on cookie related issue with curl... This is on the top priority (first item) in the to do list. CURL currently http://curl.haxx.se/docs/todo.html Introduce an interface to libcurl that allows applications to easier get to know what cookies that are received. CURLINFO_COOKIELIST to get a curl_slist with cookies (netscape/mozilla cookie file formatted), and CURLOPT_COOKIELIST to set a list of cookies (using the same format). http://curl.haxx.se/mail/lib-2004-12/0195.html WGET: gets me large files faster than IE/Firefox - using all the available bandwidth. I can also get reccursively - basically mirroring simple static html page domains. There are other small programs on many languages including perl/C to do this task.

posted by Pine Car @ 6:47 PM