Ubuntu :: Download Entire Site With Wget From Localhost Proxy?
Dec 24, 2010
i want to download android developer guide from google site but code.google is forbidden from my country i want to use wget to download entire android dev guides with freedom( proxy ) that i set in firefox these for open forbidden sites ( 127.0.0.1 port:8080 ) i use this command to download entire site
Code:
`wget -U "Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.7.3) Gecko/20040916" -r -l 2 -A jpg,jpeg -nc --limit-rate=20K -w 4 --random-wait http://developer.android.com/guide/index.html http_proxy http://127.0.0.1:8080 -S -o AndroidDevGuide`
[Code]....
View 4 Replies
ADVERTISEMENT
Mar 14, 2011
i use this code to download :wget -m -k -H URL... but if some file cant be download , it will retry Again and again ，so how to skip this file and download other files ,
View 1 Replies
View Related
Nov 25, 2015
This is the command line switch I am using:
Code: Select allwget -p -k -e robots=off -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' -r www.website.com
For some reason it seems to be downloading too much and taking forever for a small website. It seems that it was following alot of the external links that page linked to.
But when I tried:
Code: Select allwget -E -H -k -K -p www.website.com
It downloaded too little. How much depth I should use with -r? I just want to download a bunch of recipes for offline viewing while staying at a Greek mountain village. Also I don't want to be a prick and keep experimenting on people's webpages.
View 3 Replies
View Related
Mar 29, 2011
How do you instruct wget to recursively crawl a website and only download certain types of images? I tried using this to crawl a site and only download Jpeg images:
wget --no-parent --wait=10 --limit-rate=100K --recursive --accept=jpg,jpeg --no-directories http://somedomain/images/page1.html
However, even though page1.html contains hundreds of links to subpages, which themselves have direct links to images, wget reports things like "Removing subpage13.html since it should be rejected", and never downloads any images, since none are directly linked to from the starting page.I'm assuming this is because my --accept is being used to both direct the crawl and filter content to download, whereas I want it used only to direct the download of content. How can I make wget crawl all links, but only download files with certain extensions like *.jpeg?
EDIT: Also, some pages are dynamic, and are generated via a CGI script (e.g. img.cgi?fo9s0f989wefw90e). Even if I add cgi to my accept list (e.g. --accept=jpg,jpeg,html,cgi) these still always get rejected. Is there a way around this?
View 3 Replies
View Related
Aug 10, 2011
I want to do something simular to the following:
wget -e robots=off --no-clobber --no-parent --page-requisites -r --convert-links --restrict-file-names=windows somedomain.com/s/8/7b_arbor_day_foundation_program.html
However, the page I'm downloading has remote content from a domain other than somedomain.com. It was asked of me to download that content too. is this possible with wget?
View 1 Replies
View Related
Aug 17, 2010
is there a way to force wget to use a specific squid proxy when making connections ? - I use a squid proxy normally, but I need this specific request to go via a different one. I dont have to use wget, I just need a way to test squid's blocking rules by requesting various pages through it, this proxy is not my normally proxy on the network and so I cant rely on wget taking the environment variable.
Also, this is as part of a script, so anything that avoids editing wget config files would be best. - Perhaps curl can do this ? - currently im using the exit code of wget to determine if the connection was made.
View 1 Replies
View Related
Sep 6, 2011
I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs static links.
View 1 Replies
View Related
Nov 4, 2010
I am trying to wget a site so that I can read stuff offline.I have tried
Code:
wget -m sitename
wget -r -np -l1 sitename
[code]....
View 7 Replies
View Related
Dec 21, 2010
can we use recursive download of wget to download all the wallpapers on a web page?
View 5 Replies
View Related
Jan 19, 2010
I want to replicate this small howto (http://legos.sourceforge.net/HOWTO) using wget.However I just get a single file and not the other pages and that file too is not html.
View 4 Replies
View Related
Sep 17, 2009
I was trying to download MOPSLinux from their Russian FTP server, using Firefox-->FlashGot-->KDE-Kget and it kept sitting there for about a minute, then popping up a dialog box asking for a Username & Password to access the FTP site.
I tried the usual anonymous type of login information combinations, to no avail; the box kept reappearing.
Finally for the heck of it, I tried Firefox-->FlashGot-->Wget and presto! It began downloading right away, no questions asked.
This is on Slack64 with the stock KDE installation + the KDE3 compat libs.
Here's the transfer currently going on the Wget window:
Code:
View 6 Replies
View Related
Jul 4, 2011
I am getting more adventurous, wanted to know how to preview sites in progress locally while using Kompozer, surely there must be a way to preview before publishing to the web.
New to Ubuntu with minimal expertise, have built sites with WYSIWYG programs in Windows with a some success, now want another challenge.
View 4 Replies
View Related
Aug 8, 2011
I have developed a site in localhost in my laptop. Here are the specs: Ubuntu 11.04, Joomla! 1.7, Db Ver: 5.1.54-1ubuntu4, PHP Ver: 5.3.5, Web Server Apache/2.2.17
This laptop is connected to a wi-fi network. Task: I like to see this local site from my laptop to other desktop computers in the same network. I understand that I need to do some changes in /etc/hosts what changes I need to do to be able to see my localhost site in the network.
View 2 Replies
View Related
Sep 5, 2010
I am trying to configure local https server but I get this message: "ssl_error_handshake_failure_alert"
I configured my apache and make cert and key files from this article, following section 1B: [url]
I try to add server.crt and ca.crt files in firefox, but it didn't help. I googled and didn't find any information. I try also with 1024bit key.
View 6 Replies
View Related
Aug 31, 2010
I try to download the Tomcat from my LINUX machine with command:
Code:
wget -d -v http://archive.apache.org/dist/tomcat/tomcat-5/v5.5.23/bin/apache-tomcat-5.5.23.zip
I got error as follow:
Code:
[root@s003ap23-test opt]# wget -d -v http://archive.apache.org/dist/tomcat/tomcat-5/v5.5.23/bin/apache-tomcat-5.5.23.zip
Setting --verbose (verbose) to 1
[Code]...
View 4 Replies
View Related
May 20, 2010
When I try to use wget or install anything that uses wget to install (flash player plugin), it fails, as my IT admin blocks direct access to IPs. I see wget takes the domain and resolves it's IP and then downloads (atleast thats how I get it?), and then I get "connection refused". Is it possible to force it use the domain and not the IP?
View 9 Replies
View Related
Feb 21, 2010
I'm trying to download a set of files with wget, and I only want the files and paths "downwards" from a URL, that is, no other files or paths. Here is the comand I have been using
Code:
wget -r -np --directory-prefix=Publisher http://xuups.googlecode.com/svn/trunk/modules/publisher There is a local path called 'Publisher'. The wget works okay, downloads all the files I need into the /Publisher path, and then it starts loading files from other paths. If you see [URL]..svn/trunk/modules/publisher , I only want those files, plus the paths and files beneath that URL.
View 2 Replies
View Related
May 6, 2011
if there is a mirror I could use to download a recent version of Ubuntu (e.g. natty). I'd like to use wget but can't find an address for a mirror.
View 3 Replies
View Related
Jul 28, 2011
I want to try to download an image of the earth with wget located at [URL] which is refreshed every 3 hours and set is as a wallpaper (for whom is interested details here). Wen I fetch the file with Code: wget -r -N [URL] the jpeg is only 37 bytes and of course too small and not readable.
View 5 Replies
View Related
Jul 30, 2011
I have a simple issue that I think can be solved with several different methods. I basically want to create a personal server solution that allows me to do two things:
1.) I want to be able to remotely backup data to my server.
2.) I want to be able to pass traffic through it and use it as a proxy.
I am off to college next year and I want to leave a computer/server back home to do the two things stated above. I was thinking of using an Asus Eee Box PC like this:url
I want a low power reliable machine that will only be used as a remote solution. I won't be hooking up a monitor to it (that is, after I set it up).
It will be on 24/7 for easy access.
I will be accessing this server from a Windows 7 based machine.
I do not mind at all installing Linux on my server, but I am not an experienced coder so I will need software with a GUI that can help me set this all up.
View 3 Replies
View Related
Oct 6, 2010
I'm doing this wget script called wget-images, which should download images from a website. It looks like this now:
wget -e robots=off -r -l1 --no-parent -A.jpg
The thing is, in the terminal when i put ./wget-images www.randomwebsite.com, it says
wget: missing URL
I know it works if I put url in the text file and then run it, but how can I make it work without adding any urls into the text file? I want to put link in the command line and make it understand that I want pictures of that certain link that I just wrote as a parameter.
View 1 Replies
View Related
Mar 6, 2011
I would like to use wget to downlaod file from Redhat linux to my windows desktop , I tried some parameter but still not work , can advise if wget can do download file from linux server to windows desktop ? if yes , can advise how to do it ?
View 14 Replies
View Related
Oct 16, 2010
I have a link to a pdf file, and I want to use wget (or python) to download the file. If I type the address into Firefox, a dialog box pops up asking if I want to open or save the pdf file. If I give the same address to wget, I receive a 404 error. The wget result is below. Can anyone suggest how to use wget to save this file?
View 1 Replies
View Related
Jul 10, 2009
I install Redhat9 and run proxy(squid) on this server. I test with a PC with IE put it proxy ip address any other web site is working fine but not for radio&TV channel online, i found many radio and TV online web site can not start to play after click the link.Please anybody help to advice what should i check on squid?
View 1 Replies
View Related
Mar 7, 2010
i download files from megaupload and hotfile. is there any possibility of making wget download more than 1 file at a time? or do you suggest any other download programme? i have ubuntu 9.10
View 3 Replies
View Related
May 21, 2010
i try to make wget download automatic in startup in ubuntu
View 8 Replies
View Related
Jul 27, 2010
I am trying to download site using wget :$sudo wget -r -Nc -mk [URL] but it is downloading the contents of all directories and subdirectories under the domain :[URL] (ignoring the 'codejam' directory) so it is downloading from links like : [URL]... i want to restrict the download so that wget command should download only the things under 'codejam' directory
View 9 Replies
View Related
Jan 21, 2011
I tried to download some images from google using wget:where wget cbk0.google.com/cbk?output=tile&panoid=2dAJGQJisD1hxp_U0xlokA&zoom =5&x=0&y=0However, I get the following erros:
--2011-01-21 04:39:05-- http://cbk0.google.com/cbk?output=tile
Resolving cbk0.google.com... 209.85.143.100, 209.85.143.101
Connecting to cbk0.google.com|209.85.143.100|:80... connected.
[code]....
View 3 Replies
View Related
Jul 28, 2010
is there a way to actually download the entire fedora repo? i've a friend with dialup who cannot install big things and i want to just downlaod the repository for him. i know ubuntu has some sort of way of doing this. i tried the following:
Code:
sudo yum --downloadonly --downladdir=~/fedora_repo install *.i?86
but this seems to, aside from all sorts of conflicting packages only get the things i dont have installed.
View 2 Replies
View Related
Jun 21, 2010
is it recommended to download an iso file of fedora 13, will the file be destroyed?because i did it twice and it seems not working.
View 6 Replies
View Related
