General :: Using Wget To Mirror A Website And Everything From The First Level Of External Sites
Sep 14, 2010
I need to mirror a particular website (all the pages under that particular domain) any pages (but not whole sites) that the website links to.
How to do this
wget -r --level=inf (or some other variant) will mirror the site.
wget -r -H --level=1 will get all the links (from all domains) to the first level.
Anyone have any ideas on how I could combine these, to get the entire of the main site and one level deep into external sites. I've been banging my head against the manual all afternoon.
View 1 Replies
ADVERTISEMENT
Sep 16, 2010
How can I mirror drives on the hardware level? I have a Sun X2270 server with 4 1 terabyte hard drives and need to mirror two drives.
View 1 Replies
View Related
Nov 4, 2010
I am trying to wget a site so that I can read stuff offline.I have tried
Code:
wget -m sitename
wget -r -np -l1 sitename
[code]....
View 7 Replies
View Related
Jan 21, 2010
After reading a lot of docs, I'm still having problems using wget to download a Centos repo from a mirror. Here's my best attempt so far:
$cd /repos/centos/5.4
$wget -r -nH --cut-dirs=3 -np [URL]
Of course I get all the unwanted index files etc, but I seem to get a lot of other downloads from the mirror, not just their 5.4 directory. It's like it's following other links on the web pages. Maybe I should be using "ftp://" instead of "http://" considering it's an ftp site, but I seem to have connection problems that way.
View 1 Replies
View Related
Mar 9, 2011
I'd like to ask a question on installing software.
I cannot install software using "yum" but I can with the GUI program "Add/Remove Software".
For example, I tried:
It will exhaust all the mirror sites without success.
All the error messages look like:
But I can install "screen.i686" easily from the "Add/Remove Software" GUI and it is very fast.
However, what's puzzling is that I could install some certain packages although I still see the same kind of error messages.
For example,
will work. But in general, "yum" is running very slow and is rejected at every mirror sites.
My settings: I just installed "Fedora 14" (Linux 2.6.35.11-83.fc14.i686) as a virtual machine (using VirtualBox) on a Windows XP (Professional, SP3) host on my laptop (CPU: Intel Core2 Duo), which accesses internet through a HTTP Proxy server since it is behind a corporate firewall.
View 7 Replies
View Related
Dec 29, 2010
I'm trying to have wget retrieve the pics from a list of saved URLs. I have a list of facebook profiles from which I need the main profile picture saved.When I pull such up in my browser with the included wget command I see everything just fine; however, when I do it reading in a file (or even manually specifying a page to download), what I receive is the html file with everything intact minus the main photo of the page (that pages' user picture).I believe I need the -A switch, but I think that is what is causing the issues (because the page is not a .jpg, it's getting deleted).
View 1 Replies
View Related
Nov 30, 2010
Index of /distribution/openSUSE-stable/repo/oss/suse/i586 How can i download packages with wget from that mirror with a string "alsa"?
Such as:
alsa-1.0.23-2.12.i586.rpm
alsa-devel-1.0.23-2.12.i586.rpm
alsa-oss-1.0.17-29.2.i586.rpm
View 9 Replies
View Related
Sep 27, 2010
I'm wondering if its possible to add an external, non-Ubuntu mirror to my mirror server?
We have a few packages which need to be deployed during kickstart (Cinelerra and Eclipse plugins), and I would like to put them into my mirror repo so they are grabbed during kickstart, rather than manually adding them after the system is up.
View 1 Replies
View Related
Sep 1, 2011
Strangest thing...yesterday my 11.04 box lost the ability to access anything external. I can ssh to the box from outside my network, and all other boxes on my network are fine, but this one box can't hit anything external. If I attempt to ping a site it's doing the same resolution, but no packets are delivered.
Up until yesterday it was running fine, and I didn't make any changes to my home network or install any updates. Totally stumped.
Edit: this is a wired connection with a static internal IP.
View 3 Replies
View Related
Jan 13, 2010
I am looking for a solution to mirror a directory on my companys computer that runs in windows to an external ubuntu server on the internet.
View 7 Replies
View Related
Jan 15, 2011
hello everyone, im having a problem when my computer enters in the run level 4 as the default when i start slackware. The strange thing is that it not seems that is a X window problem, it looks like more like a configuration problem in some part of the kde script to initialize the log in, because if i manually start the X service it works fine, i dont know what is the source. Thank you in advance for the help.
View 3 Replies
View Related
Apr 12, 2010
Have just installed Ubuntu 9.10 in Vmware workstation 6.5.1 and unable to connect to external sites, or servers (using NAT). I can ping on host name and IP but can't resolve either host or IP in browser (Receive message the connection has timed out). I can't ping the host (Windows 7 Professional) IP from the guest and vice versa. I use a mobile wireless network card, and my Windows operating systems work successfully in vmware. Currently in my VMNet8 properties I just have Obtain IP address and DNS Automatically (same as host).
View 3 Replies
View Related
May 27, 2010
I have a directory that has a another directory inside it. The top directory is rather redundant since it only contains the one other one. Is there a way to delete the top level directory and have the contents simply "move up a level"?
View 6 Replies
View Related
Jul 29, 2011
What is the best and easiest way to download music from an external website onto Ubuntu? I have both Rhythmbox and mplayer installed, but it's not clear to me how you download music so that it is recognized by those players. Would it be easier to use a terminal for downloading?
View 7 Replies
View Related
Mar 30, 2011
i have a website named [URL]... now i want to access this website using proxy server(squid,or etc...) under my personal server named [URL]...means that [URL]...
View 4 Replies
View Related
Feb 9, 2011
In the past week or so I've noticed some weird network behaviour. I find accessing some sites such as Amazon, Paypal, and Bigstockphoto really slow. Sometimes the page will not load at all. Other sites are fine. The problem sites are not a problem for others on my LAN at home. When I try to open the problem sites, I can see in Firestarter blocked connections coming from 2.1(8/9).xxx.xxx on various ports such as 36007. This only happens for the problem sites. I attached a typical output from firestarter.
This happens with Firfeox or Chrome. Using Ubuntu 10.10
View 4 Replies
View Related
Jun 19, 2011
If a wget download is interrupted (like if I have to shutdown prematurely), I get a wget.log with the partial download. How can I later resume the download using the data in wget.log? I have searched high and low (including wget manual) and cannot find how to do this. Is it so obvious that I did not see it? The wget -c option with the wget.log as the argument to the -c option does not work. What I do do is open the wget.log and copy the URL and then paste it into the command line and do another wget. This works but the download is started from the beginning, which means nothing in the wget.log is used.
View 2 Replies
View Related
Aug 10, 2010
I've been able to kludge a kill script which finds the correct pid for the kdeserver (or gnome server) after my system comes up in run level 5 so I can drop back to run level 3 mode. Lots of experimentation showed me that using telinit 3 and telinit 5 would occasionally leave the video memory in a mess and I would have the black screen of death.
I set the security parameter setting to autologin for me since I am the only user of my machine, but I still have to kludge the default setting under sysconfig (the DEFAULT_WM) under Window Manager to pick a certain window manager, so it takes time to manually switch the desktop.
Right now I can leave the gui and drop back to cli, but painful experimenting showed me that killing the X server is a no no. Right now I kill the kde server, which sends the SIGTERM to the X windows manager, which then figures out that it has to shut down.
Questions: Is there a better way of doing this? Apparently openSUSE figures that we have multiple users logging into the gui desktop, so the gui is always kept running and a login window with the desktop manager option forces the user to login in. With autologin, this never happens, but no choice of desktop is possible on the fly.
Can some type of script be set up to painlessly enable this to happen? And what is the best way of bringing either the Gnome or KDE desktop manager down gracefully? I do get lots of error messages as the system attempts to recover and X shuts down. It appears that apparently the single user with autologin is left out in the cold.
View 4 Replies
View Related
Feb 6, 2010
I want to install a software called TinyOS which is an operating system designed for wireless sensor embedded networks in my account. The problem is it has instructions to install the software as an administrator since i'm not an admin of the department network i can not able to install. Is there any method to install this software as an user level rather than admin level.
View 3 Replies
View Related
May 8, 2010
How can I adjust the levels at which the battery is considered to be critically low?ight now it seems this is set at 5 or 10%. I want to make it 20 or 25%This is for Gnome. I am using Lucid x86
View 1 Replies
View Related
Jan 6, 2009
I want to know what are the advantages and disadvantages for accessing spi(serial peripheral inerface )from kernel level and user level. like methods of doing it, speed ,memeory utilization etc
View 1 Replies
View Related
Mar 29, 2011
suppose i have a website named www.site.net now i want to access this website using proxy server(squid,or etc...) under my personal server named www.anotherwebsite.net:8080, means that www.site.net.
View 1 Replies
View Related
Jan 14, 2011
I dual boot XP and FC14 and have 2 routers. I can connect and ping one of these routers when I'm in FC and I have an IP address I just can't load any websites. When I connect to the other router (my main router) it works fine. When I boot into XP and connect to the problem router I can load pages fine. It's only when I'm on FC14 and connect to the problem router that I can't load pages even though I have an IP and can ping around.
View 1 Replies
View Related
Jul 21, 2010
[code]...
you can find a cutout of an output generated by "iwlist s" command for a cell. Does anybody know what Quality, Signal level and Noise level mean? What is the definition of them? I searched for it and could not find good information on it. What do these Extra:*** fields for? What does it show in the example above? How were those values computed?
[code]...
View 1 Replies
View Related
May 12, 2009
Hi Guys,
here is my scenario:
I am running F10 x86_64 with Nvidia Quadro FX 3400.
I installed the drivers and it worked just fine. No errors generated according to Nvidia log files.
However, while in run level 5 when I try to switch to run level 3 by typing "telinit 3" or "init 3" the X server goes down and I get a blank screen. Same thing happens when I try to switch to a different tty session by <ctrl>+<alt>+F2 (or any F key for that matter).
I also noticed that when I shut down my system I can't see any more the screen where services are shutting down and their status ([OK] or [failed]).
Would appreciate your help on solving this issue.
View 3 Replies
View Related
Jun 17, 2010
Hey Guys, I'am new here and request your help on this:
After installing emacs and a few dependencies on my RHEL 5, on restarting the system, the keyboard was not able to work. I restarted several times in vein.
However, when I start in single mode, it works perfectly. I have tried looking for a solution from google, but in vein.
Kindly help...
Regards, Andrew
View 1 Replies
View Related
Jun 2, 2010
How do you download a whole distribution at once from an ftp mirror? Ive never used ftp to DL more than 1 file at a time from konsole I tried mget, get as well as using wild cards like this get /slackware/*/*/*/*. Ive been looking for how to's but can't find any that deal with what I'm looking for. I know there is probably a simple solution but I can't find it.
View 8 Replies
View Related
Mar 4, 2010
I have apt-mirror working perfectly for my local repo.I have it sending me an email everyday after it runs.However, there is nothing in the email. I'd like for the email to show me how much data was downloaded, number of files, (the normal apt-mirror information when ran from terminal). Lastly, the clean.sh script contains nothing to do. What can I add to the script to clean up my repo (remove older packages, etc..) My repo keeps getting larger and larger. It's at 66.7GB now.
View 4 Replies
View Related
Jun 25, 2011
A friend recently introduced me to linux and I've experimented with a few different distros and now have 2 working puppies; 1 system i slapped together from misc. parts lying around and the other is my netbook which boots to puppy via USB. I have had to play around with formatting using gparted.
I very recently acquired a server unit with a pentium II 200Mhz and I would like to LEARN linux. Thru careful research I have figured out that Slack is the best OS for those who want to learn the in's and out's. I guess my question would be which slackware distro would be best for this somewhat older system? ...And where can I find a mirror to download the iso?
View 5 Replies
View Related
Jan 9, 2010
I'm using f12. My university has recently become a mirror for fedora packages. But i'm facing a trivial problem. I have set proxy for yum so that the packages that are not available on local mirror can be downloaded from other mirrors. But then i don't know how to set no proxy for my local server. Consequently it is not using my local mirror at all. Tell me how to set no proxy for the local mirror. I want my local mirror because it's damn fast. My proxy settings are like this (they go in /etc/yum.conf):
[code]....
View 2 Replies
View Related
