I have a crontab that wgets a PHP page every five minutes (just to run some the PHP code), and I want to send the output to /dev/null. I couldn't find it in the wget manual.
I'm looking for something like:
wget -o stout http://whatever.com/page.php > /dev/null
My friend has a website whereby once you have logged in on one page, you are redirected to another page, with a url similar to:
[URL]
the random string changes each time you log in, however the login page has a static url What i was attempting to do is run a script to get some data from the members page (after uve logged in) - however ive been having some trouble in how to do this, as the variable url with the random string will become invalid after a certain time, and i did not want to consantly change it.
While reading through some documentation i read that wget should be able to login to a form login website however ive had no luck, the command i was attempting to use was:
wget --user USERNAME -password PASSWORD [URL]
similarly i also tried
wget --post-data "username=USERNAME&password=PASSWORD" [URL]
and even both combined. However neither has worked as the html dl'd is simply the login page website. I cannot post a direct link to the website as it is private, however ive looked at the source coding and ive extracted (what i think) is the relevant bit, which is:
Code:
<form action="/cgi-bin/sblogin/login.cgi" method="post" name="login" id="login"><br /><br />
<div class="user_text"><span class="text3">USERNAME:</span></div><div class="user_box"><input type="text" class="text" name="uname"></div>
[code].....
The issue is that my CentOS workstation is in a vlan from where the Intranet's DNS servers are unreachable. For browsing the web there is an ISA proxy server, which I presume resolves DNS for my firefox. However, wget, host, ping and aria2c fail to get any sort of DNS resolution since they're being run from command line.I have exported HTTP_PROXY value, which provides me internet access on console, but,only when I connect using IP address. It fails on name resolution.
My question is:May I redirect the DNS queries to my home PC which would be running a DNS server on a non standard port?I was thinking of putting nameserver 127.0.0.1 in /etc/resolv.conf and then put iptables rule to redirect 127.0.0.1:53 UDP to a.public.ip.address:3535 UDP..I don't know if I am shooting blanks or what, I am not very much aware of this kind of setup.My main need is to provide DNS resolution to console apps.I want to utilize my company's idle bandwidth for bulk downloads, so, using proxy, SSH tunneling through my Home PC is out of question.
If a wget download is interrupted (like if I have to shutdown prematurely), I get a wget.log with the partial download. How can I later resume the download using the data in wget.log? I have searched high and low (including wget manual) and cannot find how to do this. Is it so obvious that I did not see it? The wget -c option with the wget.log as the argument to the -c option does not work. What I do do is open the wget.log and copy the URL and then paste it into the command line and do another wget. This works but the download is started from the beginning, which means nothing in the wget.log is used.
View 2 Replies View RelatedI am trying to grep multiple numbers from file, grep does have the -f option for that.
Code: grep -f <`seq 500 520` /etc/passwd I know this could be done with
Code: for i in `seq 500 520`; do grep "$i" /etc/passwd; done But my question is fare more behind this example. It is possible to redirect one command output which will be treat as a content of file for another command ?
I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs static links.
View 1 Replies View RelatedI have a computer under Linux with several network cards, for example: eth0, eth1, eth2, eth3. Is there some way to run any downloader, like aria2 or wget only through one interface, for example eth0?
Main problem: for some reason I can't use iptables
I'm doing this wget script called wget-images, which should download images from a website. It looks like this now:
wget -e robots=off -r -l1 --no-parent -A.jpg
The thing is, in the terminal when i put ./wget-images www.randomwebsite.com, it says
wget: missing URL
I know it works if I put url in the text file and then run it, but how can I make it work without adding any urls into the text file? I want to put link in the command line and make it understand that I want pictures of that certain link that I just wrote as a parameter.
i use this code to download :wget -m -k -H URL... but if some file cant be download , it will retry Again and again ，so how to skip this file and download other files ,
View 1 Replies View RelatedI need to download about 100 packages so I'm using wget-list to make it easier. My question however, is once I've made the list (I assume it's in a .txt format), is there a way I can insert comments into it that wget will ignore? Something like this:
#This is a comment
http://someurl.com
http://anotherurl.com
I had the bad surprise that wget doesn't redownload when a file of the same name already exist.
Is there an option to force him to redownload without deleting the file first on linux ?
I have a website that I need to go to often and disable a monitor. To disable I need to login to the website-> click on monitor -> then uncheck a box.
I am told that I can do this through a scipt using the WGET command. I got the parameterized query and then tried to execute it through a *.sh script.
The script generates a php file in the location from where it is executed. When I go to the site and check the monitor is not disabled.
I m trying to access a site through a perl script for a project of mine, and i use a system call fora wget.
The login form is this
Code:
I mean should i add in the --post-data all the hidden fields? should i try using perl's md5 function for the last two fields? anyone has any idea on what are the elements i should be sending along --post-data?
Is there a way to --load-cookies from mozilla or something similar instead of creating new cookies with wget?
I would like to use wget to downlaod file from Redhat linux to my windows desktop , I tried some parameter but still not work , can advise if wget can do download file from linux server to windows desktop ? if yes , can advise how to do it ?
View 14 Replies View Relatedi am using Ubuntu 10.04 when i downloaded some thing using wget like wget [URL] where this page will get downloaded and second thing sudo apt-get install perl-doc i installed documentation for perl the same i have for postgreSQL... how to use these perl documentation in learning perl.
View 1 Replies View RelatedI want the wget command to work on my linux machine.This is the output of the uname command on my machine Linux kalpana
Quote:
I get the error -ksh: wget: command not found.So can anyone tell me how do I install the wget utility on my machine.
I am trying to download data/file from web server where htpassword has been setup, I have tried with browser it its working fine, but when trying to same with 'wget' its not working, how to download the file. Below is the command I am using. [URL]... admin[:]password (may be smily get overide)
View 4 Replies View RelatedWhen I wanna use wget to download some file by http, which conditions fulfilled on the server would make that successful. I mean that such service httpd is running and so on.
View 1 Replies View RelatedI'm wondering if it's POSIX + ASCII or something else is mixed in?
View 1 Replies View RelatedI ran into an issue that was written up here on LQ and/or on other sites, the one dealing with an error similar to the following one: error trying to exec 'cc1plus': execvp: No such file or directoryThe solution seems is to make g++ and gcc versions consistent. I've since remedied that, but am slightly confused with my findings and concerned with my solution.
When I first looked in /usr/bin for all references to g++ and gcc, I saw that there were two version of gcc and one version of g++:
(Note the "??? ?? ????" as date for gcc. I've since made some changes, but this was essentially the set of gcc and g++ files that existed before I started.)
I'm told that the system I'm using is an "out of the box" installation, i.e. no modifications. As installed, gcc is the newer version and does not correspond to g++34:
Confusion and concern: When I use makefiles from software I've adopted, I run into the cc1plus problem eluded to above. The cc1plus error occurred because the makefiles expected there to be a command named "g++," so I created a symbolic link, pointing /usr/bin/g++ -> /usr/bin/g++34. And in doing so, g++ was not consistent with gcc. I've since fixed that, i.e. copied the gcc34 version to gcc and my software builds fine.
My question are:
1. Will copying the gcc34 version to gcc cause issues in the future, possibly related to upgrades and/or istallations of other packages that rely on "gcc"? Currently the files are as follows:
2. Were the g++ and gcc files in /usr/bin the "out of the box" versions?
3. How can I answer a question like this in the future, without posting to LQ, i.e. is there a reference to find this type of information?
Is there some sort of standards file path convention for installing softwares that I could follow through? For example, I just learnt how to build Nginx from source. But the default binary path set by nginx is "/usr/local/nginx/sbin". I have seen a couple of tutorials which they specify the location of the installed binary and it is very different from those usual default paths. Thus, got me thinking whether is there some form of file path convention that I should follow?Is there some kind of list which states where do those packages on Debian.org Repository usually installed to?
View 6 Replies View Related[1st off...I used to host my music (A&R) website on a dedicated windows server...'cause I thought it was necessary for streaming .wma files. when I discovered it wasn't...I went w/ a standard hosting package, w/ GoDaddy...
BUT...
I moved my recording studio's site to it's own domain. [it used to be a series of pages on my A&R site.]
I now need to create a '301 Redirect' in my old site's directory to redirect surfers (who have my old url) to the new site...but more importantly...to retain my ranking w/ the major search engines...!
It cannot be done on a Windows server (which I'm on, w/ GoDaddy)...not an .htm redirect anyway, which I need. I've tried all the php, asp, etc. scripts...they don't work. [&...I don't think I really need windows features like php or asp...I code my site(s) in Dreamweaver as html pages.]
so...If I migrate my site(s) to a Linux server...
1) Will standard .wma streaming continue to work...? (i.e., a .wax file in the directory that points to the .wma).
2) Which type of server is FASTER for this task...(if any)...Linux or Windows...?
I am trying to use wget to access a RESTful interface, but I can not figure out how to do HTTP PUT with wget. How can I do it? Or isn't it prossible?
View 2 Replies View RelatedI'm trying to download two sites for inclusion on a CD:URL...The problem I'm having is that these are both wikis. So when downloading with e.g.:wget -r -k -np -nv -R jpg,jpeg, gif,png, tif URL..Does somebody know a way to get around this?
View 2 Replies View RelatedWhy doesn't this work?:
View 1 Replies View RelatedA friend of mine put up a bunch of mkv files on a public server, how can i download them all with one wget command?
I have tried
wget -r [path]
which simply grabs the index file, robots.txt and skips the mkvs. I also tried
wget -r -A.mkv
If i try getting a individual file directly it works fine, what am i doing wrong here?
Let's say there's an url. This location has directory listing enabled, therefore I can do this:
wget -r -np [URL]
To download all its contents with all the files and subfolders and their files. Now, what should I do if I want to repeat this process again, a month later, and I don't want to download everything again, only add new/changed files?
I was unable to find an answer to something that (I think) should be simple:
If you go here: [URL]
like so:
wget [URL]
you'll probably end up with a file called "download_script.php?src_id=9750"
But I want it to be called "molokai.vim", which is what would happen if I used a browser to download this file.
Question: what options do I need to specify for wget for the desired effect?
I'd also be ok with a curl command.
Example: [url]
This was what I tried...wget -A rar [-r [-l 1]] <mediafireurl>
That is to say, I tried with and without the recursive option. It ends up downloading an HTML page of a few KB in size, while what I want is in the range 90-100 MB and RAR.
What happens with MediaFire for those who may not be aware, is that it first says
Processing Download Request...
This text after a second or so turns into the download link and reads
Click here to start download..
How to write a proper script for this situation.
I'm trying to download all the data under this directory, using wget: [URL] I would like to achieve this using wget, and from what I've read it should be possible using the --recursive flag. Unfortunately, I've had no luck so far. The only files that get downloaded are robots.txt and index.html (which doesn't actually exist on the server), but wget does not follow any of the links on the directory list. The code I've been using is: Code: wget -r *ttp://gd2.mlb.***/components/game/mlb/year_2010/
View 4 Replies View Related