Server :: Apache Overloaded By Robots?
Jan 12, 2010
im running vps with debian lenny, with apache2 webserver, memcached, postgres and django-framework using mod-wsgi. Some of the pages served by django take long to generate (aprox 15 s), but there is also the memcached, compensating this.
My problem is that, when a robot visits the site, it starts traversing the site visiting all the pages and also the nongenerated, thus slowing it down to point where it is not responding.
What Im looking for is a solution to identify that the request comes from a robot (user-agent, ips, etc) and limit the resources, so that f.e. only one thread serves the robot etc...
View 2 Replies
ADVERTISEMENT
Apr 22, 2010
what it is that's overloading my web site.I'm running LAMP on Fedora 12, on an AMD 64 processor. My web site is relatively low volume; a good day is over 250 visitors, and most days it's below 200. I can't see anything there that would overload even a small box like mine.Several times a day -- perhaps 5 or 6, it's hard to say because I'm not always there -- I get flooded with requests. I run "top" in the background all the time to watch it, and what I see is the load average going through the roof -- I've seen the 1 minute figure go over 50 in about 3 minutes -- but the cpu numbers stay reasonably low, which I interpret as the system being I/O bound. The "top" display will show at least 40 or 50 http sessions in flight, with PID numbers spanning around 150 numbers slightly out of sequence, suggesting that the requests hit in close proximity but not precisely at the same time. These episodes can last up to 40 minutes before the system clears and the load avg goes back down to something sane, although I've had instances where I get a flurry of activity that lasts maybe 5 minutes, and the load avg goes no higher than about 20. The httpd log does not show any particular pattern of clients hitting my server. The requests appear to be historical pages from the blog (e.g. I notice requests for images from older pages, not the current front page.)
My best guess is that what I'm watching is Google or some other web caching service scanning my site for caching purposes. But I don't know. Maybe I pissed off some aggressive hacker (it's a political site) and he/she/it has figured out a way to periodically cause me grief from masked sites.I have two questions:
1) Does anybody recognize this pattern? Can you tell me what it is?
2) How can I streamline mysql and apache so these incidents don't cripple me for half an hour?
View 3 Replies
View Related
Jul 7, 2010
I need to stop the viewing of robots.txt on my website. I get the contents of the file displayed in my browser when I issue the command: [URL] stop this as it displays all the directories I don't want them to go to.
The main problem is that I can look at any directory on my site and get a file listing and then right-click on that file name and then save it to my client hard-drive all from my browser. ex: [URL] I think I can change this behavior in the apache config but don't know enough.
View 3 Replies
View Related
May 19, 2010
gauge when the server is getting overloaded with users. At present I run the server mainly as a proxy server with about 100 users. The bandwidth at the data centre is 100Mbps connection with total bandwidth used last month = 17431.16 MB I would like to add a VPN in future but feel that this might overload the bandwidth as instead of it just being web traffic it will the entire client TCP connections. I would like to monitor this before it gets to the stage where users are complaining but not sure how to gauge whether the proxy is being overloaded. It is used mainly for video traffic.
View 11 Replies
View Related
Sep 24, 2010
I've got this problem for a few weeks and I cannot figure out. I'm pulling my hair out. I have a server installed PHP, lighttpd and redis. Sometimes, I got the following messages in the error log of lighty: Code: 2010-09-24 13:57:33: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to anoth er backend instead: reconnects: 0 load: 567 2010-09-24 13:57:33: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to anoth
er backend instead: reconnects: 0 load: 626 and:
[Code]..
View 3 Replies
View Related
Oct 14, 2010
I am trying to solve a problem where Apache stats aren't displaying correctly in Munin. I've ran through quite a bit of checks and tests regarding Munin setup, but I think my issue is related to Apache, but my skill set there is lacking.
first, system info:
monitored server:
CentOS 5.3 2.6.18-128.1.1.el5
[code]....
View 7 Replies
View Related
Jul 7, 2010
I need to stop the viewing of robots.txt on my website. I get the contents of the file displayed in my browser when I issue the command: [URL] stop this as it displays all the directories I don't want them to go to. I am right at the level of Linux knowledge to be dangerous, capable but still dangerous.
View 5 Replies
View Related
Oct 11, 2010
I'm using the lego RCX2 to create robots for a class, and I can't figure out how to get the tower to work with Ubuntu
View 4 Replies
View Related
Jan 17, 2010
My mom's Ubuntu machine has been rather strange the last day or so. It takes a while to turn on, delaying apparently at loading "powernowd".
GDM seems to load fine, but then logging in to Gnome takes a long time. The panels ususally crash, as well as the desktop, but eventually everything loads again.
Most programs take a long time to load; also, "sudo" takes a long time to request a password, but after that the "sudoed" command runs normally.
Gksudo never seems to show up, so I can only access programs like "Software Sources" via terminal (e.g. sudo program).
Strangely, Gnome System Monitor says the processor and memory are not completly full.
Does anyone know what this is? Are there any commands I can run to diagnose it?
View 9 Replies
View Related
Nov 19, 2010
In my logs for Apache I have lots and lots of failed attempts for incorrect incarnations of [URL]. None of them are anywhere near my alias for the index.php but yet phpmyadmin is broken. Is there away I can mess up robots like this. Send IP's that create multiple wrong page requests on my server back to their own IP address maybe? I would then just set thresholds to decide how strict to be. I did try fail-to-ban before but it is cryptic. I don't have it on this particular server.
View 2 Replies
View Related
Feb 4, 2010
Any one have an idea How to clear apache cache without restarting apache server.
View 5 Replies
View Related
Jan 24, 2010
I am upgrading my server and I have a lot of sites. Since I cannot take my server down for a few days, maybe a week until I manage to migrate all the sites to the new machine, I figured I could migrate them one by one. After migrating one, I would somehow tunnel the requests of that name virtual host to my internal machine. When everything is migrated, I would then switch the machines, update ip's and stuff and everything will work just fine.
However I cannot seem to find a way to do this tunneling. is this at all possible? If not, what alternatives do I have?
View 5 Replies
View Related
Jun 29, 2011
What is the best way to determine in Linux if a multi-processor machine is overloaded? I thought load was a good measure but I run a large number of tasks which don't consume a lot of CPU but which drive up the load. A 4 processor machine has a load of 66 right now according to top for example but mpstat reports that the all CPU idle time is 89%.
View 1 Replies
View Related
Jan 20, 2011
I am installing Big Brother on a CentOS 5.2 running the default Apache 2.2.3. When I try to access any web page I get the following error: Forbidden You don't have permission to access /bb/ on this server. Apache/2.2.3 (CentOS) Server at fmsubbnix Port 80 So far I have:
1) Set the Directory options to FollowSymLinks
2) Verified all directory and file permissions are at 755
3) Set permissions temporarily to 777 and received same error so I am assuming the issue is in a config file somewhere
4) in hhtpd.conf verified <Files ~ "^.ht"> is correct
5) verified the "default" directory is correct (/var/www/html)
I have read and tried several ideas in posts listed on the web but to no avail and am at a loss as to what to look for next..
View 3 Replies
View Related
Apr 21, 2011
I'm running a linux cloud server with the following config
1.2ghz Processor allocation
752MB Ram
The site loads slow and clicking a link almost freezes the page for a second. Also, the page loads could be much faster. We've been running mysqltuner and have pretty much optimized all slow queries. Is there anything we can do to fine tune the server for faster and more responsive?
Httpd.conf
Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
code....
View 2 Replies
View Related
Apr 16, 2011
I'm having a bit of an issue using overloaded operators in an already overloaded operator. In my following code, I have overloaded the && operator to compare two Course objects. The operator in turn goes to a function which calls other overloaded operators to compare private object variables of that object to compare them.
View 8 Replies
View Related
May 22, 2011
Does anyone have some material about statistics using ubuntu / linux server, or a text which generally describes the ubuntu server?I need urgent, i'm writing specialization work about administration apache and ftp server on ubuntu 10:10 server, so I need something for the conclusion.
View 7 Replies
View Related
Feb 2, 2011
I try to config my apache server to list all my files: c/c++, php, java files, like the txt file on my server,
e.g /var/www/mydomain/pub
i want to dump all my c/c++, php, java file under the pub directory and I can access it from my domain name,
if I dump txt file, I have no problem to view it, but when I dump c/c++ or php files under pub directory, then I can't view it like regular txt file,
Q: is there anyway I can configure my apache server to view all the c/C++, php, java file as like txt file?
View 1 Replies
View Related
Feb 23, 2011
config my apache server to list all my files: c/c++, php, java files, like the txt file on my server, e.g /var/www/mydomain/pub i want to dump all my c/c++, php, java file under the pub directory and I can access it from my domain name, if I dump txt file, I have no problem to view it, but when I dump c/c++ or php files under pub directory, then I can't view it like regular txt file, Q: is there anyway I can configure my apache server to view all the c/C++, php, java file as like txt file?
View 1 Replies
View Related
Dec 7, 2010
I and setting up a home web server using Ubuntu 10.04 server (local only). I am currently using Webmin 1.53 to access it remotely all is going great very easy to use. Webmin - Check, ftps-fileZilla - Check, Apache -It Works BUT I cant seam to set up Apache as a named server using Bind DSN. Tried most of the help in the fourms and ..... I think my problems is in the master server selection, do i have to use [URL]... or can i just use myservername. I have tryed both with no luck. First time with the server addition.
[code]...
View 6 Replies
View Related
Mar 21, 2011
I have install Mandriva 2009 and install apache server but its not strating on default configuration file . What I should do for starting apache . i have tried /etc/init.d/httpd start but apache is not running .
View 1 Replies
View Related
Aug 19, 2009
I installed Apache server with Debian 5.0.2 Lenny. I am trying to write a script which would analysis web log files. I found the log files on /var/log/apache2. There is an access log file, `access.log`. My question is what configuration file determines the location and the name of the access log file. How can I change them? I used CustomLog in /etc/apache2/apache2.conf like below.LogFormat ": %h %l %u %t "%r" %>s %b" common
CustomLog /home/test/my_log_file common Apache2 generated /home/test/my_log_file. But no logs were written in the file even after I run `/etc/init.d/apache2 restart`. Ichanged the log file location. It still didn't work. However, Apache2 still wrote logs in the file `/var/log/apache2/access.log`
View 1 Replies
View Related
Jul 2, 2010
Can anyone tell me in performance tunning of apache-tomcat and jboss application server?when I deploy some application in apache-tomcat the performance is say 100 users per second and incase of jboss it is even worst (35-40 users per second)i want to improve my performance 1000 uesrs per second..
View 1 Replies
View Related
Feb 10, 2010
I have a requirement of using a wildcard certificate for 5 subdomains running under apache httpd server and 1 subdomain under tomcat.Is there any possibility of using the single wildcard certificate both in tomcat and apache
View 3 Replies
View Related
Apr 17, 2010
I installed apache server on my Fedora 12. Its showing the test page 127.0.0.1, but when I give the address 97.168.234.76 (0r www.chaalu-kam.com), it tells that the requested url could not be retrieved.
I have made the following additions to the /etc/httpd/conf/httpd.conf file
Code:
In addition to this, I have added this to my /etc/hosts file
Code:
I wanted to test the Apache server on my laptop only by putting an html file in /var/www/html and naming it as index.html. I want this index.html page to be displayed when I put www.chaalu-kam.com in my browser. What should I do? My firewall is off too
View 10 Replies
View Related
Feb 26, 2011
I'm setting up a full server configuration on Fedora, moving it from Mandriva which sadly faces a rather uncertain future. Things are going well and I will be ready (and more familiar with Fedora) in time to upgrade to Fedora 15.
What I have done so far in regard to networking setup:
I first removed Network Manager and replaced it with the network service (chkconfig --level 25 network on). That put an end to the overwriting of /etc/hosts, which now reads simply:
View 2 Replies
View Related
Jul 22, 2011
I am running a CentOS 5.6 Server with a website and Forum. The website has a contact form and users can email various people in the organization. I have setup sendmail to send any mails like this to my email address on a Windows machine.The problem is that there emails don't have a header which has the senders original email header. It comes from localhost on my sendmail. How can I get any mail generated from the apache site to ALSO send mail to root on the Centos Server? That way I should be able to see the header and report the spammer or block them.
View 2 Replies
View Related
May 12, 2010
I'm moving an office intranet server to a new machine running CentOS 5 with apache 2.2.3. Everything worked fine on my old CentOS server and nothing new or fancy being tried on the new machine: default server, no name virtual hosts etc.. However, on the new server I can only see the index.html page when I browse to it. Index.html is a frameset which calls other pages to load from subfolders to /var/www/html. The frameset loads but in each frame I see "The requested URL /hot_stuf/data_colm.html (etc.) was not found on this server.".
View 3 Replies
View Related
Sep 29, 2009
My current setup is:
old server:
www.mydomain.com main site
www.mydomain.com/subdirectories related sites from same server, different directories
I am adding an additional server that I would like to initially only use for the main site, something like this:
new server
www.mydomain.com main site
www.mydomain.com/subdirectories would be pointed back to the old server instead
What's the best way to redirect the traffic for the sites found in sub-directories on the old server?
View 2 Replies
View Related
Oct 8, 2010
I have CentOS 5 configured with apache web server. but it is not asking for password authentication while accessing the web page. the config detail is as below code...
View 15 Replies
View Related
