Programming :: Bash - Parse Apache Log For Requests Per CIDR / 24
Feb 6, 2011
I'm trying to figure out how I can get a request count per CIDR/24 from an apache log in combined format - e.g.:
Code:
24.147.44.122 - frank [10/Oct/2011:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "[URL] "Mozilla/4.08 [en] (Win98; I ;Nav)"
I'm stuck using BASH for this and I generally write everything in Python, or even Perl if necessary so BASH isn't the most comfortable for me. I've got enough to extract the IPs and get a count I just need a slick way to come up with a sum on a per CIDR basis.
I am trying to think of a logic where my file contains some data I had to read and do some processing. Issue is that file contains data multiple times. For example:
::::::::::: var1=value1 var2=value2
[code].....
I have to read first paragraph of variables and do some processing and then move on until the end of file. Variable names are same in whole file but for each paragraph the value is different. I can't think of a logic to attain this task. How can I do it? It should be a simple bash script, but I am not able to work out.
What's the easiest way for me to make a program that requests a file location and then tar balls it. I basically want to start making a simple method of using pv with tar (lzma).
Question 1: Hey. I noticed i have quite a few logs that end with .[number] for example "syslog.1" "mail.info.1" etc, why is this and why are they there since almost nothing is logged in them ??
Question 2: on my server im running a script like imagebam and imageshack with hosts images so i have quite a few apache requests to my server. I am wonder why apache takes up so much CPU for some of the requests ? in Htop some requests take up 1.2% CPU while other take up 3-5% etc, so the total load is about 1.50 0.58 0.84 to 2.61 1.08 1.14 with about 128-150 apache requests all the time while sometimes the CPU load can be almost 0 with the same ammount of requests. is this normal? what could cause this in apache ?? the server is just running apache2. MYSQL is running on another server.
I've been scouring the message boards and trying different things for two days and haven't found a solution. I set up Ubuntu 10.04.1 a few days ago using the server iso and selected the LAMP installation option. I will be using it as a sandbox to try out things from a PHP/MySQL book I purchased.
I have installed Gnome since sometimes I just can't get the command prompt to let me do things and Gnome may at least tell me why. I also installed PHPMyadmin.
My problem is that I can work with this thing all I want in any way I want - HTTP, SSH, SFTP, etc. - from within my home network but Apache refuses to respond to an outside request. Since it's only a test box it usually wouldn't be that big of a deal but I am going out of town for a couple of days and would really like to start working with my new book.
I am 99% sure my ISP is not blocking any ports. I have a ComTrend ADSL modem with router. I have given my Apache server a static IP on the NAT (xxx.xxx.x.101) and set up a dyndns address so I don't have to keep remembering the IP address.
Here is what I have tried so far: Setup a virtual server (port forwarding) on the router to to direct traffic on port 80 to internal IP .101The router told me its interface is using 80 so it would move itself to 8080 Made sure to also add port trigger for port 80 Made sure to save/reset the router Used my iPhone to connect via 3G - didn't work using IP or dyndns name Used iPhone to connect to 8080 and router responded
Set port forwarding/triggering for 8080 -> 80. No joy there, either Tried changing listening port to 8000Set port forwarding and triggering to allow port 8000 Changed ports.conf to read NameVirtualHost *:8000 and Listen 8000 Changed first line of /etc/apache2/sites-enabled/000-default to <VirtualHost *:8000> Restarted Apache service
Apache responded to dyndns.com:8000 from home machine No response when trying same on iPhone (sorry, it's the easiest way for me to test from outside my home network)
At one point, I added "ServerName localhost" to the otherwise empty httpd.conf file but that didn't seem to do a darn thing.
Many of the posts I have perused are at least a couple of years old and have included information on taking action with files that are not there or are no longer where they were. I have interpolated where I can but so far nothing has worked.
my server into the public internet and must have my webserver [2.2.9 (Debian)] act secure. But this does not look very easy [I am searching, reading and working on it already the whole day ]. I read the apache docs, but there is a lot of stuff, which is different in Debian [lenny, 5.0.6]. Apache ignores the host-header given by the browser: [URL] are all served, but should be blocked. I new new to apachy, but on my IIS this works as expected. All browsers act equal [so no browser header problem].I configured two VirtualHosts, an excerpt:
The I go to my hosts file on any box, and add hugo's ip-address under the new name x. Then, x is served, although the host-header in apache Every user coming from the internet could make the same!
I noticed i have quite a few logs that end with .[number] for example "syslog.1" "mail.info.1" etc, why is this and why are they there since almost nothing is logged in them ??
Question 2: on my server im running a script like imagebam and imageshack with hosts images so i have quite a few apache requests to my server. I am wonder why apache takes up so much CPU for some of the requests? in Htop some requests take up 1.2% CPU while other take up 3-5% etc, so the total load is about 1.50 0.58 0.84 to 2.61 1.08 1.14 with about 128-150 apache requests all the time while sometimes the CPU load can be almost 0 with the same ammount of requests. is this normal? what could cause this in apache ?? the server is just running apache2. MYSQL is running on another server.
I'm running wordpress and have gone to the painstaking effort of setting up the ftp server on my 5.3 machine so it can do its own updates and download plugins. However, I've found that if I try to download anything, I get a "unable to resolve host..." error from the script. If I watch Iptraf while making the request, I see all the port 53 requests going to 127.0.0.1. Pings are too.
However, if I ssh in, I can ping and wget and whatever I want all day long and all the DNS requests to go the router (Clarkconnect 3.2 gateway machine). The web browser in the terminal works fine too. What is it about the php scripts that is causing them to misdirect pings and dns requests? I've never seen anything like it and can't find anything on the web about it either.
I'm working on a thorny mod_rewrite problem. I have a mac connected to my LAN running MAMP (Mac/Apache/MySQL/PHP). I request a non-existent file:
Code: http://192.168.1.2:8888/careers/db/1.html I see this in the mod_rewrite log file:
Code: 192.168.1.102 - - [14/Nov/2009:13:46:07 --0800] [192.168.1.2/sid#807df8][rid#8ec850/initial] (2) init rewrite engine with requested uri /careers/db/1.html 192.168.1.102 - - [14/Nov/2009:13:46:07 --0800] [192.168.1.2/sid#807df8][rid#8ec850/initial] (1) pass through /careers/db/1.html Note that the requested uri is /careers/db/1.html
If I change just the file extension on my request to PHP like so:
Code: [URL]
Then the request uri is totally different now. Here's the rewrite log:
Code: 192.168.1.102 - - [14/Nov/2009:13:47:23 --0800] [192.168.1.2/sid#807df8][rid#8fc850/initial] (2) init rewrite engine with requested uri /Applications/MAMP/htdocs/careers/ 192.168.1.102 - - [14/Nov/2009:13:47:23 --0800] [192.168.1.2/sid#807df8][rid#8fc850/initial] (1) pass through /Applications/MAMP/htdocs/careers/ Note that the requested uri now has a full path which does not include the actual filename, /Applications/MAMP/htdocs/careers/
What the heck? More info. If I request [URL], I can actually access p1.php. The requested uri is /careers/db/p1.php. The problem appears to be because the filename starts with a number. I can also request [URL] and get thru to 1.php with requested uri /careers/db//1.php. Does mod_rewrite think /1 refers to a backreference or something? Why can apache handle the html file request properly and not the php file request?
On my server I provide OCR file conversion service but the problem is when a user uploads a file and it's being converted then if you open another tab and try to load the site it won't respond until that conversion is completed. In other words until the PHP script finishes execution apace doesn't serve any other request to the same browser.
Here is my apache configuration:
Code:
ServerTokens OS ServerRoot "/etc/httpd" PidFile run/httpd.pid
[code]....
You can check what I mean if you try to upload and convert a file and while the file is converting try opening the site in another tab.
I've had a VPS running Ubuntu 9.10 x64 server, hosting 3 websites of mine for a few months now. This problem has been happening for a while. Every once in a while, probably every 2 or 3 days, I'll wake up in the morning, and apache won't be responding, no web pages will load. /etc/init.d/apache2 status, reports that apache is functioning properly. Every time I simply have to restart the daemon and things run fine for another few days.
I thought maybe it was a memory issue, so I lowered the MaxClients in the prefork module from 50 to 30 a few days ago, but the same thing is still happening. My VPS has 512MB of ram, burstable to 1GB, and according to Virtuozzo, there was only one night of high traffic where I even came close to that soft limit. I've checked my syslog, and there's absolutely nothing in there about apache. I've checked apache's error.log as well, and there's nothing in there that would indicate a problem either.
A Linux (CentOS5.3) server is setup with apache reverse proxy. The reverse proxy server is opened to outside and an internal server is mapped to ProxyPass configuration. SSL certificate is also installed on the Apache reverse proxy server. The problem is, it is extremely slow in serving http requests through reverse proxy. There is no problem with server resources or bandwidth. When the internal server is directly accessed through Internet, there is no delay. The backend server and the reverse proxy server are also on the same switch (same subnet). When I searched the Net, there were recommendations to enable cache in Apache. I did so as follows in httpd.conf.
But still there is no progress. Do I want to enable cache in ssl.conf too? Or is there any other workaround to speed up Apache reverse proxy. Is there a way to check that caching is happening?
I want to write a bash script to parse a text file with the following lines and set variables for each line so that I can use them in the rest of the script.
Timestamp=123456789 Company=ABC Company Server=Server Recipient=Joe Smith Email=joe@abc.com
simplest way to read each line one at a time for everything before the =, set that to a variable name with the value equal to everything after the =
I have a log file that contains information like this:
---------------------------- r11141 | prasath-palani | 2010-12-23 16:21:24 +0530 (Thu, 23 Dec 2010) | 1 line Changed paths: M /projects/ M /projects/
[code]....
what i need is, i need to copy the data given between the "---" to seperate files, for, e.g. the first set of data between the "---" should be in one file and another set of data in another file.
what is the best command to use to parse strings?I have a variable $str and need to parse this string.Can you provide an example of the command used to get a substring of $str based on the index values of start and end
My php knowledge is very poor (only worked with strings so far), and I am faced which a task that is a real challenge for me: I have a variable, that contains data of different type, in this order: byte, byte, string, string, string, string, short, byte, byte, byte, byte, byte, byte, byte, string.
Strings are of variable length. How could this data be parsed into variables of the right type, and then all converted to strings? What are the functions to use? Strings are unicode ones, and they are delimited by "
I have to parse a file containing billions of records and populate them in the Data structure. I have used a lot of C++ class and creating objects of the class I am storing the information retrieved by parsing the file.
Now as the file become huge and number of objects become very large my code is getting bad_alloc error as it is not finding any space avalable in the heap for allocating new object.
To run a parallel chemistry program, I need to build the host list. The cluster already has SGE(grid engine) installed and it generates a host list file with the following contents:
The important bits are hostname(ex compute-1-1) and number of cpus to use(ex 2). And for this program, it wants them in this form, a shell variable: HOSTLIST=hostname:cpus=X hostname:cpus=X .... I've tried this script, but it doesn't work
I have a basic HTML file set up which allows my to input some data. I have a MySql database set up behind the scenes with a table for this information to go in. When I click the submit button in my HTML file this PHP file opens and it comes up with "Parse error: parse error in C:wampwwwcomp39xinsert_student.php on line 32"Here is my code I am using in the PHP file:-
<html> <head> <title>COMP 39x FInal Year Project - Student Added?</title>
How does coalescing of requests takes place during merging if the following scenario occurs.. say request for sector number 3 and 5 already exists...and a new request for sector 4 arrives, then if it front merges with 5, will it also back merge (i.e coalesce in this case) with 3 ? if it happens ..which functions are called? and how do we get to know whether a single merge or multiple merges have taken place?
references : blk-core.c, elevator.c, deadline_iosched.c all under /block.
I don't think this is a "perl one-liner" of find and replace. I'm trying to auto-fill some information in a listing of files. The simplest example is that in the files the following exists:
I would want the script to find this and populate it with something like -- Date : 20101004-1758
I have a few more similar fields to autofill, and I'd like to do this from within a larger perl script I'm developing to process these files. So, how I perform in-place file modification from within a perl script?
