Programming :: Parsing Out Squid Access Log With Awk And Grep?
Apr 25, 2011
I'm trying to recreate a simple script I wrote to parse out the access.log to get a rough idea of websites that users are going to on our corp network. The issue I'm having is I want to pull out any line from access.log that ends in .com/ .org/ .net/ or whatever to only see what the user entered into the address bar and drop pictures, js's and everything else and log only this. so what I do is :awk '{print $8} | grep -e '[cong]|[ore]|[mgtv][/]'$ and nothing happens.I know there is an easier way to do this with awk alone,
I am using squid to controlling access to the internet all is working fine expect one of the user who is using outside organization portal to connect internet. But whenever he tries to enter in the portal by typing (EXAMPLE)url. Permission denied error from squid occur.
How can i allow this portal in squid. So squid will allow this to access.
I have logs files from freeradius that have looks as follows:
$ grep "Login incorrect (rlm_ldap: User not found" /var/log/radius/radiusd-inner-tunnel-20090831.log Mon Aug 31 09:25:27 2009 : Auth: Login incorrect (rlm_ldap: User not found): [John Doe] (from client oficina port 0 via TLS tunnel)
[code]....
I use the following line to get the amount of users that don't exist on ldap:
Code:
grep "Login incorrect (rlm_ldap: User not found" /var/log/radius/radiusd-inner-tunnel-20090831.log | awk '{print $14}' | sort -fu | wc -l
Now, awk on line one for example parses [John Doe] and [Joon Williams] as "[John" and that it's not what I'd want. I mean how could I do for awk looks username field as closed between squared brackets?
I have an old FC2 box running Squid version 2.5. It has been running since 2003 so I am in the process of replacing it. I have a new machine with FC11, iptables, and Squid 3.0 installed.
On the old machine I use iptables to intercept Port 80 traffic and send it to Squid. By default I block all internet access and allow only sites that are in an Allowed_Sites.txt file. Within Squid I also have statements to allow certain users to bypass Squid based on their IP address.
I have set up the same thing on the new box. I have iptables intercepting the Port 80 traffic and sending it to Squid. That is working because if I remove the redirect statement from iptables all internet access is blocked.
The problem I am having is that Squid is not blocking any websites. It acts like the ACL is set to http_access allow all. I have worked on this for several hours and am stumped.
Is there a Linux system call that can be used to get the group name from the GID returned by stat()? I realize that I could parse /etc/groups (if my user had sufficient permissions).
I've been loosely following this:http://norvig.com/lispy.htmlAnd I have a problem: the parsing function throws an array out of bounds exception. I thought that maybe I'm doing it wrong, so I copy and pasted the code from the page, and still the same error
What I am after is to get the string text from the clip tags. But for now I just tested to see if it can finds the command tags and print something if it does. But it doesn't find it. Anyone knows why ?
Looks like the xml is not good, i test it with a xml validator:
I have a log file (test.log) starting & ending within dash (--) as below. I am looking to write a parser for test.log. This test.log file currently has single value for one Job ID but I wish to parse for repeated N values of different Job ID - Job, User, Queue, Dispatched Date, Dispatched Time, Completed Date, Completed Time, Hosts/Processor, CPU_T and TURNAROUND. I can either output this 10 values in another .log file or dump into cgi.
The selected parameters from test.log for parsing with above 10 attributes are -
I have a variable in which the data is stored as below:
variable_test=0m0.001s 0m0.001s 0m0.001s 0m0.001s 0m0.001s 0m0.001s .....an so on.
There are lots of values in format like "3m1.057s" are stored in variable_test separated with an space between two such values. For exapmple, value is "3m1.057s" I need to save different parts of a value in three separate array variables such as the
var_hour=3 var_min=1 var_sec= 057
tell if this can be done using "awk". A "WHILE" loop might be used to separate and store theses values I guess?
I want to know how to get eg. the contents of a form on a webpage which has been passed to a server side PHP script, inside for example an array which I can read. I've been reading a ebook on PHP which as far as I can see doesn't cover this inside it.
My C foo is terrible! I am working with some code which reads lines from a file and then reformats the lines and writes them to a new file.The input lines look like this:
I wrote a multithread program(approx 1000 thread have to run) and each thread has to parse a file(for each thread there is one file, ex:thread1 has to parse file1 and thread2 has to parse file2 like this....). I wrote "parse" program as follows. It is working well, if i create 50 threads. but if i run more than 200 thraeds Im getting doublefree corruption as follows:
And some time I am getting parsing problem and error af follows:
Code:
powersetting.6607:1: parser error : Start tag expected, '<' not found (where powersetting.6607 is file name, when i check this file it is started with '<').
I've searched online and found many examples using getopts, but nothing that clearly explains it use, nor any examples of what I'm trying to do. I have a script named "process". It can take from 0 to 3 different options. I'd like to be able to handle these options regardless of the order that they are entered.
Syntax: process [-v] [-d #] [-h|-?] [string] -v = verbose mode on -d # = how deep to do the process, expecting a number parameter -h or -? = show command usage string = only process lines containing the specified string
As a curious side project I'm playing with mzXML data(an xml format for holding mass spec data). A typical scan can be quite large, even up into GB size. I'm wondering how would one go about parsing an xml file in sections, one section at a time. The idea being if the computer doesn't have enough memory to load up the entire data file, work on chunks of it at a time.
Is there a way to process individual characters one-by-one from a text file in Bash, or is that hoping for a little too much from this lovable old clunker?
I have a bash variable where the content looks like this where ;f1; and ;f2; are delimiters: ;f1;field1value1;f2;field2 value1 ;f1;field1value2;f2;field2 value2 ;f1;field1value3;f2;field2 value3
So what I need is to extract and put into variables each combination of f1 and f2 in a loop to something like that:
#first pass of the loop I need: f1=field1value1 f2=field2 value1
#second pass of the loop I need: f1=field1value2 f2=field2 value2
# third pass of the loop I need: f1=field1value3 f2=field2 value3
PI'm trying to write a script to list all open ports in the MINIUNPND chain in iptables and use the procotol, port and destination ip to open ports on another router using upnpc.Here is the output of iptables -L MINIUPNPD
No matter what i do i cant seem to remove the first 4 characters from the MYPROT array to leave only the digits. Also i cant seem to read the array back???
I thought it would simply be a loop reading each line and passing the fields in variables, executing upnpc commands i need then moving to the next line of the file until it reached the EOF.
I ran into a bit of trouble making a bash script. (Desktop is a directory, and I try to get it's modification date)
Code:
lamp:~# cmd='ls -l Desktop | grep -o "....-..-.. ..:.."' lamp:~# $cmd ls: cannot access |: No such file or directory ls: cannot access grep: No such file or directory
[code]....
When I type in the command directly, without using an inbetween variable, it works fine.
We have a sipmle office network set up that we also use use to connect to the internet, however of late the number of users has increased thus slowing internet access. Bandwidth upgrade is not an option thus i have to do bandwidth shaping on our linux router. The question is how do set the squid configs to allow certain IP's range a certain percentage bandwidtheg 60% and furthe divide the rest. Alternatively how can allow certain IPs to have higher bandwidth access.
I am trying to monitor how long an ldap search takes and maybe notify or something that a search takes longer than say 10 seconds.
Code: tail -n 1000 /var/log/ldap.log for SRCH in $( cat monitorldap.log |grep 'SRCH'); do echo search string is echo $SRCH
[Code]....
ok, so to start off with it doesn't appear to get the whole line, just a piece "Aug". How can I get the whole line into a variable so I can then cut it up into the pieces I need?
I'm using Zabbix on which I can use give bash command to the agent.This 1-liner will give me all the interfaces with their IPv4 addresses.I have a 2nd expression which returns a checksum so I can detect a difference whenever someone deletes/adds/changes an ipv4 interface.This is the output on my Ubuntu-server:
