Programming :: Parsing Out Squid Access Log With Awk And Grep?
Apr 25, 2011
I'm trying to recreate a simple script I wrote to parse out the access.log to get a rough idea of websites that users are going to on our corp network. The issue I'm having is I want to pull out any line from access.log that ends in .com/ .org/ .net/ or whatever to only see what the user entered into the address bar and drop pictures, js's and everything else and log only this. so what I do is :awk '{print $8} | grep -e '[cong]|[ore]|[mgtv][/]'$ and nothing happens.I know there is an easier way to do this with awk alone,
View 8 Replies
ADVERTISEMENT
Nov 22, 2010
I need to kind of grep within grep. My input file would be something like:
[Code]....
and I need to find the first occurrence of hello before MY PATTERN (hello 9008 in this case), so the output should be:
[Code]....
View 4 Replies
View Related
Dec 29, 2010
I am using squid to controlling access to the internet all is working fine expect one of the user who is using outside organization portal to connect internet. But whenever he tries to enter in the portal by typing (EXAMPLE)url. Permission denied error from squid occur.
How can i allow this portal in squid. So squid will allow this to access.
View 1 Replies
View Related
Aug 31, 2009
I have logs files from freeradius that have looks as follows:
$ grep "Login incorrect (rlm_ldap: User not found" /var/log/radius/radiusd-inner-tunnel-20090831.log
Mon Aug 31 09:25:27 2009 : Auth: Login incorrect (rlm_ldap: User not found): [John Doe] (from client oficina port 0 via TLS tunnel)
[code]....
I use the following line to get the amount of users that don't exist on ldap:
Code:
grep "Login incorrect (rlm_ldap: User not found" /var/log/radius/radiusd-inner-tunnel-20090831.log | awk '{print $14}' | sort -fu | wc -l
Now, awk on line one for example parses [John Doe] and [Joon Williams] as "[John" and that it's not what I'd want. I mean how could I do for awk looks username field as closed between squared brackets?
View 1 Replies
View Related
Jun 11, 2010
I have an old FC2 box running Squid version 2.5. It has been running since 2003 so I am in the process of replacing it. I have a new machine with FC11, iptables, and Squid 3.0 installed.
On the old machine I use iptables to intercept Port 80 traffic and send it to Squid. By default I block all internet access and allow only sites that are in an Allowed_Sites.txt file. Within Squid I also have statements to allow certain users to bypass Squid based on their IP address.
I have set up the same thing on the new box. I have iptables intercepting the Port 80 traffic and sending it to Squid. That is working because if I remove the redirect statement from iptables all internet access is blocked.
The problem I am having is that Squid is not blocking any websites. It acts like the ACL is set to http_access allow all. I have worked on this for several hours and am stumped.
These are my Squid rules:
acl allowed_sites url_regex "/etc/squid/Allowed_Sites.txt"
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl SSL_ports port 443
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow Bypass_Users
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl our_networks src 192.168.1.0/24
http_access allow allowed_sites
http_access allow our_networks
http_access deny all
icp_access deny all
htcp_access deny all
http_port 192.168.1.254:3128 transparent
hierarchy_stoplist cgi-bin ?
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320
visible_hostname FC11.proxybox
icp_port 3130
coredump_dir /var/spool/squid
View 2 Replies
View Related
Sep 14, 2010
Is there a Linux system call that can be used to get the group name from the GID returned by stat()? I realize that I could parse /etc/groups (if my user had sufficient permissions).
View 3 Replies
View Related
Feb 1, 2011
I've been loosely following this:http://norvig.com/lispy.htmlAnd I have a problem: the parsing function throws an array out of bounds exception. I thought that maybe I'm doing it wrong, so I copy and pasted the code from the page, and still the same error
View 2 Replies
View Related
Mar 14, 2009
i wanted to know how can I pars a xml file (read data from xml file) and from C program.is there any function to do this for me in C.
View 1 Replies
View Related
Jul 7, 2011
I'll show the code first:
Code:
What I am after is to get the string text from the clip tags. But for now I just tested to see if it can finds the command tags and print something if it does. But it doesn't find it. Anyone knows why ?
Looks like the xml is not good, i test it with a xml validator:
Code:
View 1 Replies
View Related
Nov 9, 2010
I have a log file (test.log) starting & ending within dash (--) as below. I am looking to write a parser for test.log. This test.log file currently has single value for one Job ID but I wish to parse for repeated N values of different Job ID - Job, User, Queue, Dispatched Date, Dispatched Time, Completed Date, Completed Time, Hosts/Processor, CPU_T and TURNAROUND. I can either output this 10 values in another .log file or dump into cgi.
The selected parameters from test.log for parsing with above 10 attributes are -
--
Job <345010>
User <xyz>
Queue <gaussian>
[code]....
View 3 Replies
View Related
May 11, 2010
Does anyone have source code on Email parsing in C++
View 1 Replies
View Related
Jan 24, 2011
I have a variable in which the data is stored as below:
variable_test=0m0.001s 0m0.001s 0m0.001s 0m0.001s 0m0.001s 0m0.001s .....an so on.
There are lots of values in format like "3m1.057s" are stored in variable_test separated with an space between two such values. For exapmple, value is "3m1.057s" I need to save different parts of a value in three separate array variables such as the
var_hour=3
var_min=1
var_sec= 057
tell if this can be done using "awk". A "WHILE" loop might be used to separate and store theses values I guess?
View 1 Replies
View Related
May 30, 2011
I'm working on a script that will parse a mail machine's logs and print a list of email addresses in this format:
authen@domain.com | sender@domain | recipient@domain
exam
account1@domain1.com | sender2@domain2.com |
[code]...
View 2 Replies
View Related
Feb 1, 2010
I want to know how to get eg. the contents of a form on a webpage which has been passed to a server side PHP script, inside for example an array which I can read. I've been reading a ebook on PHP which as far as I can see doesn't cover this inside it.
View 1 Replies
View Related
Oct 25, 2010
I have spent lots of time trying to figure out how to parse the domain part of this file but am stuck.
How should this be approached:
Code:
Code:
The domains need to be parse at a maximum for 4 levels then the remainder will be put the last field.
View 3 Replies
View Related
Sep 10, 2010
My C foo is terrible! I am working with some code which reads lines from a file and then reformats the lines and writes them to a new file.The input lines look like this:
Code:
+[NEOTEST?]
+[NEOTEST]
[code]...
View 2 Replies
View Related
Apr 8, 2010
bash script:
Code:
$ bash --version
bash --version
GNU bash, version 3.1.17(2)-release (i486-slackware-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.
Code:
[serv:]$ cat test.sh
#!/bin/bash
[Code]...
The question is, how to config it works through 'function cool {}' or cool() {} with position parameter
View 5 Replies
View Related
Aug 28, 2009
language: C
OS: linux
I wrote a multithread program(approx 1000 thread have to run) and each thread has to parse a file(for each thread there is one file, ex:thread1 has to parse file1 and thread2 has to parse file2 like this....). I wrote "parse" program as follows. It is working well, if i create 50 threads. but if i run more than 200 thraeds Im getting doublefree corruption as follows:
Code:
*** glibc detected *** ./simulator: double free or corruption (fasttop): 0xb70007b0 ***
And some time I am getting parsing problem and error af follows:
Code:
powersetting.6607:1: parser error : Start tag expected, '<' not found (where powersetting.6607 is file name, when i check this file it is started with '<').
parse program as follows:
Code:
void parse_first(char* filename, int portnum, struct Poweroff *Powerptr1)
{
struct Poweroff *Powerptr;
Powerptr = Powerptr1;
[code]....
View 1 Replies
View Related
Apr 6, 2010
I've searched online and found many examples using getopts, but nothing that clearly explains it use, nor any examples of what I'm trying to do. I have a script named "process". It can take from 0 to 3 different options. I'd like to be able to handle these options regardless of the order that they are entered.
Syntax:
process [-v] [-d #] [-h|-?] [string]
-v = verbose mode on
-d # = how deep to do the process, expecting a number parameter
-h or -? = show command usage
string = only process lines containing the specified string
[Code]..
View 5 Replies
View Related
Nov 5, 2010
As a curious side project I'm playing with mzXML data(an xml format for holding mass spec data). A typical scan can be quite large, even up into GB size. I'm wondering how would one go about parsing an xml file in sections, one section at a time. The idea being if the computer doesn't have enough memory to load up the entire data file, work on chunks of it at a time.
View 3 Replies
View Related
Aug 11, 2009
Is there a way to process individual characters one-by-one from a text file in Bash, or is that hoping for a little too much from this lovable old clunker?
View 13 Replies
View Related
Jun 1, 2011
Google directed me to the ones written in .Net/C# etc. Any ideas on the ones written in C/C++?
View 14 Replies
View Related
Oct 14, 2010
I have a bash variable where the content looks like this where ;f1; and ;f2; are delimiters:
;f1;field1value1;f2;field2 value1 ;f1;field1value2;f2;field2 value2 ;f1;field1value3;f2;field2 value3
So what I need is to extract and put into variables each combination of f1 and f2 in a loop to something like that:
#first pass of the loop I need:
f1=field1value1
f2=field2 value1
#second pass of the loop I need:
f1=field1value2
f2=field2 value2
# third pass of the loop I need:
f1=field1value3
f2=field2 value3
View 15 Replies
View Related
Jan 4, 2010
I am going to parse the Cisco voip CDR(Call Detail Records), please tell me the field format of the CDR files.
View 1 Replies
View Related
May 29, 2011
PI'm trying to write a script to list all open ports in the MINIUNPND chain in iptables and use the procotol, port and destination ip to open ports on another router using upnpc.Here is the output of iptables -L MINIUPNPD
Code:
>iptables -L MINIUPNPD
Chain MINIUPNPD (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 192.168.3.124 tcp dpt:19955
ACCEPT tcp -- anywhere 192.168.3.124 tcp dpt:20054
ACCEPT tcp -- anywhere 192.168.3.130 udp dpt:10654
ACCEPT tcp -- anywhere 192.168.3.121 tcp dpt:29955
code....
No matter what i do i cant seem to remove the first 4 characters from the MYPROT array to leave only the digits. Also i cant seem to read the array back???
I thought it would simply be a loop reading each line and passing the fields in variables, executing upnpc commands i need then moving to the next line of the file until it reached the EOF.
View 12 Replies
View Related
Mar 16, 2010
I ran into a bit of trouble making a bash script. (Desktop is a directory, and I try to get it's modification date)
Code:
lamp:~# cmd='ls -l Desktop | grep -o "....-..-.. ..:.."'
lamp:~# $cmd
ls: cannot access |: No such file or directory
ls: cannot access grep: No such file or directory
[code]....
When I type in the command directly, without using an inbetween variable, it works fine.
View 3 Replies
View Related
Feb 19, 2010
I have a three input day,month and year. Currently i am using the following script to take the data.
Any grep advanced technique to avoid for loop?
View 2 Replies
View Related
May 27, 2009
We have a sipmle office network set up that we also use use to connect to the internet, however of late the number of users has increased thus slowing internet access. Bandwidth upgrade is not an option thus i have to do bandwidth shaping on our linux router. The question is how do set the squid configs to allow certain IP's range a certain percentage bandwidtheg 60% and furthe divide the rest. Alternatively how can allow certain IPs to have higher bandwidth access.
View 1 Replies
View Related
Aug 23, 2010
I am trying to monitor how long an ldap search takes and maybe notify or something that a search takes longer than say 10 seconds.
Code:
tail -n 1000 /var/log/ldap.log
for SRCH in $( cat monitorldap.log |grep 'SRCH'); do
echo search string is
echo $SRCH
[Code]....
ok, so to start off with it doesn't appear to get the whole line, just a piece "Aug". How can I get the whole line into a variable so I can then cut it up into the pieces I need?
View 4 Replies
View Related
Dec 28, 2010
I'm using Zabbix on which I can use give bash command to the agent.This 1-liner will give me all the interfaces with their IPv4 addresses.I have a 2nd expression which returns a checksum so I can detect a difference whenever someone deletes/adds/changes an ipv4 interface.This is the output on my Ubuntu-server:
Code:
~# ifconfig | grep -B1 ' inet ' | grep -oE '(^[a-z0-9:]*|addr:[0-9]+.[0-9]+.[0-9]+.[0-9]+)'
eth0
[code]....
View 8 Replies
View Related