Debian Programming :: Consolidate Several Lines Of CSV File With Firewall Rules
Oct 10, 2013
Consolidate several lines of a CSV file with firewall rules, in order to parse them easier?
I have a .csv file, which I created using an HTML export from a Check Point firewall. The objective is to have all the firewall configuration lines where a given host is present. I have to do this for a few hundred, manually is not a reasonable option. I'm going to write a simple Python script for this.
The problem is that the output from the Check Point firewall is complicated to work with. If a firewall rule works with several source or destination hosts, services or other configurations, instead of having them separated with a symbol other than a comma, I get a new line.
This prevents me from exporting the line where the host is present, since I would be missing info.
Let me show you an example, hostnames are modified, of course:
NO.;NAME;SOURCE;DESTINATION;VPN**;SERVICE;ACTION;TRACK;INSTALL ON;TIME;COMMENT
1;;fwxcluster;mcast_vrrp;;vrrp;accept;Log;fwxcluster;Any;"VRRP;;*Comment suppressed*
;;;;;igmp;;;;;
2;;fwxcluster;fwxcluster;;FireWall;accept;Log;fwxcluster;Any;"Management FWg;*Comment suppressed*
;;fwmgmpe;fwmgmpe;;ssh;;;;;
;;fwmgm;fwmgm;;;;;;;
3;NTP;G_NTP_Clients;cmm_ntpserver_pe01;;ntp;accept;None;fwxcluster;Any;*Comment suppressed*
;;;cmm_ntpserver_pe02;;;;;;;
View 0 Replies
ADVERTISEMENT
Oct 11, 2013
I have a CSV file, which I created using an HTML export from a Check Point firewall policy. Each rule is represented as several lines, in some cases. That occurs when a rule has several address sources, destinations or services.
I need the output to have each rule described in only one line. It's easy to distinguish when each rule begins. In the first column, there's the rule ID, which is a number.
Here's an example. In green are marked the strings that should be moved:
See example. The strings that should be moved are in bold:
NO.;NAME;SOURCE;DESTINATION;SERVICE;ACTION;
1;;fwgcluster;mcast_vrrp;vrrp;accept;
;;;;igmp;;
2;Testing;fwgcluster;fwgcluster;FireWall;accept;
;;fwmgmpe;fwmgmpe;ssh;;
;;fwmgm;fwmgm;;;
What I need ,explained in pseudo code, is this:
Read the first column of the next line. If there's a number:
Evaluate the first column of the next line. If there's no number there, concatenate (separating with a comma) the strings in the columns of this line with the last one and eliminate the text in the current one
The output should be something like this. The strings in bold are the ones that were moved:
NO.;NAME;SOURCE;DESTINATION;SERVICE;ACTION;
1;;fwgcluster;mcast_vrrp;vrrp-igmp;accept;
;;;;;;
2;Testing;fwgcluster-fwmgmpe-fwmgm;fwgcluster-fwmgmpe-fwmgm;FireWall-ssh;accept;
;;;;;;
The empty lines are there only to be more clear, I don't actually need them.
View 1 Replies
View Related
Jun 19, 2011
I need to create filename 70-android.rules in the directory /etc/udev/rules.d/I have Adm privileges in my user account properties, but when I use sudo to create this file the Ubuntu OS does not allow me the privilege... I am running Ubuntu 10.04 LTS and here's the Terminal output below:daddy@gatomon-laptop:/etc/udev/rules.d$ sudo cat > 70-android.rulesbash: 70-android.rules: Permission denieddaddy@gatomon-laptop:/etc/udev$ ls -ltotal 8drwxr-xr-x 2 root root 4096 2011-03-16 18:03 rules.d-rw-r--r-- 1 root root 218 2010-04-19 04:30 udev.conf
View 2 Replies
View Related
Sep 17, 2009
I am trying to delete lines of a file if they contain text that is present on another file. For example
> cat one.txt:
a
b
c
d
[code]....
I get the following output:
> ./test.sh one.txt two.txt
a
b
d
e
[code]....
View 6 Replies
View Related
Nov 29, 2010
Even though I've set up HTTPS to be trusted, it still blocks my school's https site: "mnsu.edu/eservices" same with SAMBA and SSH.
If enter the GUI and authenticate as root, change anything and apply, then exit: it works fine and so does SAMBA. However, after restarting, everything stops working again.
yet secure firewall configuration that doesn't require any login or headaches.
View 6 Replies
View Related
Aug 26, 2010
can I deny the access to my server for a specific OS? I have one PC which I want to give it acces from winxp, but if it's boot into ubuntu I want to deny all access to my server, same IP, same ethernet card
View 8 Replies
View Related
May 9, 2010
I have a set of iptables rules generated by Firestarter, and i'm in the process of trying to familiarise myself with iptables itself, but there's one particular rule which is confusing me, perhaps somebody could explain it to me
My INPUT chain reads as follows:
Code:
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- cdns01.plus.net anywhere tcp flags:!FIN,SYN,RST,ACK/SYN
ACCEPT udp -- cdns01.plus.net anywhere
ACCEPT tcp -- cdns02.plus.net anywhere tcp flags:!FIN,SYN,RST,ACK/SYN
[Code]...
Given that the firewall is actually blocking packets, it can't be this simple, so what am I missing?
View 1 Replies
View Related
Apr 7, 2010
I just install 1 firewall using Iptables.
Firewall includes 2 NIC:
NIC1 <IP PUBLIC>
NIC2 192.168.10.1
I installed 1 web server IP: 192.168.10.2
I have some PC IP range: 192.168.10.10->20
I set rules NAT on firewall and PC & web server can connect internet good, but I have problems:
When PC access to web server with IP 192.168.10.2 that ok, but PC can't access to web server when using IP Public. But outside internet, I can access to web server using IP Public.
Rules on IPTables
Code:
# Generated by iptables-save v1.3.5 on Sun Mar 7 21:01:16 2010
*nat
:PREROUTING ACCEPT [950:126970]
:POSTROUTING ACCEPT [89:5880]
:OUTPUT ACCEPT [19:1342]
-A PREROUTING -d 209.99.242.124 -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.10.2:80
-A POSTROUTING -s 192.168.10.0/24 -o eth0 -j SNAT --to-source 209.99.242.124
*filter
:INPUT DROP [1599:157409]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [232:34452]
-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o eth1 -d 192.168.10.2 -p tcp --dport 80 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT
View 2 Replies
View Related
Jul 2, 2010
i have two files with thousands of line, I am trying to combine these two files but i want to insert each line of one file to the another file after certain lines. I am using awk with the following command but it does not work.cat file1 | awk ' { print $0; if (NR%3004==0) {print "file2"}}' > outputfile
View 14 Replies
View Related
Aug 23, 2010
I'm trying to build the package "minitunes" [url] but when I attempt the code:
The build exits with the code dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
This actually happens when I try to build any package with "dh_make" so I'm wondering if it is because of my debian/rules file, which I'm not so certain how to edit.
What should actually be in such a file? I have mine posted below for the minitunes package debian/rules:
View 6 Replies
View Related
Jun 7, 2011
every now and then Firewall Builder fails to open rules (*.fwb)and I have to use some old backup. it does load 'object libraries' but the main 'currently editing policy' panel is empty.(in gnome, debian testing amd64)
View 1 Replies
View Related
Jun 6, 2015
How can we do a file replacing string on debian/rules file using sed and bash variable ? I don't seem to be able to do so. I have tried below under the install section with arch dependent amd64, as far as I know all the bash commands are allowed to be executed in debian/rules file.
I have tried this :
Code: Select all
debian/rules file
ipaddr=`<long command to find ipaddr>`
myVar=`hostname`
sed -i -e 's/somestring/'$myVar'/g' $(configs)*
sed -i -e "s/somestring/$myVar/g" $(configs)*
[Code]...
Nothing works. Sed works but the hostname replacement doesn't work.
View 13 Replies
View Related
Jan 29, 2011
For example, I have a file called "file" like this one:
type=strongsubj len=1 word=absolve pos=verb stemmed=y priorpolarity=positive
type=strongsubj len=1 word=unique pos=adj stemmed=n priorpolarity=neutral
type=strongsubj len=1 word=absolutely pos=adj stemmed=n priorpolarity=neutral
type=weaksubj len=1 word=taking pos=verb stemmed=y priorpolarity=positive
type=weaksubj len=1 word=friend pos=noun stemmed=n priorpolarity=positive
type=weaksubj len=1 word=usually pos=adverb stemmed=n priorpolarity=positive
type=strongsubj len=1 word=purecolor pos=anypos stemmed=n priorpolarity=negative
type=strongsubj len=1 word=accusingly pos=anypos stemmed=n priorpolarity=negative
I want to add the plural for the noun, for example if find this line:
type=weaksubj len=1 word=friend pos=noun stemmed=n priorpolarity=positive
will add one more line :
type=weaksubj len=1 word=friends pos=noun stemmed=n priorpolarity=positive
where we add "s" for the word friend
I did try to do like that:
<code>
cat file | while read LINE ; do
set -- ${line}
if [[ "${4#pos1=}" == "noun" ]];then
#I tried this line but it doesn't work properly:
v3==$(echo $line |sed 's/$3/$s') #I want to find the third word "word=friend" in that line and add "s" after that word
# I don't know what command to add this new line "$v3" to the file ???
done
</code>
View 12 Replies
View Related
Oct 1, 2010
Every now and then I have to indent the lines in my script to 4 space characters. I generally do it line by line. Is there an automated command in vi using which I can indent some set of lines to desired number of space characters in one go.
View 5 Replies
View Related
Mar 17, 2009
I'm trying to find the correct sed syntax to match from a given pattern to the end of the file and then append that pattern to the end of the file.
I was trying:
But that prints each line right after its original occurrence. I want to match the block and then append it to the end of the file.
View 12 Replies
View Related
Aug 21, 2010
I have a plain text file with 360 lines of varying length text. How do I add a comma or other symbol to the end of each line so that I can convert the file to csv format that I can open in a spreadsheet (45 rows, 8 columns). That means each 8 lines of text forms 8 columns, with 45 rows.
View 9 Replies
View Related
Mar 26, 2011
Using awk I pull the first field of a random line from my datafile.myvar1=`awk -F" " 'NR=='$randline' {printf "%s", $1}' myfileThis works fine. The problem is there will be empty lines at the end of the file. Rather than using awkto filter out blank lines I would like to figure this out first.So I test $myvar1 for a blank string after setting $randline to one that I know is blank:test -z "$myvar1" && echo "true" || echo "false"But, this returns "false"? So the string is not zero length. Why? It's a tab-separated file. Is awk storing the tab with the $1 field or something.This is where I get headache. I try to echo my variable to see what it looks like.
echo "$myvar1"
outputs: nothing
echo "My variable is [$myvar1]"
outputs: [y variable is [
Why is the closing bracket at the beginning? What character could be stored in $myvar1 that would do such a thing and how did it get there?
View 7 Replies
View Related
Mar 16, 2011
I am trying to read certain lines within a file and give the output of the certain lines that dont equal my value, I think showing you would be easier. There is multiples of these inside one file...
Code:
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 300lable/syncd
[code]....
I want to read everything in the file, if the status is not available then it should display the name (directly above status). If they are all availbale then do nothing. I think I know how to do it which includes putting the info in string form and placing in hash but it is proving to be out of my skill range.
View 12 Replies
View Related
Sep 10, 2009
To save on the writing of WAY to many files with very little in them, I want to put it all in one file and read a specific few lines. There will be six variables to be read at a time. Format is as such:
//Set 1
string name
5
12
[code]....
From name to 5th number is a set. The name will be of different lengths for each set. This will be a big file of probably 40+ sets. My problem lies in reading one and only one set be it set 5 or set 34. It needs to be done in C++.
View 2 Replies
View Related
Jan 21, 2011
I'm trying to come up with ideas for a simple way to strip a specific "entry" from a text file.I know tools like sed and perl can remove specific lines from a file but I haven't been able to come up with an elegant way to do my group of lines.In my file, the first "Location" line and the "SVNPath" line should be unique every time... but are they enough to strip out the whole set of the group plus the trailing one line of white space separating each group? Add to this, my file will grow as new entries are added (always appended to the end) but new entries will have the same formatting.
View 9 Replies
View Related
Apr 12, 2010
I have two files, where the 1. file has special lines that need to be updated by lines from a 2. file:file1:
foo
foo
foo 0.00 0.00 0.00 pattern
[code]....
View 1 Replies
View Related
Mar 10, 2011
I'm trying to extract specific lines from a flat file. I need lines that fall within a range of coordinates. The -F can be either ! or = If the line is in this set range I need all of the data on that line. ranges lat 36 to 39 and longitude -74 to -84
awk -F '=' '{lat=substr($2,1,2); lon=substr($2,10,3); (lat >36 && lat <39) && (lon >-74 && lon <-84); print lat"--"lon}' < net.log
example line from the flat file
K4MQF-3>APN383,VA2-2,qAR,N3HF-5:!3818.65NS07800.17W#PHG77306/W3,VA3/Clarke Mnt
View 9 Replies
View Related
Mar 11, 2011
For example, I have a text file with data which lists numerical values from two separate individuals
Code:
Person A
100
[code]...
View 1 Replies
View Related
Jun 12, 2009
Eventually, does this program already exist for bash/console ?
View 14 Replies
View Related
Aug 25, 2010
I've been trying to sort this out for several hours and I?m totally lost? I?ve been searching around, but haven?t found the solution to my problem. I have a directory with 100 files. I need to copy 10 lines of each files (let?s say from line 45 to 55) into one unique file. So I guess I could use sed ?w, but I didn?t manage to write the right script. I also tried using a loop to create 100 different files, each one with the 10 lines) to concatenate them later on. But I only got 1 file, not 100.
View 12 Replies
View Related
Jun 6, 2010
I have a file which contains milion of records. It contains 12 columns seperated by "||" (delimeter).
First two fields contain first name and last name of a person. Now my requirement is to delete all those records from this file for which:
First two fields does not contain any alphabet.
For e.g i have below mentioned records in file:
gaurav||gandhi||123||456||789
#a%bcd||123abc||89|90||91
12345||@@@||89||123||234
***||!!!!||98||76||90
Now, last two lines should be removed from this file since first two fields does not contain any alphabet for these two records.
View 12 Replies
View Related
May 19, 2010
I have a very, very large log file (360MB) that I'm trying to thin out. As it turns out the majority of this file has entries that aren't necessary so I'm attempting to build a command that will strip these out. The following command works to display only the data that I do not want:
Code:
cat xml_results.log | grep -B 6 -A 5 "YYY ZZZ"
This displays exactly the data I want to delete from the file by displaying the expression and six lines above it and five lines below it. However I'm at a loss as to how to remove this data from the output and display everything else. I looked into the -v option with grep redirecting the output to a new file:
Code:
cat xml_results.log | grep -B 6 -A 5 -v "YYY ZZZ" > xml_filtered_results.log
However it doesn't work, the new file is the same size as the old one. What am I doing wrong? Is there a better method of doing this? I'm a bit out of my element since the method I'd normally use can't handle files of this size.
View 7 Replies
View Related
Apr 15, 2011
I have a file looks like the following:
digraph topology
{
"192.168.3.254" -> "10.1.1.11"[label="1.000", style=solid];
"192.168.3.254" -> "10.1.1.12"[label="1.000", style=solid];
[code]...
Order of these lines are random... So I cannot delete line #19, for example... And you can see that top four lines I want to delete are pairs. So there might be some clever way to detect the lines, if a line has both "1.9" and "1.11", then delete the line... I am new to perl language. The following is the code I have now... I think I just need to write some code inside the while loop checking if I want to delete the line $dotline before I write to a NEW file.
Code:
#!/usr/bin/perl -w
$TOPPATH = "/tmp";
$NAME = "topology";
$FILENAME = "$TOPPATH/$NAME.dot";
[code]....
View 16 Replies
View Related
Jun 21, 2011
I a csv-file (A.csv) with a total of 4.600.000 lines. Thats to many and only a few is necessary. I have a txt-file with 150 lines (X.txt) (all lines is dataset from a mainframe and looks like abc.def.123.456. How do I remove lines from A.csv where none of the dataset from x.txt is present?
View 13 Replies
View Related
Jun 16, 2010
I have a file with three consecutive blank lines. I want to delete two and keep one.Also, if anyone could direct me towards a guide on regular expressions particularly as they apply to sed, I would be grateful. I am having a hell of a time figuring out the syntax.
View 8 Replies
View Related