Ubuntu Servers :: Using Iptables To Get Web Usage Statistics And Filter Urls?
Dec 16, 2010
I'm deploying new ubuntu server which should act as a router. I've already set up the NAT for local network, and also did some shaping for different groups of users, but now I'm facing new problem.I need to make a scheduled URL filter. I know it's not a problem with cron and simple script, but maybe there is existing way to do that? And also, I need to make statistics on web-traffic. I need to have list of URLs visited by users (source ip, destination url). Is it possible with iptables? or with any other software but without using proxy servers.
I'd like to set up an iptables configuration as follows:- Allow all traffic by default- For one user account (anonymous), block all traffic except:- All traffic on lo- All DNS requests, which should be redirected to 127.0.0.1Here's what I tried:
I have a very simple set up.With Network Manager I can have my laptop act as a router (sharing all connections).I also have apt-cacher-ng as a debian package cacher.I would like to set up iptables to filter only the urls that are meant for a debian package cacher. For example:I could use a "forward all" rule:
Code: iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3142 Except then I would get a bunch of error pages every time I tried to do normal navigating.My question (again) is: Can iptables handle forwarding only on a specific url? If so how?Or is there another solution? (prferably without full fledged software like squid)
i wanted to do bandwidth management/traffic shaping on my Internet link(have two internet connections), but i have some questions to ask: I want to know how could i for example filter some traffics using tc and iptables (e.g Peer-to-Peer,IM,Download Managers,Flash videos..) i can do filtering for known services like http,ssh,... but since these applications doesn't use one port, i am confused a little bit. I also want to do some bandwidth allocations (based on protocol) thats why i need filtering.
The other question is that currently i am using tc for bandwidth allocation and iptables for marking packets to send to these classes, am i doing it right? I mean it does work, but is it better to use for example "U32" filters for filtering? P.S: i tried to use ClearOS in gateway mode, but it doesn't have bandwidth allocation functionality. Does anyone know if i could do bandwidth allocation in ClearOS/Endian.
I have a scenario.A domain [URL].. then there are 4 private computers on which applications are hosted at port 80. So when some one from outside access the site it look [URL]..I added
I'm using wget to retrieve a long list of URLs, a small proportion of which fail, hence:
Code: wget --input-file=urls.txt Is there a way to log the urls that have failed? Unfortunatley wget does not output the current URL being processed (and then the status), so hard to see grepping the output helping.
Or should I use some alternative like curl, wmget?
I am unable to restore my iptables from iptables-save after upgrading Fedora. I cannot get iptables-restore to work, and I have resorted to entering rules manually using the GUI.
I've set up a transparrent squid box with two nics. Eth1 = Internet eth0= LAN +Dchp my question is, can I log the data usage of a skype call. My proxy server already records all http an https requests but doesn't record some programs like skype. I know that it is not http traffic, but can I tell my system to record data use by an ip address over a nic with the help of iptables for example?
Can I, with only the use of IPTABLES, limit the incoming bandwith for a protocol? We have for example servers that have a FTP and HTTP server running and whenever HTTP has a lot of connections open, the other uploads/downloads get a timeout. I know I can limit the number of connections but prefer to limit on protocol level. Is this possible using IPTABLES and if so, can someone indicate how to proceed or provide a link? If it's not possible can someone point me to the right tool for the job?
I have a system running at home that uses Getmail to retrieve mail from my ISP's pop server. Dovecot then offers that mail over IMAPs to my desktops running Thunderbird.
The reason I have resorted to using Getmail is because I don't have a static IP (from my ISP) for my server, and thus this server doesn't act as an MX.
I have implemented Spamassassin in my Getmail script as described here.
From further research, I understand that in order to fully utilize Spamassassin 's potential, I have to resort to training it with SA learn.
Currently I still receive spam messages, but 50% of spam is marked as ****SPAM****, and the other half is not marked at all.
My question is this:
1) How do I get getmail to move messages marked as spam by spamassasin to be moved to a JUNK folder within my mailbox automatically?
2) I thought of creating a folder where my users can move messages they deem to be spam, and set up a crontab script to invoke salearn regularly on this folder to get the bayes engine to learn from it. Is this the correct way of doing it?
I've been looking for a way to filter out emails on my postfix server by GeoIP data. I couldn't find anything that fitted the bill so wrote my own in Python as a postfix policy. Thought I'd post it here to see if it's of any use to anyone else. I've attached a tar of the files as the Python formatting will get mucked up by the forum code. Comments/improvements are welcome (be kind ) First file is : policyd-geoip which is owned by root:root and placed in /usr/bin with 755 perms
I am running a basic squid + privoxy combo for web caching/filtering proxy and it works fine. I'm basically running a stock config w/ a few minor edits to allow the relevant hosts access etc.now I am trying to find an easy way to specify privoxy to not filter a specific site (which it breaks). I have tried wading through the privoxy manual + google but I find the config file incredibly complicated. Are there any experts out there who can tell me: what is the easiest way to tell privoxy to 'pass through' a specific website?
I want to install a ShoutCast Server on Ubuntu, but I coudn't decide qualities of the server. I am planning to have 5,000 listeners continuously with 32kbs bit-rate. How would be usage of CPU and RAM ? I have 99MBit bandwidth, I think it is enough.
i am learning to using ubuntu as my server and learning using vps too
now i getting consfuse about my server memory usage i just have 3 sites , 1 blog site and 2 company profile but apache memory usage is more than 300MB and total of memory use in my server is more than 500 MB (maximum 512MB burst memory)
i am using drupal for my website is this normal ? because in last week, memory consumption in my server no more than 380 MB
i always use gedit to write and edit text file under ubuntu. However, i cant find the way to get some statistics, like how many times a word presents in the txt file
I am looking to build a Ubuntu based web filter. What we would like it to do is block access to certain sites for our company. We have had several employees get caught spending hours on end on gambling sites so we would like to restrict access to websites on a per user basis.What I am looking for is a piece of software or suite of software that can filter websites based on a blacklist/whitelist or category based scenario. I need to be able to authenticate users. For example I would like it so that when the CEO logs in he can go to whatever website he wants, while most other staff members are blocked from accessing things in the blacklist or categories.
I remember from a recent trip to a hospital that they had all internet traffic re-routed to their landing page and that you had to agree to specific terms on that page before you could do anything else. Something like that might be useful as well.
i have a pc that is set up as a torrent slave / file server and media pc i want to set i up so that is goes in to suspend when there is no need for it to be on. I wnat it to wake at a set time to run a cron script for me, to check if there are any new torrents out, if there are , download an seed to till rtorrent auto stops the seeding (for me thats set at 1:1), then go back in to standby mode, if there are no active downloads then i want it to go back in to standby straight away.
as its also used as a file server, is there any way to have the shares still visable, with the pc in stand by, then when a user accesses it wake up the server, and if there is no activity after a set time out then go back in to standby. i would also like it to wol when i try to ssh in to it as a htpc is runns moovida, i would also like it to only go in to standby mode if i am not playing a video.
Is there something which can act as a fully fledged proxy (exactly like squid) but which can also monitor data usage?
At the moment what I do is I log data usage of IP addresses (allocated by DHCP) by using IPFM. Obviously getting a new IP address from a DHCP server isn't hard and this could be abused.
So I was thinking if I require proxy authentication and log usage that way, there is no way for anyone to abuse the system.
Does anyone know of a proxy server capable of logging data usage?
I am running latest apache2 available in the lucid repos on my desktop. All packages are updated as of this moment. Now in the root of my web server I have placed several soft links that point to folders on another ext3/ntfs partitions on the same disk. When I try to download any large file (say above 500M)on this server using firefox, when the 'save' window appears, my desktop freezes, I notice very high cpu-ram-disk usage, even though I have not yet clicked on 'ok' to save the file. This issue is not present when the file size is small. Note that firefox and the webserver are running on the same computer.
Also I have tried nginx and lighttpd and the issue is present there as well. When I tried downloading the same files using Internet Explorer 6.0 using a XP VM the issue is not present. However on Windows as well using Firefox the issue recurs.
I am playing with my new 5 disk 2TB software RAID 5 setup and after I had a hard drop out, I deleted the RAID set and recreated it with only 4 drives. I am working on getting the other drive included by doing a grow. (I hope)
The RAID is now rebuilding and seems to average a write speed of around 87 meg/sec, with lows in the high 60's to highs in the mid 90's but mostly in the higher 80's. But it is only showing about 16% CPU usage and maybe a second usage of around 7%.
Is this normal? It seems low to me, but it is a a quad core 2.5 GHZ processor. Screen shots attached.Is it "normal" for software RAID to drop a drive? I thought that was only a hardware RAID issue. Coder68
I've just installed ubuntu 10.04 and the message text that shows when you ssh in shows the disk usage of /home.How do I get it to show the disk usage of the entire root / instead? (like it used to on some older version of ubuntu)
I'm monitoring all kind of things like ( Mem, network, cpu, IOPS,..) But still not found a command where i can see the CPU usage but in MHZ ( or Hz). Using top or looking into /proc/cpuinfo doesn't give me the info i want.
Is there a way to see the global statistics in KTorrent? I'm referring to the overall share ratio and the total amount of data downloaded & uploaded. - not for the current session, but an all-time info edit: KTorrent version 3.3.4
When I try to run anything that uses iptables, even just iptables -L, I get:
Code:
1+drm33.2/modules.dep: No such file or directory iptables v1.4.4: can't initialize iptables table `filter': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded. This is on a fresh Ubuntu minimal install, using my VPS hosts image (so they could have messed something up). I know there have been issues in the past with iptables on Ubuntu on xen. Is this a Ubuntu bug? Is there a solution?
incidentally depmod -a gives:
Code:
WARNING: Couldn't open directory /lib/modules/2.6.32.11+drm33.2: No such file or directory FATAL: Could not open /lib/modules/2.6.32.11+drm33.2/modules.dep.temp for writing: No such file or directory
