General :: Shell Script Or Command To Remove PDF File From Large Logs?
Jul 13, 2011
I need to remove a large binary file(PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the logs smaller and manageable
I need to take out the texts (or binary file) between the strings
<my:PDF> and </my:PDF>
<applicationForm> and </applicationForm>
<image> and </image>
<extractedSignature> and </extractedSignature>
I am not sure whether sed utility can do this, these are large files and need to be pruned .I am not seeking logrotation advice just a script or command that can strip these large logs of texts between the characters above . I am not sure how to do this.These files are rather large.I am not sure how to achieve this with sed , tail, head , tr or any other facility .
1. An external hard disk with VFAT32 file system has a continuous 23GB file (old HD disk image). It is too large to 'remove to wastebasket' and unlike MS Windows remove to wastebasket does not sense file size and wipe file index .
I'm relatively experienced with UNIX and Linux, but this has me thrown for quite a loop, and it seemed like such a simple question. How would I go about finding the newest file in a file system? I thought something like:
Code:
ls -ltr `find /usr -type f`
would work, but I seem to be exceeding the argument maximum for ls:
ksh: 0403-029 There is not enough memory available now
I thought something involving xargs might work, but I really suck with that command.
i want to remove words "Max" and "constrained" in a file given below:
Max 0.003745 constrained Max 0.004549 constrained Max 0.001689 constrained
[code]....
and further want to replace "Max" by line number so that i can plot the resulting file. i searched in forum, but couldn't do what i wanted to do. e.g. i used
I am trying to write a script that connects to a server and executes some commands on there. Something like this:
#!/bin/sh telnet remote_machine cd /home/some_directory cat a_file_in_current_directory
Unfortunately after login/password I guess the script doesn't jump past the telnet command, until I exit. What do I need to do to make the script start executing commands in the remote shell?
Is there tool or a regexp that can convert shell escape characters to HTML code?
As an example, here is a logfile from GNU screen:
Which I would like to convert to something like this:
And send as HTML e-mail to an e-mail address, to archive my work.
Here is a related question, which shows how to convert it to regular text, but it would be nice to convert to HTML and not just throw the escape characters away.
Has anybody else seen this kind of attack? I see those messages on 2 exim mailservers. Looks as if someone sends a 50MB big mail header :S What is their goal except from increasing my traffic?
Code: 2011-02-12 07:48:53 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=ns33.medialook.net [91.121.108.5] input="GET / HTTP/1.1 Accept: */* Accept-Language: en-us
So, my issues since upgrading to Jessie seem to compound. When I fix one issue, two more arise. Right now, I have a full system disk. How it got so full. So I started poking around. I ran
Code: Select all find / -type f -size +50M -exec ls -lh {} ; | awk '{ print $NF ": " $5 }'
Found a few files I could delete, and did, but I also found Code: Select all/var/log/syslog.1: 33G /var/log/messages: 33G /var/log/user.log: 33G
What I find strange is that they're all exactly 33G each. So that accounts for the missing 99GB I deleted them, however only recovered 27Gb. Whats weird is when I type df -h I get
What are the tmpfs's and how can I reclaim that space, and what is /dev/dm-0 and why is that taking up so much space?
I have 2 LVGs vgdisplay -v
Code: Select allroot@SETV-007-WOWZA:~# vgdisplay -v DEGRADED MODE. Incomplete RAID LVs will be processed. Finding all volume groups Finding volume group "WOWZASERVER"
[Code] ....
After deleting the log files, I was able to regain access to my GDM session. But I still cant find out what /dev/dm-0 is, and where all the 75 GB is being taken up.
I just noticed, however, even though I can access the drive A-OK via browser, terminal, and web services (Our wowza) when I enter gParted I get this error for sda, my primary OS drive!
Code: Select all Libparted Bug Found!
Error informing the kernel about modifications to partition /dev/sda2 -- Invalid argument. This means Linux won't know about any changes you made to /dev/sda2 until you reboot -- so you shouldn't mount it or use it in any way before rebooting
Now that I'm in gParted I see 3 partitions: [URL] ....
It reports now, that I have used ALL of my disk space.
Post Log delete, and fresh reboot, this is what Code: Select alldf -h outputs
I want to be able to check the contents of a text file for a specific string and remove it from the file from the command prompt. I would basically be searching through a number of files and if a specific string is found I would like it removed automatically. pretty much a find and replace, were the replace is nothing. any one got any ideas on how you would do this. I already have the search part sorted just need to be able to remove the string I don't want from the multiple files.
I am using RHEL 5.I have a very large test file which cannot be opened in vi.The content of the file has some 8000 lines.I need to view ten lines between 5680 to 5690.How can i view these particular lines in a large file.what is command and option i need to use.
I'm trying to copy a 6Gb file across from my laptop to an external usb drive but it quits at about 4.2Gb every time with a "file size limit exceeded" error. i have checked the output of ulimit -a and there is no limit there on the file size. I'm using the Slax live Cd for this as it always gets the job done
The requirement was to write a shell script for a cron job set for every two hours for all days.The Script has to scan log files (*.log) for the logs posted only for the last two hours.... and append them in a new file...I am clueless abt how to scan/compare based on time stamp seen in above logs.
A colleague gave me a shell script ("dti_motion") which needs to be run from the directory containing all the files it works on. I want to run the same script for several different directories. But I don't want to have to cd into each directory, run the script, wait for it to finish, and then cd to the next directory (there are 52 to do altogether).
So I wanted to write a very simple script that will cd to each directory and perform the script there, before going on to the next one. My colleague's script ("dti_motion") is stored in my home/bin/ and is executable. My home/bin/ is in my path, as verified by echo $path. When run from a directory containing the necessary information, the dti_motion script works perfectly well. I wrote an extremely inelegant script called "dti_motion_do_all" which is also stored in my home/bin/ and executable:
#!/bin/sh #Get motion information for each subject, using Mark's script, called dti_motion cd /imaging/cr01/PD_DTI/C_10/12x5
[code]....
I know there will be more elegant ways to write this script with loops, rather than simply using cd, but for the moment, I just want this to run, until I have learnt to use loops properly. how to correct either of those "command not found" or "no such file or directory" errors, given that both the original dti_motion and my dti_motion_do_all script are in my /home/bin/ (which is on my path) and both scripts are executable?
I need help creating a script that makes a log file in wich to save information about every user that uses the ftp command (information like username and date) and the server to wich he is trying to connect.
I need to send large files from a Linux machine to another using cryptography. The sender machine knows the recipient IP but not vice-versa. I don't need strong cryptography and prefer higher-speed less-secure solutions.
There are no problems with presharing crypto keys but I'd prefer not dealing with SSH users creation.
I think to HTTP PUT over TLS, but I never had experience with it and I prefer to hear which are the possible solutions. I know that it can listen as a daemon but I don't know anything about cryptography. So pipeing with OpenSSL may be a solution.
I am new in perl, i have a question i.e 'How to read individual logs from linux server into another log file using perl script', I need to capture the individual logs from different paths and output the result of those log files and store to a file in another location.These Logs are generated in Linux Server..
I have a file with tens of thousands of lines. I need to remove specific letters eg eggs, from every line that has the letters in it. Is there a command which can help me do that easily?
I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-)
However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit. I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.
Is there a clever way to monitor the progress (as percentage or hash) of copying a large file (using pv could be an option)?Like monitoring the progress of a copy command such as this:Code:cp linux.iso /tmp/
I work for a school consulting company.We helped a school deploy about 1500 computers.The computers have windows XP but we have been using G4L for the restore partition on the drives.So far the software works great. We did however run into a problem in that many of the computers we deployed are missing the restore partition. The reason they are missing is long and convoluted and not really that important. What I have been charged to do is try and fix the restore partition problem. One solution that I had, which im not even sure if it will work, was to backup the recovery file, that g4l created, to DVD and write a basic script to recreate the partition and then copy the file over. This process would need to be as automated as possible since this disc will be inserted by the end user(the students). The backup file that g4l created is 5.9GB so it wont fit on just one disc and Dual layer discs are too expensive to use for this project, so the file will either need to be compressed again (not sure if that's a good idea or not) or split across two DVD's.
I have searched the forums here and I was not able to find anything to fix this problem. I was able to find some info on splitting files across two discs but im not sure how to use that to fix my problem.
After untaring a mysql file (very large) I'm trying to find where the file listed below has gone. I did a search on the file name: fine / -name 'mysql-qui-tools-5.0' -print But can't find the file. -rwxr-xr-x root/root 9651820 2007-05-02 11:46:01 mysql-gui-tools-5.0/mysql-query-browser-bin
I have configured my Cisco ASa Firewall to send its logs to my ubuntu server in /var/log/cisco/ I see the logfiles populating in real time, but i can also see all the logs are also wtitten to /var/log/messages. How can i make sure i do not have a log redundancy? I dont want my firewall logs displayed in messages since there are now sent to /var/log/cisco.
