I am looking for a way to search for large numbers in text files and print the nearby lines.
For example if I had a text file like:
Event: 11
blah: 3
blah: 41 bleh: 19
Event: 2
blah: 31
[Code].....
I have two NASes. I work off of one, and the other is used as a backup. As I have it set up now, it's slow. Running a backup takes a week. Even for 7 TB, with 1,979,407 files, this seems a bit outlandish,particularly as both systems are RAID-5 and the network is all gigabit. I've been digging about in the rsync man pages, and I really don't understand what differentiates the various topologies.Right now, all the processing is being done on the backup NAS, which has the main volume from the main NAS mounted locally over SMB. I suspect that the SMB overhead is killing me, particularly when dealing with lots of files.
I think what I need is to set up rsync on the main nas as a daemon, and then run a local rsync client to connect to it, which would hopefully allow me to completely avoid the whole SMB-in-the-middle affair, but aside from mentioning that it's there, I can find very little information on why one would want to use the daemon mode for rsync.
Here's my current rsync command line: rsync -r -progress --delete /cifs/Thecus/ /mnt/Storage/input? Is there a better way/tool to do this? Edit:Ok, to address the additional questions: The "Main" NAS is a Thecus N7700. I have additional modules installed that give me SSH, and it has rsync, but it's not in the $PATH, and I havn't figured out how to edit the local $PATH in a way that persists between reboots. The "Backup" NAS is a DIY affair, built around a 1.6Ghz Via Mobo with a Adaptec Hardware RAID card. It's running CentOS 5 with a full desktop environment. It's the hardware I'm running rsync from. (Gigabit is through a additional PCI card).
Further Edit: Ok, got rsync over SSH working (thanks, lajuette!).I had to do a bit of tweaking on my command line, I'm running rsync with the args:rsync -rum --inplace --progress --delete --rsync-path=/opt/bin/rsync sys@10.1.1.10:/raid/data/Storage /mnt/Storage (Note: I'm specifically not using -a, because I want to change the ownership to the local account, to not freak-out SELinux)
Is it possible to search ALL files on a system for a specific text string? I've been messing around with grep without any luck.
View 2 Replies View RelatedI am a member of a group which has written a program whose source code is being held in a specific directory (~cs252/Assignments/basicAsst/project) and we want to go through and change the parameters for the function "sequentialInsert." My job is to find all occurances of the function call to "sequentialInsert" and to also list the files from where the code came from. Also, I have to be in the commandsAsst directory when I do this. I have tried grep and find combined together, and I am at a lost.
View 1 Replies View RelatedFor my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size. Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.
View 7 Replies View RelatedI want to automate this using script.How to automate it?
File1:
s.no# 1 name:aaaaaa
city:abcd
[code]...
I need to fgrep a list of things which are in a file. The file in which I will do the SEACHING is a large text file and I need fgrep to output each item from the list as a file with the item from the list as the file name.
Its kinda like this:
./script list.txt largefile.txt
output would be
jack.txt
screen.txt
blah.txt
am writing a small search program for my class. I have decided to use indexing for my program. Ive researched online about indexing and how search engines do it. If im gonno do that I need to create inverted files to associate files to numbers ( numbers being the index of my paths ) . Now I was wondering what would be the best way to create an inverted file ? I was going to create sql tables using mysql api in C but then again there is no array data type or vectors to store few numbers in a single column in mysql and it is not advised to use Enum or SET
View 14 Replies View RelatedI am on Ubuntu 11.04 and using Libre Office 3.3.2 to compose new documents and am saving them using .doc, .ppt and .xls files. (due to having to share them with others who are on Windows systems)
I have a lot of doc files and I need to search for text INSIDE these files. I am perplexed with the fact that no search tool is able to search for text INSIDE these file types. "cat" can display them of course, but grep is not able to locate text INSIDE these file types. I even tried to save a .doc file as an .odt file, but no luck. The Applications>Accessories>Search for Files does not search INSIDE doc, xls or ppt with the option "Contains the text".
I'm working with a rather large file of data taking from a tracking program on my phone, and trying to pull only the longitude and latitude from it. Any given line in the data looks more or less like this:
{"lon":-122.3083848,"time":1.273515839496E9,"source":"skyhook","nap":28,"altitude":0,"name":"location","hpe" :29,"bearing":0,"ncell":0,"lat":47.6544453}
I've run it through this command:
grep skyhook log-2010-05-10_18-17-28.json | cut -d"," -f1,10
to get this:
{"lon":-122.30872015,"lat":47.65812201}
{"lon":-122.3076686,"lat":47.6569975}
{"lon":-122.3079161,"lat":47.656395}
{"lon":-122.3096168,"lat":47.656218}
{"lon":-122.3096285,"lat":47.656206}
Which is a lot nicer, but I would prefer not to have to hand remove the non-number characters by hand since there are thousands of data points. what I could do to get it to just be longitude and latitude in 'number number' format?
I'm writing a bash script where I read a text file (containing a column of numbers) and store each line in an array. There seem to be some problems with the whole thing however, but only for some files and not others. Here's what I do:
Code:
#!/bin/bash
file=time_notOk.txt ### The file with a column of numbers
i=0 ### Array counter
### Read the file
[Code]....
I have 24" dual monitors with 1920x1080 resolution on both of them. Consequently the text appears so small. I use the following text-intensive applications frequently:
Web browser (Google Chrome)
IDE (Komodo)
Terminal (Gnome Terminal)
Email (Thunderbird)
I can configure text size on IDE, Terminal and Email. But for Chrome, it is not a good idea to set proportional font size because often one wants to see the entire (not just proportional fonts) site to be zoomed. So I am asking: Is it possible to increase DPI in Ubuntu (much like on Windows) so as to increase the text size across all apps? OR Is it possible to set permanent 'zoom' in Google Chrome, using a third-party extension maybe?
I have on my windows machine several hundred files that are a format of .nc .ncs for a CNC machine. I need to convert them to txt which is something as easy as opening in notepad and then saving as .txt but there are so many that this kind of action would take way too long.
The reason I am writing the linuxquestions is because I would feel more comfortable in loading a live CD and using some sort of terminal command to do this than I would to download one of the many "freeware" type programs I have found for windows (even more so since I have had a root kit before and had to start all the way over to get rid of it).
I need to know:
1. Is this possible to do with the terminal without super advanced knowledge.
2. Can one please point me in the right direction; something to read or an example
I'm trying to manipulate a large text file full of records (metadata - one complete record per line). I need to delete every line on which certain words appear - there are five different words, all pretty simple all-caps strings with occasional whitespace. I tried using grep -v, which worked a treat, but only string-by-string. Ideally I'd like to run this as grep -v -f, where the file targeted by the -f contains the strings I need to match in order to delete the lines they're in.
i.e. grep -v -f filecontainingSTRINGS.txt targetfile.txt > outputfile.txt
When I try this, however, I don't get any matches - or more specifically, no changes are made in the output file. It works fine if there's only one string in filecontainingSTRINGS, but it doesn't work if there's more than one (I'm using newline as the delimiter). (Also my machine doesn't recognise /usr/xpg4/bin/grep - no idea what that's all about!)
I want wondering if I can ls files that start with numbers only in a folder.
Suppose:
001.txt
302.txt
[code]....
I am looking for all the files that contain the text string 'moo.sql'. I ran the following:
find . -name '*.php' | grep -lir 'moo.sql' *
Unfortunately it seems to return non-php files in addition to php files. I thought the find portion of this would filter the file names so grep would only search php files.
I'm a frequent user of grep. I know that I can recursively search a directory using the -r flag:
Code:
// will recursively search all files
grep -r 'some string' *
However, if I want to limit my search to PHP files, the -r flag is suddenly useless:
Code:
// for some reason, this only searches the PHP files in the current dir
grep -r 'some string' *.php
Any good way to recursively search a directory and its subdirs for a string but ONLY look at PHP or HTML files (and possibly TXT files too) ? I'm really hoping for a nice, short command that doesn't involve using an exclude file and which isn't really painful to type. I do this kind of search very frequently and have resorted to either searching EVERY file which is really slow (TAR and ZIP files really slow it down) OR typing repeated commands to search *.php, */*.php, etc.
I am trying find files in a directory that contain numbers. I have tried ls /etc *[0-9]* but that doesn't work. If I cd to /etc and run ls *[0-9]* it almost works but it also includes results from within files. My last thought was to try: find /etc [0-9] -type f but this does not work either. My second problem is that I am trying to get list of files in a directory that were changed less than 10 hours ago, using grep, while leaving out directories. I am completely stuck with the second problem.
View 8 Replies View RelatedI have word like initialize_my_var:in sample.php and I included three library files, take it as a.php, b.php, c.php ,I really don't know where my label(initialize_my_var:)definition is present in my library files, is it possible with a pattern matching string to find which library file really have the exact term "initialize_my_var:" , I'm really looking for an exact pattern match.
View 1 Replies View RelatedI was trying to develop a script which needs to check the count of files on hourly basis and if it find any addition it has to sftp and send a email on the status with filenames and number of files copied via sftp. I will put it on cron to run every hour.
I'll use ls /abc|wc -l to count the no. of lines for the first time and from then whenever a new file will be inserted it'll copy that file to another location or I'll take the date of the files and whichever is having a new date that will be copied to another location.
I am writing a shell script that finds all files named <myFile> in a directory <dir> or any of its subdirectories, recursively. I also need to take care of symbolic links that may form cycles, to avoid infinite loops. I am not supposed to use find command for the same
I started writing the code but got stuck. I thought using recursion may be a smart way, but its not working.
Code:
#!/bin/sh
findFiles()
{
thisDIR=$1
#cd $thisDIR
code....
I just want to know if we can search for a particular string in all files of a particular folder in linux. if a folder has 100files only that hundred files needs to be checked for a particular string.
View 5 Replies View RelatedIs it possible for listing directory and files' permission into numbers?
Generally ls command shows directory and files' permission as follows;
[root:maestro:/home/ddisk_net]# ls -al
Total 44
drwx------ 4 ddisk_net users 4096 7월 20 10:40 .
drwxr-xr-x 21 root root 4096 7월 20 10:28 ..
-rw------- 1 ddisk_net webteam 5 7월 20 10:40 .bash_history
-rw-r--r-- 1 ddisk_net users 33 1월 22 2009 .bash_logout
[code].....
I have PDF files that I would like to be able to search the text of. I tried the GNOME Search for Files program (Places->Search for Files...) but it didn't find the PDF files that contained the text. What do you use to search for text in PDF files?
View 2 Replies View Relatedmay be an advanced question but I need to know how to do this. Here at work I am in charge of recruiting and we have about 1,000 resumes in already. All of the resumes are in a .pdf format. I need to rename every .pdf in the following format:{firstnameLastname}.pdfThe only way I know how to do this is to convert all the .pdf files to text, extract the name out of the first few lines of text, import into excel, and then use VBA to rename the files in mass:Here is my logic so far:~Deskop/a = houses all the .pdfresumesOpen terminal: Code: cd ~/Desktop/afor f in *.pdf; do pdftotext -raw $f; done That will convert all of the preceding resumes into text filesNow I would like to append the name of the text file into the last line of the text file. So, for example, for Resume1.txt, I want to append "Resume1.txt" to the last line within Resume1.txt. So after I run the command I open Resume1.txt and on the last line within I want to see "Resume1.txt" on the last line, at the end of the resume.How can I do this? I would like to use a loop and have the terminal append the filename to the body of the text file until all of the have been appended.
View 1 Replies View RelatedCurrently, when I'm searching text in files of my PHP project, I use this line :
Code:
grep -r 'myTextToFind' *
But now, I would like to search only in ".lang" files. How can I do that ?
Code:
#!/bin/bash
VARR=`cat /proc/asound/cards | grep HDMI | cut -c 1-2`
VARX="defaults.ctl.card $VARR"
VARY="defaults.pcm.card $VARR"
FILE1="alsa"
FILE2="alsa.new"
echo $VARX
echo $VARY
sed "s/defaults.ctl.card*/'$VARX'/g" $FILE1 > $FILE2
This is what I have right now. Well, I thought I knew sed, and apparently I don't... I tried writing this for someone else, and this has given me trouble, so since the user pretty much figured it out on his own, here it goes.
Say VARR=1, so VARX and VARY contain the above text, appended by 1.
What I am trying to do is replace the text "defaults.ctl.card 0" by VARX and "defaults.pcm.card 0" by VARY. The contents of FILE1 is the file being used to search for both text fields, and FILE2 is the output file. I tried using single quotes, double quotes, and a mixture of both, and no go whatsoever. So my question... What is the proper way of searching for text within a file and replacing with a variable?
I need sed to be able to search a string that includes both single quotes (') and double quotes ("). can anyone help me out, there has to be a way to do this.
So far I have tried:
But none of these work and I cannot think of how else to escape the sed quote inside of brackets.
I need to download some very large files (circa 75 GB) from a remote server via SFTP. I've been using SFTP via the command line on my Linux netbook. Around halfway through, the transfer stops and says "stalled." Can anybody recommend a reliable way to download these files?
View 1 Replies View Relatedhow I can randomly write / create a 1 GB file in bash to test disk / network i/o? I was told I could use the 'dd' command but I don't know if there are some better ways and or what the 'dd' command looks like.
View 7 Replies View Related