i waas wondering if anyone knew of a script or program that removes duplicate words in a txt file. im making an install script and the install list has gotten a bit long so i want to ensure there are no duplicates in the file
View 2 RepliesI am basically trying to remove duplicate words in my <title></title> tag after I got hit by Google Panda. I have around 750 .html files and it will be difficult for to me remove one by one. I am looking for a way to remove only from within <title> </title>
Example of a duplicate title I have:
Code:
<title>Pasta, Pasta Recipe and Pasta Guide</title>
I dont want to replace those words anywhere else in the file except for within the <title>
I want to find and remove duplicate consecutive words from a text file. I've tried working with array but is very difficult..then i've tried using sed...somebody hint me with this sed :
sed ':f;N;$!bf; s/(.*)
1/1
/g; s/(.*)1/1/g'.
It works fine but if i have 3 consecutive identical words it only remove first one and the last two remain intact.
Is their (I assume their is) of searching for and deleting duplicate files that exist in different paths?
View 3 Replies View RelatedI copied a back up of my windows 'my documents' fold and all of its' sub folders into my linux (Mint Debian) Documents directory. I found that many of my files can be found in more that one directory so, what I want to do is to find all the dups and deal with them. Is there a good linux application to resolve this 'duplicates' problem. (I don't want to touch the linux system files.)
View 2 Replies View RelatedI have been messing with diff and grep for 2 days now without result
I am trying to match a file consisting of words to many separate other wordfiles in a specific directory. one by one.
What i want the script to do is to report how many matching words my main file has with every file in the directory, each in turn
setup:
Each of em are plain text files with 1 word per line
Output should be something like:
SCRIPT REPORT:
I recently upgraded to Ubuntu 11 and a few days later my ecryptfs filesystem began misbehaving in a weird way. In my home directory, many subdirectory names are duplicated verbatim. Here's an ls -F excerpt:
Desktop/
Desktop/
Documents/
Documents/
Downloads/
Downloads/
I can no longer access files in those directories (if I ls the directory, it appears empty; I can cd to it, but there's nothing inside). Not all of the directories are duplicated/damaged like this, but most are. A few non-directory files are also duplicated in this fashion, so for example:
[Code]...
Is there a command that could be used to find word content in a file? I.e I want to find all files containing the word 169.254.0.0 in /etc directory.
View 4 Replies View Relatedi have this file, and considering it's obnoxiously huge i'd prefer not to have to do this manually. Is there some way i can manipulate sed or awk to change every other letter in all the words in a file to capital letters?
View 7 Replies View RelatedWhat I plan to do is, create a duplicate file of a text file, and then append some text into the new text file.
View 1 Replies View RelatedI am using Oracle Enterprise Linux version 4 update 7.
Sometimes I have to access large files with thousands of lines in them and I would like to locate a particultar word. e.g.
vi /etc/passwd.
The contents of file passwd are displayed.I want to find a username of joe assuming the passwd file is 2000 lines long.
I would like to use a linux command that will locate joe and highlight in the passwd file as to where word joe is.
Is there a linux command that can do this?
I exported a spreadsheet file into cvs format.
The cvs file is formatted this way
field1,field2,field3,..etc
I want it to be in a Quote delimited format like so
"field1","field2","field3",..etc
I tried Suse five or six years ago and ran into an issue that was not comfortable to work with so I went back to windows. The problem was open spaces between words was not permitted with my music files. I have transferred all of my CDs and LPs to MP3 and have a tremendous number of them and the Suse of five years ago required I convert a title like Foggy Mountain Special.mp3 into something resembling Foggy_Mountain_Special.mp3
I don't care to convert literally a hundred thousand titles to fit the latter format. Does the current version of Suse allow the use of spaces between the words or is the 'no open space' convention still required?
i want to remove words "Max" and "constrained" in a file given below:
Max 0.003745 constrained
Max 0.004549 constrained
Max 0.001689 constrained
[code]....
and further want to replace "Max" by line number so that i can plot the resulting file. i searched in forum, but couldn't do what i wanted to do. e.g. i used
1)grep command
grep -v "Max" inputfile >outputfile
deletes whole line,and hence whole text.
2) sed command
cat inputfile |sed 's/ .{1,12} //g' >outputfile
gives output
0.003745constrained
0.004549constrained
0.001689constrained
[code]....
I am trying to use grep to only tell me files that include both words matching in a pattern file. However when i specify:
grep -f <pattern file> <file>
It pulls out anything that matches one or the other.
Not both.
how to get it to match AND not OR.
I have a text file that gets produced at the end of the script being run.
For this example the text file will produce the following:
Quote:
THE COW
THE DOG
THE CAT
THE HORSE
In the script I am using either echo or printf to print on the screen each line and then it is doing a test and produces a good or bad result.
Another example:
Quote:
THE COW -- IS HOME
THE DOG -- IS HOME
Each whitespace between the -- are 5 spaces. How can I get them to be in-line and formatted when it gets to more words, when THE HORSE arrives.
Example:
Quote:
THE COW -- IS HOME
THE DOG -- IS HOME
THE CAT -- IS HOME
THE HORSE -- IS HOME <-- This has only 3 whitespaces but is still formatted.
i am trying to find all 3 and 4-character length words in my file (which is huge and has alot of entries in it, a big fat wordlist!).My attempt with this regular expression (which I thought should work, found something on length search here: [URL]
cat sorted_noapostrophe.txt| grep '.{3,4}'
but it returns no results? Also to find any words starting with 'f' which are between 3 and 5 characters (inclusive) long, how can this be done?
How can I install the dict file to look works?
I am running SUSE 11.1
Here is the error:
I'm trying to highlight some key words as tailing a log file using a perl script; about my case: I want to search for a keyworrs just once and highligt all occurrence of them. I want just highlit the keywords but not the whole line but the problems are that perl just catches the first keyword in a line and skip checking for other occurrence. for example if a line like "Error: some exception happen, Unable to process" it just highlight the error and do not process the remain part of the line where it should hilight the word "exception" and "Unable"
2-How can I do some action if for exapmle at least 4 time "unable" message appear ( not just in one line but diffrent line)in below is how I use perl search and replace : Code: s/(?:(unable|exception|warning))/e[1;31$&.......
Contained within each of these 67 text files is about 1 million urls. Yes. I have 67 text files that contain 1 million lines of urls each. I am sure I am swimming in duplicates. I tried opening one text file and clicking sort ----->remove duplicates. Now Gedit is not responding my processor is maxed out to 100% and I think I am finally ready to delve into some command line code. Can anyone give me idiot proof instructions on how to sort the duplicates out of each one of these 67 text files? How about no duplicates across all 67?
View 7 Replies View RelatedAnyone know of a good Linux application that will remove duplicate files interactively? I've recently spent a lot of time (read weeks) pruning my music collection, basically by hand. and now I'm moving on to my family photos. Most of the work with the music was done under Windows XP. As for the photos, I have a fantastic Windows application, D'Peg, that I had actually purchased some years ago. This app rocks for Windows. In my opinion it's so good that I would happily pay double the asking price. However, I'd prefer to use Linux if possible, so, what's out there, anything that is worth it's salt? Currently playing around with Picasa.
View 2 Replies View RelatedIs there a program for linux which can show me a list of all duplicate music files in a directory? This will allow me to delete all duplicate files without searching for them manually.
View 9 Replies View RelatedI tried running a back-up/restore script in a WordPress install to migrate from one server to another... long story made short, I ended up doing it manually and all is well on that front
The one remnant from that botched script is that it tried creating a directory 'wp-backup' and then a file inside that directory - but it tried using '' instead of '/'. So what it created was a file named 'wp-backupindex.php' with a file size of 0 bytes.
The problem is thus: I can't change the permissions nor delete the file, because of the invalid file name. I don't have direct shell access (that cost *extra*, of course) and every time I try with the web-based file manager (Quixplorer) it sees it as 'wp-backupindex.php', as though the '' is acting as an escape sequence in the file name. Same thing in FileZilla, I can't do anything to the file without it complaining about the invalid file name.
how to ixnay this one file given the limitations above (no shell access) short of calling and bugging tech support to delete the file for me?
i have a big file of random numbers i generated at some point in time, after working with it with different things(how fun that was)... i want to remove duplicate lines and i'm not sure i'm doing this right
heres the command
Code:
sort random.txt | uniq -u > rand-shorter.txt
the file is pretty big, everything on a new line. i found the command on a web site so i'm sure its correct(bit of a command line in linux newbie)
can anyone confirm if this will remove lines duplicate lines (keeping one copy) and dump what is left in a file named rand-shorter.txt?
EDIT: i think its actually working, just taking a reallllly long time (on an old pen 4 from 2000)
I have a text file that is filled with references to duplicate files. I'm trying to create a text file for each duplicate file found that contains the paths to the duplicates. I would also like the text file names to be based on the size and file name.
Some thing like:
231.5 KB - P&S.doc.txt
138.5 KB - LIMITED#C71.doc.txt
Code:
NamePathSizeLast ChangeLast AccessFile TypeOwnerAttributes
P&S.doc(3 Files)
P&S.docZ:Leg\_Pri_LegPurP&SBUYBarry V231.5 KB11/2/2001 4:07 PM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)Lou_AC
P&S.docZ:Leg\_Pri_LegP&SBUYBarry V231.5 KB11/2/2001 4:07 PM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)DMsC
P&S.docZ:Leg\_Pri_LegPropsPurP&SBUYBarry V231.5 KB11/2/2001 4:07 PM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)DMsC
LIMITED#C71.doc(2 Files)
LIMITED#C71.docZ:Leg\_Pri_LegPurCV138.5 KB12/15/2003 1:04 PM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)Lou_AC
LIMITED#C71.docZ:Leg\_Pri_LegPropsPurCV138.5 KB12/15/2003 1:04 PM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)DMsC
ps revised.8.30.05.clean.doc(3 Files)
ps revised.8.30.05.clean.docZ:Leg\_Pri_LegPropsPurP&SSellVPSummit54.5 KB8/31/2005 11:46 AM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)DMsC
ps revised.8.30.05.clean.docZ:Leg\_Pri_LegP&SSellVPSummit54.5 KB8/31/2005 11:46 AM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)DMsC
ps revised.8.30.05.clean.docZ:Leg\_Pri_LegPurP&SSellVPSummit54.5 KB8/31/2005 11:46 AM11/22/2010 2:38 AM.doc (Microsoft Office Word 97 - 2003 Document)Lou_AC
Copy of 08 Lee All July Billing.xls(2 Files)
Copy of 08 Lee All July Billing.xlsZ:IS\_Sh_ISDevDocDocl 26 upgradeAS6 backup codeAPImport131.5 KB7/30/2010 12:11 PM11/22/2010 2:38 AM.xls (Microsoft Office Excel 97-2003 Worksheet)AdministratorsC
Copy of 08 Lee All July Billing.xlsZ:APKellie131.5 KB7/30/2010 10:03 AM11/22/2010 2:38 AM.xls (Microsoft Office Excel 97-2003 Worksheet)KellieC
Its my first post in here so please be patient I am trying to use regex in perl script to detect allowed words from the file and then print output to the screen.
As an example : I have text file with orders and returns :
Item2-SKU-2-11.08.2010-online
Item3-SKU-3-11.09.2010-return
Item4-SKU-4-11.09.2010-store
My question: is it possible to make sure that i am ony outputing to the screen orders based on few conditions like Item,order form e.g. online.And is it possible to have multiple matches (Item2 only diplay if ordered online etc)
I am using Ubuntu 10.10 and 10.04 on two different computers.I have the same problem with both .... when I delete a file on my hard drive or a removable drive I dont get the space back even after I empty the trash.The file is gone and deleted but its as if its only hidden from me seeing it and still sitting on my drive.For example when I have files on a thumb drive and I delete them and try and put new files on there it will tell me I dont have enough disk space even though all files have been deleted, the only way for me to get the disk space back is to format the drive.I have now realized I have the same problem with my hard drives, I delete files but I dont get any space back, eventually I will have a full hard drive but no files on there
View 9 Replies View RelatedThis has happened twice to me. I'm editing a filename on the desktop, for example, I have a part of the name highlighted and press delete. Inadvertently, I press delete again, but with nothing highlighted. The file is deleted, but is not added to the recycle bin (possible bug).
I believe that is what is happening. I cannot seem to recreate it purposefully on my work computer --I had done this at home this morning while sans-coffee.
Is there a way to recover the files?
In debian/ubuntu I want to:
a) Create a list of all the files in one directory tree
b) Do the same for a second directory tree
c) Compare the two lists such that, only the file NAMES are compared (i.e. just comparing the "file.txt" part so that "/home/folder/file.txt" == "/home/secondfolder/folder/file.txt)
d) Output a list of all the duplicates
How to do this using scripting languages or regex or something?
Trying to remove lines from a syslog text file that have duplicate strings
Mar 10 06:51:11[http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360]
then a few lines down
Mar 10 06:52:03 [http-8080-1] INFO com.MYCOMPANY.webservices.userservice.web.UserServiceController [u:2533274802474744|360] Authorize [platformI$tformIdAndOs=2533274802474744|360, userRegion=America|360
got the same thing in terms of a u: number but the issue is I need to remove duplicates and just leave one and the file has multiple duplicates of different u: numbers and it's 14,000 lines long. can anyone tell me if I can use awk? sed? or sort for something like this to? removing lines that have a certain string in there that's a duplicate.