General :: Merging And Sorting Multiple Files With "sort"?

Jun 3, 2010

I have a bunch of text logfiles in the following format:

ID (17 characters)
Timestamp (14 characters YYYYmmddHHMMSS e.g. "20060210100040" -> 2006/02/10 10:00:40)
Random data (? characters)
end of line

The files are already sorted by timestamp.I need to get 1 log file with all the logs from multiple logs files, sorted by timestamp. Note that the log files are really huge, around 3-4G each (and there are dozens of them) I tried the following command:

sort -s -m -t '|' -k1n,1n +17 -o data_sort.txt *.TXT

Here is how I ended up with this command:

-s : don't bother with tie results
-m : merge all logs files
-t '|' : there is no | in my logs, so the whole line should be field 1

[code]....

Actually... it fails miserably. The output file data_sort.txt is just the concatenation of all files, not sorted at all.

View 4 Replies


ADVERTISEMENT

General :: Bash - Merging Strings (perhaps With Sort | Uniq)?

Nov 16, 2010

I have multiple strings (eg. say two, firstLIST=(0 1 2) and secondLIST=(2 3)) and want to create a single string composed of their unique sorted elements. For the sample strings above, I'd like to build masterLIST=(0 1 2 3).I suppose I could write the elements of firstLIST and secondLIST to files

Code:

echo ${firstLIST[@]} > firstFILE
echo ${secondLIST[@]} > secondFILE

then use

Code:

sort firstFILE secondFILE | uniq > masterFILE

as this gives me a file populated with the elements I'm after, but I'm not sure how to read the elements back into masterLIST... and it doesn't seem "right" to create files to accomplish this. Is there a way to do this by manipulating the strings ${firstLIST[@]} and ${secondLIST[@]} directly? The closest I've come (not close at all) is

Code:

masterLIST=${firstLIST[@]}" "${secondLIST[@]}

but masterLIST built this way has only one element

Code:

$echo ${masterLIST[@]}
0 1 2 2 3
$echo ${#masterLIST[@]}
1

and I don't have access to the individual digits to then try to figure out how to remove duplicates.

View 4 Replies View Related

General :: Merging Avi And Mp3 Files?

Jun 9, 2011

i am totally new to terminal!!! i extracted both the audio and video from an mkv file. both can be played back without any problem, yet i can't merge them into a single one

View 9 Replies View Related

Ubuntu :: 'sort', Not Sorting Some Characters, Just Ignoring?

Aug 28, 2010

i used 'sort' to sort a 160k line file and looked though it and noticed this

RS:HALP
RS HD
RSHD

[code]....

View 2 Replies View Related

General :: Splitting And Merging Two Files?

May 29, 2010

how to split two files like

big.zip.1
big.zip.2

then, my second is how to merge them like windows

in windows, i can use copy /b

in linux,

cat big.zip.1 > big.zip
cat big.zip.2 >> big.zip

View 1 Replies View Related

General :: Merging Two Files From Two Different Directories?

May 14, 2011

I'm totally new to Linux and this website. I was wondering if anyone had or could help me create a shell script that would merge two files from two different directories and then have that new merged file in a third differnt directory.The merged file would need to eliminate duplicates and sort the contents.

View 6 Replies View Related

General :: Merging Columns From Different Files?

Oct 10, 2010

I have three files with the following structure:

file1
1 2 3 4
5 6 7 8

[code]...

View 4 Replies View Related

General :: Merging Files With Different Number Of Rows Using Awk?

Apr 18, 2011

Does anyone have a solution for merging files if the number of rows in the two (or more) files is non-equivalent.To exemplify, how about merging the following 3 files:

file1:
1
2

[code]...

View 8 Replies View Related

General :: Sorting Files In A Directory?

Jan 6, 2010

I have some files in a directory like

file1.txt
file10.txt
file2.txt

[code]...

View 5 Replies View Related

General :: Include Special Characters When Sorting Files By Name (Fedora 12 I386)?

Feb 15, 2010

I am dragging my files over to a new Fedora 12 installation and I just noticed that special characters are not taken into account when sorting files by name (I want '_js' to come before 'images').Is there a way to make the sorting process behave like Windows, where files starting with a special character are listed first?

View 5 Replies View Related

OpenSUSE Multimedia :: Merging Video Files?

Jul 4, 2010

I would like to know which software can merge different videofiles (mpg, avi) into 1 file.
Kino makes a DV-file, which is to big.So I search the equivalent of Microsoft Movie Maker.

View 5 Replies View Related

Software :: Merging Files Together Based On Latex Syntax?

Mar 4, 2010

I have a latex code file which links itself to many other latex files. The syntax is as follows:input{*path of file to be inputed*}The path is relative to the current working directory, so if my file is stored in /home/kevin/mybook.tex and I want to include a file in /home/kevin/latexstuff/copyleft.tex I simply write:input{latexstuff/copyleft.tex}

The latex compiler includes these files just as if they had been copied and pasted into the main latex file at the point specified. My problem is that I have a document which depends on quite a bit of these input commands,ut I am trying to use a latex preprocessor (ratexdb, adds database fields to your latex documentshich does not support input commands, leaving my file only half processed.So I was wondering: is there any easy way to parse through my main file, detect only the input commands, interpret the syntax and include the files specified (where specified)nd produce a second, populated file, which can then be processed by ratexdb?

View 2 Replies View Related

Ubuntu :: Automatically Sorting Files?

Mar 25, 2010

I download files onto my networked file server which is running Ubuntu 9.10.By default, all files are moved into the Downloads folder, I'd like to use a script or program to automatically send them into one of 5 shared folders based on what the file is.For example: archived files would be software by default, .avi or mpeg Videos, and .flac or .ogg would be Music.

View 1 Replies View Related

Ubuntu :: Bookmark Manager - Merging Files Into Single Collection

Jun 2, 2011

During the past eight years I've used a number of computers with different operating systems and browsers. On each one I made a habit of using the bookmark utility of each browser and saving the bookmarks file. I never ensured the continuity of the bookmark file - with each new computer I started a new bookmark file. Even when I was reinstalling the operating system I didn't import the old bookmark file in the newly installed browser: I've always started a new bookmark file. As a result I have tens of bookmark files for Firefox (json format - some kind of xml?) and IE (html file format?) each one containing hundreds or thousands of saved links. I have also some files containing links in text format (created usually when I was using someone else computer).

I would like to be able to manage this bookmarks files by using some sort of "bookmark manager" software. The "bookmark manager" should be able to merge the bookmark files into a single collection/file. It should be able to identify and remove the duplicate entries (I have timely versions of the same bookmark file) and also it should be able to group the entries/links on categories (for example the bookmarked articles on codeproject.com should be grouped under the codeproject category). Not to mention that it should provide a search facility to quickly locate the interesting bookmarks. I couldn't find such software in ubuntu software center. Do you know something that even comes close to what I need?

View 9 Replies View Related

General :: Rename Multiple Files In Multiple Directories/subdirectories Recursively?

Aug 23, 2010

I am to rename all the files within a directory (which contains multiple subdirectories) recursively without invalid characters.

I tried the coding posted above.

find . -type f -printf '%p
' | while read file; do
oldfile=$(basename "$file")
newfile=$(echo "$oldfile" | sed 's/[^A-Za-z0-9_.]/_/g')
if [ ! "$newfile" == "$oldfile" ]; then
echo mv "$file" "${file%$oldfile}$newfile"
code....

but I get an error on both of them stating "find: bad option -printf find: [-H | -L] path-list predicate-list"

View 9 Replies View Related

Ubuntu :: Sorting Files Based On Filename?

Nov 2, 2010

I'm currently trying to organize a media server so that things will be in some kind of logical order rather than the current setup of dumping everything of a certain content type into a single folder. However, the size and diversity of content within these disorganized folders precludes me doing things manually. Does anyone know of a program or script that could sort the files into folders based on part of a filename

View 7 Replies View Related

Fedora :: Automatically Renaming Files While Sorting By Date?

Apr 17, 2010

I often record music from the webradio with Streamtuner.How can I rename the mp3 files recorded in a way that numbers are added to the beginning of the filename representing the order in which the titles were played on the webradio station?I am looking for an automated solution like a renaming tool since there are usually hundreds of files in a directory. I used autorename / ARen for windows back in the day when I was still on XP. What I could do with the tool was:a) Sort the files by date of creationb) add an increasing number at the beginning of the filename based on the position in the sorted list.

View 3 Replies View Related

Ubuntu :: Logical Volume Sorting Individual Files?

Jun 5, 2011

So I just had a quick question on logical volumes and such with ubuntu. I've been looking into setting up a storage array of 4 2tb hard drives for media storage in my house but have ran into a wall with with sort of array i should use, whether it be setting up a full raid system (raid 5 or 10 most likely) or using LVM for stripping. The one thing with LVM however, is that there is no parity or redundancy built into it in case 1 hard drive fails. I was wondering if it was possible to create something similar to that of LVM stripping, but instead the logical volume is sorted into whole individual files, not stripping them across the array. That way, if one drive fails, sure, i lose the contents of the one drive, but the rest of the content isn't lost and I have no loss of space because there is no inherent parity.

View 1 Replies View Related

General :: Creating A Script To Move Or Copy Files Into Multiple Directories Below The Files?

Aug 25, 2009

How can you create a script to move or copy files from a main directory into multiple directories below the main directory.

View 7 Replies View Related

General :: Manually Sort Random Text Files?

Jun 6, 2010

I have about 1000 text files and I need to view each, and move it to a folder if it's the correct one. I can only do basic sorting by length/size, and I can't grep because the text is random. How can I do this besides manually openiing + saving each in gedit.I'm on Ubuntu Linux.I've already done all the sorting I can based on ize,wordcount,greps,date,etc. This is what's left over. I'm trying to find an easy way to view +save/ignore the rest.

View 6 Replies View Related

General :: .EXE Files - Cannot Get It To Install After I Double Click / Sort It?

Jan 25, 2010

I just downloaded a .exe file. I cannot get it to install after I double click on it. What should I do to make it work?

View 6 Replies View Related

General :: Sort Big Files, /tmp/sortA3aLjF: No Space Left On Device

Aug 3, 2010

i'm sorting 8GB text file but came up with this error for an investigation

/tmp/sortA3aLjF: No space left on device

is there anyway that sort won't use /tmp?

View 3 Replies View Related

General :: Save / Sort Files In Sequential Order Sector By Sectyor

Aug 26, 2010

I have a china phone which has mp3 player and unfortunately it reads the file names in its memory card in sequential order according where the file is saved. The file system is NTFS. cod

**Note: DDD song was last because I saved AAA to EEE songs then later added DDD song

Then suddenly i deleted BBB song and replaced it with FFF song code...

This is kinda lame. but the OS of the phone has no capability sorting the file according to filename in its built in mp3 player.

my question is how can I sort the files sector by sector(is my term right?) so the lame mp3 player would read the files finally in alphabetically sorted order. I Will plug my phone on my PC

I am using ubuntu 10.4

View 1 Replies View Related

General :: Reinstall Grub After Merging Drives?

Nov 22, 2009

I used to have 5 drives on my 120GB Hard disk. I had Windows XP and Kubuntu 9.10 installed.today I started merging the drives after I have finished the process I found that grub menu is not loading instead I get something like thisgrub rescue>I am currently using Mepis 7.0 Live CD how can I restore the grub menu so I can boot to Linux or WindowsThe current HardDisk configuration is

Code:
# fdisk -l
Disk /dev/sda: 120.0 GB, 120034123776 bytes

[code]...

View 6 Replies View Related

General :: Sorting A File With A Variable Name?

Jun 27, 2011

I am trying to sort a file, so I can compare it to another file later. I am storing the file name in a variable called curMo. I then call sort $curMo and it hangs for a very long time and I have to quit the process. why this is not working or how to make it work?

Here is part of my code:

#sort this file for this Location
sort -u $curMo -o $curMo.sorted

View 1 Replies View Related

General :: Merging Logical Volume Groups - Initial Setup

Oct 21, 2010

I am very new to LVM, as well as not especially experienced at linux, and have some questions about how lvm works. A few months back I set up a server running FC10 and tried creating Logical Groups during the the initial setup. We've realized that we are not using all the available space on the physical drive, and I realized that for some reason (I'm thinking this might have been the default?), we initially created two Logical Groups (VolGroup00 and VolGroup01) and it appears two Logical volumes in each (LogVol00 and LogVol01). LogVol00 in VolGroup00 is mapped to /, and the other Group was actually unused. I figure that it would be simplest to just use all this space mapped to /, so I thought the thing to do would be to simply merge VolGroup01 to VolGroup00. I tried this:

[root@office mapper]# vgmerge VolGroup00 VolGroup01
Logical volumes in "VolGroup01" must be inactive

So after a bit of research, I tried this:

[root@office mapper]# vgchange -a n VolGroup01
Can't deactivate volume group "VolGroup01" with 1 open logical volume(s)

So apparently There's an open volume, but I don't know how to go about closing it. I removed the LogVol00 from that group, but LogVol01 won't budge.

[root@office mapper]# lvremove VolGroup01
Can't remove open logical volume "LogVol01"

So how do I go about closing this Volume? At one point, there was some output that told me LogVol01 was being used as swap space. How do I handle that?

View 4 Replies View Related

General :: How To Refresh Htop After Sorting (via Shift+m)

Jun 22, 2011

I love using htop, but I have noticed that whenever I sort by a parameter (for example, Shift+m for sorting by Memory usage), the htop stats stop dynamically refreshing. Is this intended? I also notice this with top as well.

View 1 Replies View Related

General :: Sorting A First Non-numerical Column In A File?

Jul 13, 2011

I have a file like:

ER- V67
ER+ V68
ER- V69
ER+ V70

[Code]....

I am using the code:

sort -k1

but it prints it by sorting the second column.

View 8 Replies View Related

General :: Sorting In Aggregate Rather Than Tree Fashion With 'ls -Rt'?

Apr 28, 2010

The command: Code: ls -lRt Shows a recusrive listing, and sorts by modification time. But this in in tree fashion, where it first lists the contents of the current folder sorted by time, and then the contents of each child folder sorted by time. How would one accomplish this type of sort, but with an aggregate listing - all items recursively sorted together, rather than by individual folder?

View 3 Replies View Related

General :: Sorting In Vi Editor & Copying To Clipboard

May 6, 2010

(1)There should be a command to send content to clipboard instead of 36 buffers available in vi just like ':set paste' allows pasting from clipboard inside vi using ctrl+shift+v in insert mode.(Currently I select area using mouse & right click then copy )


2)Let us say I have sentence "I am young" I want to arrange the words in this line in alphabetical order so that I get 'am I young'. First I thought replacing space by in sentence.Then !#j then type sort -n where # represents number of words.Then #J to join the filteredlines.It works but now I have file with hundreds of sentence of varying length.If I make a macro how will it know the number of words?

View 9 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved