Programming :: [Perl] Fail To Sort A File With 300,000 Lines By Multiple Columns?

Nov 10, 2009

Each line of the file I am sorting is in the following format:

<url> <month> <day>

For example:

[URL]

I wrote the following to sort:

Code:

#!/usr/bin/perl
$in = shift;
chomp($in);

[code]....

The script worked fine for my small testing files, but failed in my input file. The input file is 18MB and containing more than 300,000 lines. The output will contains some lines like that:

url_one 10 1
url_two 10 1
url_three 10 3
url_four 10 1

Is that because my file is too big for perl to handle ?

View 10 Replies


ADVERTISEMENT

Programming :: Delete Multiple Lines In A File Using Perl?

Apr 15, 2011

I have a file looks like the following:

digraph topology
{
"192.168.3.254" -> "10.1.1.11"[label="1.000", style=solid];
"192.168.3.254" -> "10.1.1.12"[label="1.000", style=solid];

[code]...

Order of these lines are random... So I cannot delete line #19, for example... And you can see that top four lines I want to delete are pairs. So there might be some clever way to detect the lines, if a line has both "1.9" and "1.11", then delete the line... I am new to perl language. The following is the code I have now... I think I just need to write some code inside the while loop checking if I want to delete the line $dotline before I write to a NEW file.

Code:

#!/usr/bin/perl -w
$TOPPATH = "/tmp";
$NAME = "topology";
$FILENAME = "$TOPPATH/$NAME.dot";

[code]....

View 16 Replies View Related

Programming :: Unable To Match Across Multiple Lines In Perl

May 11, 2011

I'm trying to split a text file into various parts. Everything in between "123" and "break" (including linebreaks) goes into the splitted file.

e.g. using this text file:

This should split into 4 files. However I'm only getting 2 files: one for the line "123break" and one for "123 blah break". The two occurrences that contain linebreaks are being ignored. The .* part of my match should capture linebreaks seeing that I'm using the /s modifier shouldn't it? Even when I use the match /(123 break)/gs it still doesn't capture the first occurrence. I'm using Perl v5.12.3 (from ActiveState) on Windows XP. The text file is also in Windows format.

Code listed below.

The above code generates two files Output_1.txt and Output_2.txt which contain "123break" and "123 blah break" respectively. I want it to generate four files.

View 4 Replies View Related

Programming :: Perl Regex Not Matching Across Multiple Lines Despite Ms Flags / Fix It?

Aug 16, 2010

I have written a regular expression (tested in regexpal and regextester alpha something) with which I want to replace something like code...

but it only matches functions which occupy one line only, despite my tests showing multiple line matching in javascript testers online and using the m and s flags (which should make it multi line no?)

View 14 Replies View Related

Programming :: Reading Lines Within A File (Perl)?

Mar 16, 2011

I am trying to read certain lines within a file and give the output of the certain lines that dont equal my value, I think showing you would be easier. There is multiples of these inside one file...

Code:

LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 300lable/syncd

[code]....

I want to read everything in the file, if the status is not available then it should display the name (directly above status). If they are all availbale then do nothing. I think I know how to do it which includes putting the info in string form and placing in hash but it is proving to be out of my skill range.

View 12 Replies View Related

Programming :: Perl Script Output Can't Be Redirected To A File / Sort It?

Feb 18, 2010

The perl script I wrote works fine if I print the result to screen
x0_amber.pl 1 1000 0 5

But whenever I want to output to a file with
x0_amber.pl 1 1000 0 5 > x0_out

it never really prints out to the file.

It worked earlier, but I was playing with my PATH lately, I don't know if it's related to that

View 7 Replies View Related

Programming :: Read Multiple Lines From A Text File?

Mar 11, 2011

For example, I have a text file with data which lists numerical values from two separate individuals

Code:
Person A
100

[code]...

View 1 Replies View Related

Programming :: Copy Lines From Multiple Files To One File Using Sed -w?

Aug 25, 2010

I've been trying to sort this out for several hours and I?m totally lost? I?ve been searching around, but haven?t found the solution to my problem. I have a directory with 100 files. I need to copy 10 lines of each files (let?s say from line 45 to 55) into one unique file. So I guess I could use sed ?w, but I didn?t manage to write the right script. I also tried using a loop to create 100 different files, each one with the 10 lines) to concatenate them later on. But I only got 1 file, not 100.

View 12 Replies View Related

Programming :: Deleting Multiple Lines Above And Below Expression In File

May 19, 2010

I have a very, very large log file (360MB) that I'm trying to thin out. As it turns out the majority of this file has entries that aren't necessary so I'm attempting to build a command that will strip these out. The following command works to display only the data that I do not want:

Code:
cat xml_results.log | grep -B 6 -A 5 "YYY ZZZ"

This displays exactly the data I want to delete from the file by displaying the expression and six lines above it and five lines below it. However I'm at a loss as to how to remove this data from the output and display everything else. I looked into the -v option with grep redirecting the output to a new file:

Code:
cat xml_results.log | grep -B 6 -A 5 -v "YYY ZZZ" > xml_filtered_results.log

However it doesn't work, the new file is the same size as the old one. What am I doing wrong? Is there a better method of doing this? I'm a bit out of my element since the method I'd normally use can't handle files of this size.

View 7 Replies View Related

Programming :: Split A File To Multiple File Using Awk Or Perl?

Feb 27, 2011

I have a single file that contain multi-text something likes this:

Quote:

No. Time Source Destination Protocol Info
185 27712.068199 192.168.18.23 192.168.18.191 SMTP S: 250 2.1.5 Ok
No. Time Source Destination Protocol Info
186 27715.068293 192.168.0.50 192.168.5.2 TCP suncacao-jmxmp > 44693 [ACK] Seq=1 Ack=1 Win=64807 Len=1380

[Code].....

View 14 Replies View Related

Programming :: (BASH) How To Read Multiple Lines From Text File

Mar 11, 2011

For example, I have a text file with data which lists numerical values from two separate individuals

Code:
Person A
100
200
300
400
500
600
700
800
900
1000
1100
1200

Person B
1200
1100
1000
900
800
700
600
500
400
300
200
100

How would I go about reading the values for each Person, then being able to perform mathematical equations for each Person (finding the sum for example)?

View 13 Replies View Related

Programming :: Convert Multiple File In Directory - Ascii To Hex In Perl ?

Apr 9, 2011

I have found a perl script that can convert single file: ascii to hex.

However I have thousand of file that I want to convert from ascii to hex.

Here is the perl script that convert single ascii file to hex in single line:

Quote:

So I would like to read multiple file from a directory.

Then the file will be have same name file with hex data.

Here is sample of the read and write directory file.

Quote:

View 3 Replies View Related

Programming :: AWK - Combining Multiple Columns?

Feb 23, 2010

I have a folder with only 24 files named <number>.dat (i.e. 4.dat, 6.dat and so on) where <number> is between 0 and 256. Each file has just two columns of data and nothing else.

I'm trying to combine all the second columns ($2) together. I've been fiddling around with getline and so far have

awk '{ getline ln < "6.dat" ; print ln" "$2 }' 4.dat

which takes file 4.dat and adds $2 from 6.dat, but I want a single command to take each $2 from every file and add them to (for example) 4.dat (having $1 from 4.dat is no problem). A command that takes every file in the folder and grabs $2 and places them in a common file would be ideal. Frankly I can work around if you combine both columns from every file.

View 5 Replies View Related

Programming :: AWK (or TCL/TK): Matching Rows And Columns Between Multiple Files?

May 26, 2011

I've been hitting my head against a wall for awhile with this one:As the last part of some data analysis I performing I would to construct a matrix from a series of different files. These files have the format:

file 1 file 2 file 3
AAAAA .1 AAAAA .1 BBBBB .1
BBBBB .2 BBBBB .1 CCCCC .9

[code]...

View 6 Replies View Related

Programming :: Blank Lines In A Perl Script?

May 4, 2011

there is a way to add line spaces when asking for user interaction in a script. For example:

Code:

SPACE
Hello what is your name?
SPACE
SPACE

So this is asking a question but has a space/empty line at the top of the screen and 2 spaces/empty lines below. I've seen it done in a bash script using for each line/space needed

Code:

echo ""

So what does perl use?

View 5 Replies View Related

General :: Bash Script To Read Csv File With Multiple Length Columns

Jul 27, 2011

I've searched everywhere and I can't come up with a good solution. For each line I need to find the average, min, and max. I've seen plenty of solutions where the number of columns is fixed, unfortunately for me these lines can get pretty large. My thought was to read each line individually into an array, loop through the array and find the avg, min, and max that way but i haven't had much luck. I can read each line using a while loop but I'm having trouble with the array part, or perhaps that's not the best solution?

View 14 Replies View Related

Programming :: Perl One-lines Within A Ksh Script In Aix Environment Not Execute?

Jul 19, 2011

I have a script to utilize the perl one liner to replace a ip address.

Code:

#!/usr/bin/ksh
set -x
PRIP=172.15.100.176
OPRIP=172.15.100.115

[code]....

the script runs fine. but the perl one liner seems not working at all. but if i copy the perl line to a command line, it runs correctly.

View 2 Replies View Related

Programming :: Sed - Awk - Perl - Merge Lines Unless They Match A Certain String

Apr 15, 2011

What is the best way to merge lines, in sed, awk or perl, that occur between certain strings? I'm new to sed scripting and I have been working on this for some time now. I have a large file (sample below) that I need to edit.

What I need looks something like this.

I'm working with a very large file so simply merging all the lines then adding a new line character before ">contig" and after "translated" won't work, at least not with sed.

View 5 Replies View Related

Programming :: String Concatenation In Perl Result In Two Lines - Not One?

Jan 11, 2010

I am new to perl and am having trouble adding some strings together.

My full code is below:

The problem is $NewCommandB is always split into two lines, where the second line contains the "/atlas2/<blah>/<etc>/..." string. Since I am generating a .sh file to execute a lot of similar commands I need the string to all be on one line. Any idea why I get this behaviour and any suggestion on how to tell perl to make $NewCommandB a one line string?

Btw for completeness finalFileList.txt contains just file names one line after another:

View 5 Replies View Related

Programming :: Sort Hash Of Hashes By Value In Perl?

Dec 1, 2010

I have a hash of hashes and I need to sort this by value and write to a javascript array file..Currently I am using the following

foreach my $key(sort { keys %{$trans{$a}} <=> keys %{$trans{$b}}} keys %transmission)
{
foreach my $role(sort {$trans{$key}{$a} cmp $trans{$key}{$b}} keys %{$trans{$key}})

[code]...

View 2 Replies View Related

Programming :: Randomly Selecting 160 Lines Within A Data Set Of 1000 In Perl

Apr 11, 2011

I have a dataset of around 1000 lines. Out of these 1000 lines I need to pick randomly 160 lines of data and write it to a file. The program is needed to eliminate data bias when I run the program through a reanalysis program. I am thinking I need to use the rand or srand term, but I am having difficulty writing this in perl. I have to write it in perl, because the rest of my scripts for this project are in perl, so consistency is important. The data only consists of one column of the data (YYYYMMDDHHHH).

View 2 Replies View Related

Programming :: Perl String Question:extract Lines From Somefile?

Jun 19, 2011

what's the equivalent code of Perl?sed -i 777,2545!d somefileObviously it extract lines of somefile between that 2 digits .

View 1 Replies View Related

Programming :: Perl Script Get Environment Variable LINES= For Current Session?

Apr 1, 2009

Want to return current LINES=, tryng to avoid the "hacky" backticks and string manipulation way

ie: my $somevar = `set | grep LINES`

I tried using the env command and it returns a empty value?

my $lines = $ENV{'LINES'};

$= doesnt work well for putty sessions...

my $lines = $=;

View 6 Replies View Related

Programming :: Iterate Through The File And Print The Different Columns Each Time?

Nov 27, 2010

How do I do this?The script below obviously dosn't work.

#!/bin/bash
for u in $(awk '{print $2}' user.txt)
for i in $(awk '{print $1}' user.txt)

[code]...

One text file with two columns in it, I need to iterate through the file and print the different columns each time

View 9 Replies View Related

Programming :: Inserting Multiple Lines - With Newline - Using Sed Or Awk

Mar 8, 2009

I need to replace part of a line in a file with multiple lines, however need to separate each line with a newline is this possible?

Example, need to remove line 2

And insert line similar to below:

If the multiple lines were part of a text file, would inserting a text file automatically include newline characters?

View 1 Replies View Related

Programming :: NSH Shell Scripting \ The Value For The Vatriable Would Be Coming From A File With Three Columns?

Jun 24, 2011

I'm writing a script and I have doubts on how to assign values to an already established variable. The value for the vatriable would be coming from a file with three columns. I'm using the awk command for this. Am I doing it correctly? which of the following two ways is the better one or if both are wrong which one should I use?

#!/bin/nsh
inputfile=$1
rolename=$2

[code]....

View 2 Replies View Related

Programming :: Indexing Multiple Drives With Perl?

May 7, 2010

I have 4 separate drives, all un-RAIDed.Each drive has almost the same root directories, but different contents in each folder.

For example:

/mnt/sda/TV Shows/Lost/
/mnt/sdb/TV Shows/Your_Favorite_Show/

The goal is to have a single folder that has symlinks to all the files in each of the drives. Pretty much a poor man's JBOD. Previously, I had problems with conditions like 2 drives having the same sub folder contents, but I ended up solving that with the current script I'm using now.What I'm looking for now is speed. I'm very new to Perl and the script takes about 12 minutes to complete with the current drives.

Basically, the script makes a list of all directories and files in each drive. First, it makes the directories. I didn't use any validation because if a directory already exists, it simply won't make one. However, with the files, I used a hash to only keep the unique files. Then I use the key/value pairs with ln to create every link to the files only, not directories.

Code:

#!/usr/bin/perl
use warnings;
my @drives_to_sync = qw ( /mnt/sda/ /mnt/sdb/ /mnt/sdc/ /mnt/sdd/);

[code].....

View 2 Replies View Related

Programming :: Perl - Multiple Child Processes In Parallel?

Sep 3, 2010

I'm looking for a way in Perl to be able to take a list of servers, ssh multiple commands to it and store the results. If I do this process serially, sometimes one server will hang the whole script and if it doesn't, it still takes hours to complete.

I'm thinking what I need to do is make a parent loop that calls out a separate process that passes the server name to the child sub process and then executes all the commands I have defined in its own process. If one server 'hangs', at least that won't stop the script from doing all the other servers in the list.

I'm guessing using the fork() command would serve me best, however, all the online descriptions I have found have been vague at best.

View 4 Replies View Related

Programming :: Perl's Length() Counts Umlauts Multiple Times

Dec 14, 2010

I'm programming some skript to get statistical information about some texts. This includes calculating the mean of word lengths.Unfortunately, Umlauts count as two characters. In the example below the output is 9, it should be 6.

sincercly, Max

Code:
#!/usr/bin/perl
use POSIX;
use locale;
my $test = length("ABC���");
print $test;

View 4 Replies View Related

General :: Grep Multiple Lines From A Text File

Jun 17, 2009

I have a list of words that I want to grep in many files to see which ones have it and which ones dont. in the text file I have all the words listed line by line, ex: list.txt:

check
try this
word1
word2
open space
list ..

I want to grep each line one by one. like I want it to

grep "check" *.log
grep "try this" *.log
grep "word1" *.log .. etc how can I do this?

and maybe write the output to a file.

View 5 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved