Programming :: BASH Script Optimization For Testing Large Number Of Files
Sep 18, 2010
I want to move files from a $SOURCEDIR to a $DESTBASE/$DESTDIR. Under $DESTBASE there are many directories, and I need to test beforehand if a file from $SOURCEDIR already exists in any of them.
This is obviously extremely slow, and the real use case involves dozens of dirs and thousands of files. Creating a temporary "index" file for the find command (instead of running it every iteration) speeds it up a little, but it's still very clumsy.
I have no Bash skills but Im badly in need of a script to move the first 20,000 or whatever number of files from a directory containing over 200,000 files to a new directory. The problem is that I cant access the directory because its so large so I want to break it into chunks, but keep the files in order if possible. If you know of a script to meet my needs, please post it. Otherwise Ill post my fumblings with Bash until I find the right way to fix my problem.
I am using the diff command with the -r option, to compare a large number of files and files in subdirectories. My main interest is to find out which files have been changed, and not what the actual changes are, and since a lot of files has been changed, it would be a lot easier to view the file names only. Is there and option for diff that might do this, or does there exist a similar tool/command that could do the job?
I've a directory containing around 2.8 lacs of files. I want to move them to another directory.If I use cp or mv then I get an error 'argument list too long'. If I write a script like
for file in ls *; do cp {source} to {destination} done
then because of ls command , its performance degrades.How can I do this?
I am facing problem in copying a large number of file 18 lakh (18,000,000) files from my personal hardisk to another hardisk each file is very small and size of folder is around 3.95 GB copying files using copy given by Windows is frustrating and I am not even able to compress file its giving me error that its not readable.And problem is I am not able to open this drive in Linux it showing me error there saying do diskchk in Windows and Windows disk check is also not able to repair this drive and goes into some mode unsolvable.Is there any way to open disk with error to open in Windows and if not any way I can copy data faster?ERROR: Disk labled EDU is corrupt go to windows and chkdsk /f there and reboot into window 2 times.
I understand that chroot is usually used to provide security, however, for my issue, security is a big don't care. I am very new to using chroot and don't fully understand how the chroot'd env works.
problem: Trying to use a vendor supplied cross compile environment. The environment runs as a chroot'd env and works just fine. I have a large number of additional modules that I wish to compile in the chroot'd environment. FYI, these modules are also (succesfully) compiled for other targets not using chroot'd env's. Copying the source files into the the chroot environment is not an option (don't have hours to wait for copies to finish and it would break the make system). Having them live in the environment is also not an option (the chroot build is a tiny part of the build process and we cannot revamp our entire source tree to accommodate it).
I am looking for a way to have the compiler in the chroot'd env have access to a path that is outside of the env and typically higher up in the same path that holds the chroot'd env. I have tried soft links (they don't work as expected). Hard links only work for single files and there are 10's of thousands of files that would need to be linked. I am not sure how I would go about exporting the additional files and then mounting the exported files in the chroot'd env (or if that would even work).
This problem is not exclusive to Ubuntu, I've experienced it in Windows and OSX as well, but it seems that almost every time I transfer a large number of files (i.e. my music collection) between my desktop computer and laptop via my external hard drive, I end up losing files for no reason. I usually don't notice the files are missing until later on, because I am never informed of any data loss. Now, every time I make a large transfer of files, I just do it two or three times to ensure that I don't lose any files.
We recovered a large number of files from a HD I messed up. I am attempting to move large numbers of files of a type e.g. .txt .jpg , into a folder by type to more easily sort through them.
Here are the commands I have mainly been trying with various edits:
Code:
Code:
So far the most common complaint I have gotten "missing arguments to execdir".
I'm trying to extract the sender id from a fairly large number of files and am having trouble assigning variables from a file. Here is what I have so far, (which is fairly kludgy I know, but it's been some years since I've done any scripting or programming, and I find that I have lost the knack to a large degree).
I would like to know how do I print the line # in a script. My requirement is, I have a script which is about ~5000 lines long. If there are any errors happen I just exit. And I would like to add the line # of the script where the error happened.
I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.
I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.
I would like to know how do I print the line # in a script. My requirement is, I have a script which is about ~5000 lines long. If there are any errors happen I just exit. And I would like to add the line # of the script where the error happened.
I have the following code in bash script: Code: #!/bin/bash COUNT=1 # bash until loop until [$COUNT -gt 2]; do pq A$COUNT [Pemptus].pq & let COUNT=COUNT+1 done
I did this because I'm that much of a Progress Quest geek that I wanted to have a huge group on the online server, so I decided to make a script that would open all the files for me rather than having me do it manually. I created some characters with the boring name of A1, A2, etc. When I ran the above script, it went into a continuous loop and I had to halt it, then run sudo killall pq.exe to eliminate the 500 or so Progress Quest windows that popped open. Anyway, what is wrong with my script that I can't seem to get it to stop loading files at an arbitrary number? I want to get this part finished before I make any more boring named characters.
I would like to parse an input file in which there are two columns per each row. We want to see how many lines are duplicated where we define duplicate to be having the same second field and different first field. For instance if the input file looks like the following:
I work on Linux for ARM processor for cable modem. There is a tool that I have written (as the job demands) that sends/storms customized UDP packets using raw sockets. I form the packet from scratch so that we have the flexibility to play with different options. This tool is mainly for stress testing routers.
The details are here.
I actually have multiple interfaces created. Each interface will obtain IP addresses using DHCP. This is done in order to make the modem behave as virtual customer premises equipment (vcpe).
When the system comes up, I start those processes that are asked to. Every process that I start will continuously send packets. So process 0 will send packets using interface 0 and so on. Each of these processes that send packets would allow configuration (change in UDP parameters and other options at run time). Thats the reason I decide to have separate processes.
I start these processes using fork and excec from the provisioning processes of the modem.
The problem now is that each process takes up a lot of memory. Starting just 3 such processes, causes the system to crash and reboot.
I have tried the following:- 1-I have always assumed that pushing more code to the Shared Libraries will help. So when I tried moving many functions into shared library and keeping minimum code in the processes, it made no difference to my surprise.
2-I also removed all arrays and made them use the heap. However it made no difference. This maybe because the processes runs continuously and it makes no difference if it is stack or heap?
3-I suspect the process from I where I call the fork is huge and that is the reason for the processes that I make result being huge. I am not sure how else I could go about. say process A is huge -> I start process B by forking and excec. B inherits A's memory area. So now I do this -> A starts C which inturn starts B will also not help as C still inherits A?. I used vfork as an alternative which did not help either. I do wonder why.
reduce the memory used by each independent child processes.
Though I have developed some web applications (automation tools) using PHP which have been primarily meant for Intranet (within the organizations I have worked for), I have paid little attention to SEO (it was not required until now). But now I want to understand how do we go about implementing SEO and where do we fit this mechanism in a system / website?
Do we need any other tools or learn some specific programming /scripting language to achieve it?Will implementing a good SEO yield identical results irrespective of what search engine (be it Google or Yahoo!) is being used by users?
I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-)
However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit. I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.
I have a 50000 line(ish) set of records in a file. I have another file where I have filtered out all the line numbers for those which have an error of various types. e.g column count, field type etc. I want to get all those lines into a separate file so I can sanitise them. There are abt 3-4000 of them.
How can I access those lines which I want to isolate into a single file? I have all the usual linux stuff available and a bit of understanding of regexps.
i've got a select based application that wants to support a large number of mostly idle connections. the code is java and works on windows, suse enterprise linux, mac os x. it does not work on centos 5.5 (32-bit, 2.6.18 kernel, 1G of memory).
i've read and followed the directions in various articles about tuning linux for large numbers of connections (including the C10K problem), and gotten the number of sockets up to 3200.
these didn't make any apparent difference:
[URL]
on windows, i can get up to around 78,000.
on suse enterprise linux (a few years ago), i got up to 90,000. that's where i got bored and stopped.
on my mac laptop with os x (snow leopard), i got up to 10,500.
i have used ulimit -n 10240
my current goal is 10k sockets.
the test is that i'm opening one socket at a time until it fails. when it fails, many of the sockets which have already been opened also fail, in one giant cascade. sounds like a buffer / memory problem.
each group of 64 sockets gets a thread to manage select calls for them. thus i'm only using around 61 threads total when it fails.
I need to rename the resulted searched files from a loopI have the following code:
find . -name DOC* | while read i do find $i -type f -name '*.txt' done
basically, I am searching for all txt files inside any folder starting with DOC name.this code is working fine with me.I need to rename those .txt files to .txtOLDOS: Ubuntu 10.4Bash shell
I used 9.04 for months and it work fine before restarting my PC. After I restarted my PC, the memory consumption takes up to 4.2 GB after login. However, I cannot find any process that consume such large number of memory.
I'm running 64-bit 10.04, upgraded from 9.10. The problem I am experiencing is that any user accounts aside from my main account are problematic. This includes any accounts I add, as well as the GDM guest session.The specific problems that I have thus far experienced are as follows:
1. The desktop loads often improperly. In the latest instance of this the graphics on the right side top panel were randomly chopped-up, leaving parts of my clock on either side of the volume control, among other things. 2. If I make ANY customizations to the desktop at all, the desktop takes nearly a full minute to load on log-in. 3. Flash videos don't work properly on Firefox. Sometimes they only play after refreshing a page, often they will not load at all. Also, attempting to load or play a flash video will sometimes causes Gnome or Firefox to crash. 4. (And this is the one that REALLY has me stumped) Whenever I log into my main accountant after logging out of another account, the IBus control appears in my system tray.However, when I open the IBus preferences the associated check box is (and has always remained) unchecked.Not sure where to go with this one. More than anything, the IBus bug makes me unsure of where to even begin looking for the problem.
I am attempting to upgrade a system from 4.7 to 5.2 using a (now) DVD drive attached to the onboard IDE. Originally I had tried using a remote NFS image and a USB stick but I thought maybe there was a problem with the image. I can get up to the point of the installation of selecting the keyboard for the system and then it freezes and never goes any further. It doesn't appear to be a kernel panic since I can still switch between consoles.
I've got an MSI K9NGM2-FID with 14 drives in it. It serves as a file server for our backup server. It's got a secondary 4 port Silicon Image SII 3114 SATA card using the sata_sil module, and an old IDE Promise FastTrak TX2000. Technically I could have 16 drives but the 750W PS is walking the fine line on tripping it's self-breaker with the 14 drives and 7 fans. I would like to NOT have to disconnect all of this to do the upgrade.
I thought maybe that running the install using the "noprobe" option would help so it didn't detect and load the modules for the Silicon Image or the Promise cards and detect all of the drives but it still gets stuck on the step after selecting the keyboard. The installation info console and the dmesg console don't really provide any useful information. The installation console says:
INFO : moving (1) to step welcome INFO : moving (1) to step language INFO : moving (1) to step keyboard INFO : moving (1) to step findrootparts
And the last lines of the dmesg console says:
<6>device-mapper: multipath: version 1.0.5 loaded <6>device-mapper: multipath round-robin: version 1.0.0 loaded <6>device-mapper: multipath emc: version 0.0.3 loaded
Is there a hidden "debug" option that will turn on a lot of extra logging?
I have on my windows machine several hundred files that are a format of .nc .ncs for a CNC machine. I need to convert them to txt which is something as easy as opening in notepad and then saving as .txt but there are so many that this kind of action would take way too long.
The reason I am writing the linuxquestions is because I would feel more comfortable in loading a live CD and using some sort of terminal command to do this than I would to download one of the many "freeware" type programs I have found for windows (even more so since I have had a root kit before and had to start all the way over to get rid of it).
I need to know:
1. Is this possible to do with the terminal without super advanced knowledge.
2. Can one please point me in the right direction; something to read or an example
I am an uploader to a various hosts, so this tiny script me a lot. I make a rar archive and split files with 100mb. I could get 3-4 or even 76 parts of rar files and it would take me some time to paste all these urls to remote upload function of filehosting sites. For example:
Code:
server:/home/cober/downloads/teevee# ls -al total 358784 drwxrwxrwx 2 root root 4096 Dec 8 19:38 .
