I have a directory with some data files in it. I did an md5sum find, and built an index of all the files contained:
Code:
find ./* -type f -print0 | xargs --null md5sum > MD5SUM
Now, based on my new index, I want to find the copies of these files as they appear in a new directory, where they have been renamed and reorganized.
I would like to compare two md5sum outputs to see if the files match. in my script I have
Code:
ORG_FILE="/path/to/org/file.zip"
NEW_FILE="path/to/new/file.zip"
MD5_ORIG=$(md5sum -b "$ORG_FILE")
[code]....
How do I get just the MD5 hash and not the */.... stuff so I can compare them. i tried Code: JUST_HASH=${$MD5_ORIG:0:32} but All I get is
dir_mon_notify.sh: line 79: ${$MD5_ORIG:0:32}: bad substitution
I know find can do what I am looking for, but I am wondering if there is an alternative way to find files on the filesystem either created before/after a certain point, or at a certain time.
Typically I rely on updatedb & locate for most of my file searching needs. Issues with those tools, though, are that it only has directory and file names, and it only creates a database of local directories, not anything mounted via CIFS|NFS or via -o loop (eg, .iso images).
So if I need to find files created after yesterday across the entire system (local and remote filesystems), I am currently needing to use find.
What other tools, if any, would accomplish this in a similar fashion?
I have tried ls and grep, but that requires (in my attempts so far) multiple searches:
ls -lR | grep Aug | grep 10
ls -lR | grep Aug | grep 11
What GUI Linux programs are there for finding files based upon their contents?
View 5 Replies View RelatedI need a script that will take all the files in a given directory and create new monthly sub-directories and sort all the files based on the creation date into the appropriate directory.For example, all files created between 01/01/09 and 01/31/09 will be placed in 'JAN-2009'
View 5 Replies View RelatedI have a folder named Pictures that contains a bunch of .jpg files. My problem is that they all have randomly numbered names, then there is a duplicate of the file that is random numbers then the letter a right before the .jpg.for example, there would be 123.jpg and 123a.jpg, where 123a.jpg is just a resized version of 123. What i'd like to do but have NO clue how to, is to have a script or something go through my Pictures folder, then copy the ones that end in a.jpg to a folder called Resized, and ones that dont have that to a folder called Originals. That way my Pictures folder will be in tact, and i'll have copies of them all separated out.I have to do this all through the CLI on a machine, maybe I dont even need a script and can just do it with a slick command?
View 14 Replies View RelatedI was hoping to get some pointers on how to rename files based on database entry. I got hundreds of thousands of files that has GUID name assigned to it. only way to find out the file name is to look up the database table. Its obvious that this is not efficient. I couldn't find any tutorials on how to do this. Please point me to right direction. A starting point would be very helpful.
View 1 Replies View RelatedI once had a script that when run would find the first 800GB of files in a directory (including subdirectories) and write them to a file (ie: ./800gb.sh > manifest.txt).I used this to create manifests of 800GB worth of data from large directories in order to dump to tape (LTO4).I'm sure its gotta be a pretty simple script, but I am not very skilled at writing bash scripts.
View 4 Replies View RelatedI have a single directory of pairs of files, with the pairs sharing a string as the beginning of the filename:
SF1-27F1492R-clone01_T3_A18_001.ab1
SF1-27F1492R-clone01_T7_A20_002.ab1
SF1-27F1492R-clone02_T3_A19_003.ab1
SF1-27F1492R-clone02_T7_A21_004.ab1
...etc
I need to create a subdirectory for each pair then move the pair into the subdirectory.
I accomplished the first step using:
$find /foo -name '*T3*' -exec mkdir '{}.wrk' ;
I can use a regex to designate the pair and associate the directory, but how do I use regex in a path as the output of a move command?
Originally Posted by Kenny_StrawnPlease wrap [CODE] tags aroung any code posted here. The full source that way could still be posted.I am trying to copy all the files in the directory based on the modification date (i.e created on Dec 29). Not able to find the proper command for this. This is what I have tried.
(none) login: root
#
# cd /mnt/hd/
[code]...
I'm writing a bash shell script that among various other things will traverse through a directory with hundreds of files and rename those who match a pattern found in a config file. It's expected that only about one in ten files will actually match, and those who don't, will simply just be ignored for this purpose.
This should for instance cause the file "dBase program file December 1987.prg" to be renamed "Clipper source code December 1987.prg", and conversely "C++ source August 1996.cpp" to be renamed "C source code August 1996.cpp" etc.A sample file such as "Random Data File.dat" should not be renamed here since it's not mentioned in the config file..What is the quickest, most elegant way to do this in bash?I am thinking of using bash's built-in regex matching combined with the /bin/rename utility, but don't quite know how to get started to catch this..I guess there are plenty ways of doing this in perl and elsewhere as well, but since this has to integrate into a pre-existing bash script, that's what I'm looking for.Anyone out there with a spare moment to offer a hint in the right direction?
I am trying to get a checksum for a file in a subscripted variable in a bash script. md5sum outputs a checksum and the name of the input file. For example:
Code:
eval CSUM$K=$"(md5sum file)"
This might return something like this:
Code:
3cff5d5c0113959d0be62be34b97e05c file
I want to assign just the checksum to the variable in my shell script and omit the file name that follows. Is there something besides md5sum that will generate a checksum? Or if not, then I was thinking I might be able to extract the checksum without the file name using sed.
I have very little linux experience. And need some help with a bash script. I need to a script I can set cron to run to sort files out of a holding folder into final folders. It doesn't necessarily have to be bash, but I think it would be sufficient for this. File names are formatted as such when created: Dest-Date-Time-CID-Destination# I want the files to be moved from a all in one holding folder to a folder structure like this.
.../storage/year/month/day/Destination#/VarX(type)/hour/CID/'File'
I would need an if/else if/else statement to say if Dest = A set VarX = B If for example the file name was
infinity-20100927-17:00-1112223333-4445556666.wav
I would like the above file to end up moved from
.../holding
to
.../storage/2010/09/27/4445556666/Inbound/17/1112223333/infinity-20100927-17:00-1112223333-4445556666.wav
So the script will need to make directories based on information in the file name which is delimited by single dashes. Then move files from the holding folder to the newly created "sorted" folders.
I have a directory listing with many subdirectories having many files. I want to recursively search for the oldest 5 files starting from the base directory and not 5 from each subdirectory. I am writing a shell script which sorts them using ls -lRtur|egrep "txt|jpg" > /tmp/file1 Now from this /tmp/file1 file I want to sort the files same as what the ls -ltr command does that is oldest file time to newest file time first. How do I sort based on Linux time stamp? The files itself also have Linux timestamps embedded in them So I can sort based after extracting them as well if it is easier.
My /tmp/file1 has entries like below.
-rw-rw-r--. 1 usr1 usr1 705 2010-01-22 17:25 sample20100603173659.jpg
I want to get the 5 oldest files and then delete them.
What i am trying is to check the file duplication in a folder and remove a file if it is a duplicate of another file ie the contents are duplicate; but names may be same.
Basically i am using md5sum to calculate the md5sum values of each file and redirecting to a file. And i am thinking of comparing the md5sum values.But i am finding it hard to decide how to complete the code after redirecting the output of calculation of md5sum to a file.
I'm working on an application used for backup/archiving. That can be archiving contents on block devices, tapes, as well as regular files. The application stores data in hard packed low redundancy heaps with multiple indexes pointing out uniquely stored, (shared), fractions in the heap.
And the application supports taking and reverting to snapshot of total storage on several computers running different OS, as well as simply taking on archiving of single files. It uses hamming code diversity to defeat the disk rot, instead of using raid arrays which has proven to become pretty much useless when the arrays climb over some terabytes in size. It is intended to be a distributed CMS (content management system) for a diversity of platforms, with focus on secure storage/archiving. i have a unix shell tool that acts like gzip, cat, dd etc in being able to pipe data between applications.
Example:
dd if=/dev/sda bs=1b | gzip -cq > my.sda.raw.gz
the tool can handle different files in a struct array, like:
Code:
enum FilesOpenStatusValue {
FileIsClosed = 0,
FileIsOpen,
[code]....
Is there a better way of getting the file name of the redirected file, (respecting the fact that there may not always exist such a thing as a file name for a redirection pipe).
Should i work with inodes instead, and then take a completely different approach when porting to non-unix platforms? Why isn't there a system call like get_filename(stdin); ?
If you have any input on this, or some questions, then please don't hesitate to post in this thread. To add some offtopic to the thread - Here is a performance tip: When doing data shuffling on streams one should avoid just using some arbitrary record length, (like 512 bytes). Use stat() to get the recommended block size in stat.st_blksize and use copy buffers of that size to get optimal throughput in your programs.
My ubuntu started out right, it copies files 3/4 of the way through and reads ready when you areThe foward button does not let me click it
View 1 Replies View Relatedrsync -r -v -e ssh root@nn.nn.nn.nn:/usr/local/websites/* /usr/local/websites and each time I run it it copies everything - all files. I thought rsync was only supposed to copy files that had been added or modified.
View 3 Replies View RelatedI'm looking for a fast way to verify a copy of a folder with 150Gigs of data, in 33 files. Some of the files are a few kb, while a few are 20-30Gigs. I've done a file count, which is quick, but doesn't verify that all the files are intact. I tried running md5sum on them, which works, but will probably take as long as copying the files in the first place. Diff works too, but is slow too.
View 1 Replies View RelatedI noticed something a little odd I'm hoping someone can enlighten me on. I noticed in a couple of cases that a package has the proper version, but differs in two regards.
1. The package ends up with a .el4 on the end of the version for Red Hat 4.
2. The actual MD5Sum of the files the package provides differ.
An example below:
Code:
[root@RH4ES32-MCE bin]# for i in `rpm -ql GConf2`;do md5sum $i;done;
md5sum: /etc/gconf/2: Is a directory
9f90335546f7c57ae6fb552cc2b919c5 /etc/gconf/2/path
md5sum: /etc/gconf/gconf.xml.defaults: Is a directory
[code].....
So my package changed slightly to now show .el4 versus just 2-2.8.1-1 I've indicated in the first output above that the first couple of lines differ. I stopped my comparison at that point as they truly are different.
I created a file holding all the md5 values of my files to find duplicates as follows: find /mnt -type f -print0 | xargs -0 md5sum >> ~/home.md5
I then tried to find duplicates and do ls -l on the result in such way: cat ~/home.md5 | awk '{print $1}' | sort | uniq -c | sort -nr | awk '{print $2}' | head -n 10 > ~/top10.md5
Now I attempted to do an ls -l on the files using the command: for i in `cat ~/top10.md5`;do grep $i ~/home.md5 | while read checksum path; do echo "`echo $(printf '%q' "${path}")`" | xargs ls -l; done; done
This works well on most files, however it does not work when filenames have special letters in them that gets escaped such letters with accent etc. These become for examle 303.
Are there any ways I can use the escaped 303 strings with path names, or any better way I can do this?
I have 2 external hdd in wich I have all my files.... yesterday, I have copied all the files from hdd2 to hdd1 and I want to eliminate duplicates so I used FSLint to find them, now, I have a txt file that looks like this:
Code: /media/My Book/!!!MIS DOCUMENTOS/Documentos/2 sep2003-jun2009 USB/!TESIS/TESIS/TESIS CVT LABVIEW Y CODEWARRIOR/LabVIEW85RuntimeEngineFull.exe /media/My Book/HDD_Toshiba/Borrable/Pen_Drive_4GB/Tesis/Super CD de la tesis/LabView/LabVIEW85RuntimeEngineFull.exe multiplied by millions of entries...
now I want to make a shell script to delete all the files/entries (read from the log file) that begin with:
Code:
/media/My Book/HDD_Toshiba/**** Since HDD_Toshiba is the folder in hdd1 (MyBook) that contains all the files from hdd2
I am trying to copy the files from my WHS disk to my Ubuntu Server disk. I have the windows disk mounted at /media/WINDOWS and I want to transfer to /storage so I ran;Code:sudo cp -r /media/WINDOWS /storageIt takes about 4-5 seconds and is complete, but there is about 500 GB worth of data there so I know it didn't really copy everything over. When I look at the files in console it shows them, but when I look at the /storage through SAMBA on my Windows machine, it only shows the directories.?
View 9 Replies View RelatedIn order to upgrade a machine that can not successfully upgrade to 10.4 I downloaded and burned the 10.04.1 iso image off the ubuntu alternate download site. In my first attempt I unsuccessfully burned the image with it failing at the very end. I did perform an md5sum on it and received the precise output I got from my second burn attenpt which DID complete successfully. Here is the output:
[code]...
I did research this last night and it seems the common wisdom was to reburn the iso (which I did twice) or copy down the iso again. This I also did and it came down precisely, bit for bit, the same as the first one. Here are the two cksums
[code]...
Is there something wrong with this image on the website or is the error about 1 file being unreadable (could that also mean missing?) be erroneous?
I've found these commands in [URL]:
Quote:
find -type f -print0 | sudo xargs -0 md5sum | grep -v isolinux/boot.cat | sudo tee md5sum.txt
But I don't understand these commands, even after reading their manuals.
I want to move all files and directories that are 1 month old out to back up into a separate folder. There will be a lot of files and I want to make sure it copies properly. The problem I'm having is integrating a MD5SUM into it to check integrity. MD5SUM is not recursive, so I figured it would work in a loop when it copies each individual file, I'll do a md5sum on each file and delete that md5 once its verified it copied ok.
[Code]...
I also need some sort of error handling to output all md5's that didnt pass the hash check.
I made a Bash script that is fired by a Cron job every morning. It dumps an SVN backup on some Samba shared drive. I would like to know how I can make sure the job worked correctly without having to verify the shared drive every morning. Right now, I take the job's output, save it to a log file and send this file by email. But the ouput isn't so great.
[Code]....
I know that fork() copies the address space of the calling process. Say, however, i have a linked list allocated. Will the list be copied over to the child process's space? If so, i would have to free them in the child process as well as the parent process, correct? Or will the variables be copied but not be pointing to any valid address? Or would it just kind of not do anything?example:
Code:
struct ll_ex {
struct ll_ex * next;
[code]....
I would like that users can log to X but this with a text based xdm + how to active it as last in inittab since it is text what contains my inittab
Code:
1:2345:respawn:/sbin/getty 38400 tty1
2:23:respawn:/sbin/getty 38400 tty2
3:23:respawn:/sbin/getty 38400 tty3
[code]....
I'm not afraid of bash but I'm not very good with it either. I'm assuming there's a way of using find, perhaps in conjunction with another tool, to find images in a directory (and subdirectories) based on their dimensions?
Specifically, I want to find all the landscape-oriented images and copy them somewhere else.