Ubuntu :: Lists Files With Both File Size And MD5 Hash?
Jan 20, 2010
Is there any utility that can provide a list of all files, and both the file size and md5 hash value. Preferably also including other hash values.I've got 1.5 TB of files to go through, and delete duplicates..Neither fdupes or fslint are up to the task --- both claim files to be duplicates, when they definately are not. (Movies, and OOo documents are not identical, even if one is the script for the other, which in this case is not the case. Both fdupes and fslint claimed that those two files were identical. (And yes, I did look at them.
I'm writing an interpreter and it used simple association lists for mapping varaible names to their values. Here's the code:
Code:
#include "assoc_array.hpp" #include <string.h> using namespace LANG_NAMESPACE;
[code]....
I thought that I would replace it with a hash table to increase performance. Note that I decided to store linked lists in the buckets instead of the actual values, in case the hash function outputs the same index for multiple variable names:
It runs in about 0.333 seconds on my machine. Since I'm using git version control, I decided to bring back the old version that used association lists. To my amazement, it ran 3 times faster, completing in 0.116 seconds! Is my hash table implementaion really that bad, or is this a really poor benchmark (and real-world code actually will be faster using the hash table)?
I'm messing with my samba config file. For line comments, some use a number sign/pound sign/hash symbol, while other lines use a semicolon:
Code: ############ Misc ############ # Using the following line enables you to customise your configuration # on a per machine basis. The %m gets replaced with the netbios name # of the machine that is connecting
Today I was trying to clean up my system and am a little bit stumped on something.
I used synaptic to clean up installed kernels, except for the one that was currently running. Once done I went ahead and restarted my computer.
Grub lists only one kernel available. However when I go to a prompt and
issue dpkg --list|grep linux-image I get several items listed code...
2.6.32-22-generic is the kernel I am running so I assume the "ii" files have something to do with the kernel in memory.
However, what are the "rc" files that are listed? Resource files? All three kernels referenced have been removed a LONG time ago and when I look for installed headers, or anything else, there is nothing installed for *-12, *-12 or *-14.
And when I check Synaptic for just the kernel number I get a list of packages with no version install, no latest version notation and no description (attached screenshot).
Just what are those references and how does one remove them?
I have a quirky situation whereby I'm using SED to selectively comment out a line in a crontab job (on Solaris, I know but it's connected to the Linux function I'm working with).
What's happening is this.
Remove hash symbol
Code: /opt/csw/bin/gsed -i '/^.*/usr/local/scripts/mirror-fix.sh.*/ s/^#//' $TEMPFILE Restore hash symbol
The problem I'm running into is that the script can sometimes prepend an extra hash # symbol if run more than once (I have a lockfile that I poll for to discourage this, but that's not perfect).
I wonder how I can modify that sed statement to remove any/all leading hash marks up to the first other character, in this case it's a 0 (zero) which is a crontab file.
I have not ran into this before. I did a new install of 10,04 and was adding programs. Then software center and Synaptic stopped working. error message said it was most likely due to unmet dependencies. I googled fixs which most all ran some sort of "apt-get" command. I can't run those because as soon as it gets to "Reading package lists", I get this error.Read error - read (5: Input/output error)The package lists or status file could not be parsed or opened
This is a error with my ubuntu 10.4 when I try to update it. E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-Limit. Current value: 25165824. (man 5 apt.conf) E: Error occurred while processing bomberclone (NewFileDesc2) E: Problem with MergeList /var/lib/apt/lists/archive.offensive-security.com_dists_pwnsauce_universe_binary-i386_Packages E: The package lists or status file could not be parsed or opened. E: _cache->open() failed, please report. W: Duplicate sources.list entry Index of /ubuntu lucid/partner Packages (/var/lib/apt/lists/archive.canonical.com_ubuntu_dists_lucid_partner_b inary-i386_Packages) W: Unable to munmap
E: Read error - read (0: Success) E: Problem closing the gzip file /var/lib/dpkg/status - close (0: Success) E: The package lists or status file could not be parsed or opened.
I keep getting this messageCould not initialize the package informationAn unresolvable problem occurred while initializing the package information.Please report this bug against the 'update-manager' package and include the following error message:'E:Encountered a section with no Package: header, Eroblem with MergeList var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_natty_main_bina ry-i386_Packages, E:The package lists or status file could not be parsed or opened.'
is lvresize with --resizefs options re-size the Logical Volume and then re-size the file system? i mean we don't need to use resize2fs?I looked at man pages but it doesn't explain this option.
Looks like my package manager was severely screwed. Any package manager-related command I try to run results in the following message in the console:
Code:
E: Read error - read (5: Input/output error) E: The package lists or status file could not be parsed or opened. Anything - aptitude, apt-get.
I removed all my sources.lists, I tried running apt-cache clean It didn't help. This is really serious. Not only am I unable to do anything with packages (update, remove, install), but the machine is really slow because of this.
I have an internal hard drive and an external hard drive, both with about 350 GB of data. The data came from the same source, but over the last couple of years, different people have moved files around to different directories, and some files have been deleted. Now I want to merge all the files onto the internal hard drive. I estimate that 80% of the files on the external hard drive are the same, so I don't want to copy 290+ GB of data over when I already have it.
Therefore, I need a way to find just the files on the external hard drive that don't already exist on the internal one. In other words, I need to create two lists of file names irrespective of directories and compare them, selecting only the file names that exist in one list OR the other. I've Googled for solutions but can't find anything suitable. There are ways to create text files of the file names and compare them with diff, but they have to be in the same order, and since these files are in vastly different directories, that won't work.
Reading package lists... Error! E: Problem parsing dependency Depends E: Error occurred while processing python-glade2 (NewVersion2) E: Problem with MergeList /var/lib/dpkg/status E: The package lists or status file could not be parsed or opened.
Ive been experiencing problems with my squid3 recently, i am using 10.04.3 LTS. Configured squid as always have been configuring since 8.04.04, but its not working as it should be.Ive been having issues with bz2 files, in my LAN when i try to do an apt-get update, it just says some indexes could not be downloaded cause of a sum hash mismatch.If you check squid log, for that kind of file it saysTCP_REFRESH_UNMODIFIED/206 Google around i read thats ssquid cache keeppin the files more than usual, so i add this hoping to solve the problem:refresh_pattern -i .bz2$ 0 0% 60 override-lastmod refresh-ims override-expireI dont know if thats well written or not, but it doesnt have solve the problem, and now squid log shows,TCP_REFRESH_UNMODIFIED/304but the same behaviour of the hash sum mismatch, please if someone could throw a ligth in here. The only fix to this problem so far is deleting all cache and recreate it every morning, which is far from a solution.
I need to get names of all installed packages in 2 machines and save them in 2 text files, then I want to compare these 2 files to know the differences between 2 files and from that I could know the differences between 2 machines. Is it possible to do that and what program I could use?
This happened completely randomly. One day, my synaptic and update manager is working fine and the next day, I open one of them and get a error window that doesn't say anything (no details on the error). So, i tried the terminal, sudo apt-get update, and this message shows:
Reading package lists... Error! E: Problem parsing dependency Depends E: Error occurred while processing lib�tartup-notificamion0-dev (NewVersion1) E: Problem with MergeList /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_karmic_main_binary-i386_Packages E: The package lists or status file could not be parsed or opened.
I am newer to Linux ( using Ubuntu 10.04) : I have noticed that during replacement of a file , no date and size of the new and old files are shown in the dialogue box so how to show that ( like the one in windows)
I know that it is easy question , but i really don't know how to do that , by the way I have checked folder preferences and system --> preferences but i did not find something for that
Looks to me like we're a baby-step behind Windows in not having GUI wrappers or frontends for the steganography (?) binaries I mentioned above. Or have my searches just not turned up the obvious GNOME solution?
Also, can someone tell me exactly what image file format works in login screen face lists, in place of the "generic user" icon one? And is there a command-line workaround to get unique ones assigned per-user? The "Login Window" utility in Ubuntu 9.10 seems a bit insistent on using the same image and path to images for all user icons -- hardly a user-friendly way of making user A's identifying icon different from user B's.
I came across this feature of the "less" command today. It appears not to be well-documented so I'll share it here. I found that "less" can automatically uncompress compressed files - no need to run the first command, the second does the same job:
Code:
gunzip -c textfile.gz | less less textfile.gz
Even better, "less" can read and list the contents of some archive files as well, including Tar files compressed using Gzip and Bzip2 as well as Zip files. So now when I want to list the contents of an archive,for example to determine whether the contents will be extracted to an absolute path or relative to the working directory, the following work
Code:
less archive.tar.gz less archive.tgz less archive.tar.bz2 less archive.zip
Much easier than remembering the various options that the Tar, Gzip, Bzip2 and Zip use. The best one for me is with RPM archives:
Code:
less some.rpm
The above outputs the Information, Change Log and verbose File List of the RPM (paging the output, of course!) The following commands (using short and long options) give the equivalent output using the "rpm" command:
Code:
rpm -qilvp --changelog some.rpm | less rpm --query --info --list --verbose --package --changelog some.rpm | less
"less" does not recognise the initramfs file produced by "dracut" (which is a compressed "cpio" archive), but I suppose you can't have everything ...
I have Ubuntu 11.0 installed.It is giving following error during update:
'E:Encountered a section with no Package: header, Eroblem with MergeList /var/lib/apt/lists/mirror.csclub.uwaterloo.ca_ubuntu_dists_natty_main _binary-i386_Packages, E:The package lists or status file could not be parsed or opened.'
Firstly i'd like to introduce myself before I ask my question, i am currently studying a "Network Operating System 2" course, and in the syllabus the professor decided to use Ubuntu Linux for us. This is a first for me and I am truly greatly interested in the OS. It seems like a solid system once you get to know the deep foundations of it. The professor has already given us an assignment, but I feel its not fair as its just a bit too much from out of what he taught us .. He did not teach us what he is asking for. He asked us to create to a text file which lists the Sales from two Salespersons, and each one will have an amount for each month .. So 12 entries for each salesperson.
Using that text file, we should write a bash script which would read the values from the above text file, and produce a "Monthly Sales" text file, which shows the total sales of each month (combined from both the salespersons). I would like to ask for some guidance, some help on how to start. It's just that he didn't teach us any commands about this matter.
I am curious if perhaps I am doing something wrong extracting pages from a pdf doc using pdftk and creating a new file. I am only extracting the odd pages from the file and outputting them to a new file that is now only 20 pages instead of the input's 40 pages, yet the new output file is still 1.4Mb in size, the same as the original.
It seems strange to extract only half the pages of a large document and end up with a result that is the same size. how to streamline the resulting pdf's using pdftk?
BTW this is the command I am using, in case perhaps I am missing an option to optimize file size or something:
I'm researching about symbolic links been used with samba / CIFS:I'd like that the user that uses a MS-Windows OS could see my shared folder on CentOS 5 and the symbolic links that are inside this folder. Well, it works but, the user will see that the size of the file is bigger than the real file. Apparently, CIFS gets the size of the symbolic link (aproxim.32K) and add it to the size of the file.Example 1: 100KB file, used with shared folder, MS-Windows's user will see 100KBExample 2: 100KB file, used with symbolic link inside a shared folder, MS-Windows's user will see 132KB. (Sym link + size of file)Is there a way to allow the user only see the size of the file, and not the file + symbolic links ?
I was just testing specifying limit on file size to a user and have added the following to /etc/security/limits.conf bob soft fsize 100 This basically should have said not to allow bob to create anyfile greater than 100Kb in size.
But the interesting thing is, if bob already has any file which is greater than 100Kb in size, it even doesn't allow to log him into the system both from console and SSH. Also nothing is logged in logs.. How do I configure it so that, bob can login to the system even though he has any file greater than 100Kb (but doesn't allow him to create file which are greater than 100Kb) ??
