General :: Memory Cache Losing Sync With Disk?

Nov 3, 2010

(Ubuntu Linux server, 64-bits)I was troubleshooting a problem with a file (~3.0 GB) which I had just downloaded, but it was failing the integrity test, when I discovered something really unusual.First this is the MD5 of the file after download, which didn't match the expected value:

~% md5sum media.iso
5d74facb904cc1765a468354908a8f34 media.iso

Some time passes, nothing should have changed the file during this time, but when I went to check the file again:

~% md5sum media.iso
a5b97c5016afb39bd67ccfc3fa6ca59e media.iso

This was really unexpected. Since I have a lot of RAM, I suspected this was the effect of caching and something was going awry with it. I decided to retry with the whole file from disk, for my surprise:

~% sudo sysctl -w vm.drop_caches=3 # This linux command invalidates
vm.drop_caches = 3 # everything in the memory cache.
~% md5sum media.iso
2992aa6270f6e1de9154730ed3beedc1 media.iso

I redid it and now it seems to stay consistent, although this still isn't the value I was expecting. Certainly, the contents in memory cache were different from the contents on disk.This is the big problem.To fix the download, I created a torrent on the source machine and opened it in the target machine. Five 1MB chunks out of ~3.0GB failed integrity check. I used the torrent to fix these file chunks and how the file integrity is ok.The problem now is to determine where the data got out-of-sync.

I tested the memory with memtest86+, all but the bit fading test. I was expecting to see some failing memory module, but there wasn't anything. Everything is ok.Filesystem is Ext4, over LVM2, over a 3-disk RAID5 array.Ext4 is considered stable, and if data were inconsistent between disks, mdadm would have warned. But there is nothing in the logs. S.M.A.R.T. error logs are clean, the disks are new (have less than 30 days of "power-on-hours").I'm looking for information about any data-loss bugs in my current kernel (2.6.35), but there doesn't seem to be anything, as far as I looked.what else I could check, or where exactly could be the defect/bug?It is a Ubuntu 10.10 64-bit, Core i7 930, 6 GB non-ECC RAM.

Update: I confirmed that the files are being correctly written to the disk, the pages are being altered after they are read from disk, while in memory. I did a lot more memtests (I left it doing bit fade test overnight),and still nothing. All memory modules seem ok.Some more tests:

~% md5sum media.iso
cc8bcf1ce67ff7704eadc2222650c087 media.iso
~% cp media.iso tmp[code]....(direcat is a version of cat that reads with O_DIRECT, that is, bypassing page cache)There is a clear pattern: it always happens to the 2nd byte in a 16-byte alignment. In that byte, almost always the bit 4 (LSB) flips to one, but there was one instance where bit 2 flipped to zero.

View 1 Replies


ADVERTISEMENT

Server :: Clear Cache Memory ( RHEL 5.1 ) As It Consumes Almost 100% Physical Memory?

Jan 11, 2010

let me know how to clear cache memory ( RHEL 5.1 ) as it consumes almost 100% physical memory.

View 3 Replies View Related

Programming :: SHM Memory Counted As Cache Memory With Kernel 2.6.18?

Aug 25, 2010

Is that possible that SHM shared memory is counted as cache memory on Linux with kernel 2.6.18?If find it really odd since this memory is not file backed, but I have a piece of code that loads data using shm_open+mmap, and it generates an amount of cache memory in /proc/meminfo that corresponds exactly to the amount of shared memory (I load that data from a file but I am using posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED) to ensure this file is not cached and I made sure that it is working as expected). As far as I know SHM memory was not tagged as cache memory with kernel 2.6.9.If it is the case it is really unfortunate since normally cache memory can be considered to be part of the "available" memory since it can be flushed promptly but this is clearly not the case with SHM memory... Is there an easy way to get the total amount of used SHM memory on a system?

View 4 Replies View Related

General :: Crontab To Email When Buffers/cache Memory Reached A Defined Value

Jun 18, 2010

I have a Linux server running CentOS with 2GB RAM. I am searching in google and also in this forum but could not find any luck yet. I am searching the way that we can receive notification by system to a specified email when the memory reached a specified value we defined.

Is there any tool or sort of shell script we set via crontab every minute to check that?

View 2 Replies View Related

Ubuntu Multimedia :: Losing Audio Sync When Editing Video In Openshot

Aug 31, 2010

my goal is to record video using a canon powershot camera, edit the avi file on my ubuntu 10.04 computer, then upload the rendered file to videos.

problem is that when i cut the video, the audio is no longer in sync with the video, it's off by about 1-2 seconds. this happens with both openshot and pitivi, so i suspect that it's caused by a bug with the codec. (files are avi with mjpeg codec). after searching launchpad, this is apparently a "known issue". that's great but for now i need a workaround.

i do have an old g4 powerbook with imovie hd v6 on it that i can use, but i'd prefer not to because:

1. the powerpc mac is much slower than my new dual core laptop
2. imovie compresses my videos too much so the rendered file is lower quality
3. i simply prefer openshot to imovie

i was thinking of preprocessing my avi files by converting them to another format with a non-buggy codec on linux. i downloaded ffmpeg, but not sure how to use it and what format to use. would mpeg2 be a safe one to use?

View 5 Replies View Related

General :: Windows - Using One Disk As Cache For Others

Jan 6, 2011

Given a PC with several hard drives:

Is it possible to use one fast disk as a giant file cache?

I.e. automatically copying frequently accessed data to that one disk, and transparently redirecting reads and writes to that disk, so that other drives would only have be accessed occassionally.

(writes would have to be forwarded to the other disks after a while of course)

Advantages:

The other drives could be powered down most of the time; reducing power, heat, noise speed of the other drives would not matter much. cache disk could be solid state.

How can I set such a system up?

What OS supports these options? Is this possible at all using Windows or Linux?

Example:

There are 3 disks with 1 Tb each. most of the files are only accessed very rarely, but about 5% of each disk is used frequently.

Which files are used frequently may change over time.

A solid state disk with 150GB should cache the currently frequently accessed files, so that access time is faster and the drives can be put into power saving mode.

View 3 Replies View Related

General :: USB NTFS Disk Suddenly Won't Mount - Partition Gone / Repair It And MBR Without Losing Data?

Sep 12, 2009

Just ran into an uncomfortable problem. I usually never save any documents on my machine, and keep all my stuff on an external USB hard disk. (an 80GB TrekStor DS microdisk q.u)
Well yesterday this disk just would not mount.
Read through related posts but nothing seemed to work. Even tried it on a Windows machine.

Tried TestDisk utility. Found nothing wrong with the drive, but still could not repair the MBR.log code...

Palimpsest Utility recognized the drive, but just will not let me do anything with it except format it.

How can i repair the partitions and MBR without losing all my data?

View 2 Replies View Related

General :: Performance - Best Available Technology For Layered Disk Cache

Oct 17, 2010

I've just bought a 6-core Phenom with 16G of RAM. I use it primarily for compiling and video encoding (and occassional web/db). I'm finding all activities get disk-bound and I just can't keep all 6 cores fed. I'm buying an SSD raid to sit between the HDD and tmpfs. I want to setup a "layered" filesystem where reads are cached on tmpfs but writes safely go through to the SSD. I want files (or blocks) that haven't been read lately on the SSD to then be written back to a HDD using a compressed FS or block layer.

So basically reads:
- Check tmpfs
- Check SSD
- Check HD

And writes: - Straight to SSD (for safety), then tmpfs (for speed) And periodically, or when space gets low: - Move least frequently accessed files down one layer. I've seen a few projects of interest. CacheFS, cachefsd, bcache seem pretty close but I'm having trouble determining which are practical. bcache seems a little risky (early adoption), cachefs seems tied to specific network filesystems. There are "union" projects unionfs and aufs that let you mount filesystems over each other (USB device over a DVD usually) but both are distributed as a patch and I get the impression this sort of "transparent" mounting was going to become a kernel feature rather than a FS.

I know the kernel has a built-in disk cache but it doesn't seem to work well with compiling. I see a 20x speed improvement when I move my source files to tmpfs. I think it's because the standard buffers are dedicated to a specific process and compiling creates and destroys thousands of processes during a build (just guessing there). It looks like I really want those files precached.....

View 1 Replies View Related

Ubuntu :: How To Clean The Cache Memory

Jan 24, 2010

I read somewhere that i can clean cache memory with this comand:

Code:

david@lap:~$ sudo echo 3 > /proc/sys/vm/drop_caches
bash: /proc/sys/vm/drop_caches: Permission denied

The problem is that permission is denied as you see...

View 4 Replies View Related

Debian :: High Memory Usage In Cache?

Mar 12, 2011

I've installed my debian sid about one month ago (first xfce, next gnome) but noticed that it's kind of really slow. The upgrades take ages, launching (and using) firefox takes so much time,... In comparaison to my ubuntu, archlinux (on the same computer) or previous installation of debian there is clearly a problem somewhere.Today I tried to do a "top" sorted by mem usage : 3.5% xulrunner-stub, 2.1% dropbox, 1.4% aptitude (doing upgrade), 1.4% clementine,... nothing terriblebut still I've 2.7Gb or RAM used (more than 50%)

$ free -m
total used free shared  buffers cached
Mem:  3967 26851282  0 79 1938

[code]....

View 4 Replies View Related

Ubuntu :: How To Clear The System Cache Memory

Feb 15, 2011

Ubuntu 10.10

How do I clear the System cache memory.

For the past few days , I am facing problem in opening websites (firefox & Chrome).

I have to restart the laptop and then access the web sites.

View 3 Replies View Related

Software :: Free - Buff - Cache Memory On Debian

Jun 18, 2010

What it's the difference between buff(ered) and cache portions of memory?

Code:

View 3 Replies View Related

Ubuntu Servers :: Cache Directory And File Structure In Memory?

Aug 11, 2010

i am running Ubuntu Lucid x64 as a fileserver that shares its files via SFTP, NFS and Samba. Currently the hard disks are configured to go to standby if they are not needed. This works perfectly as long as no one browses the shares or my HTPC is running: That one repeatedly looks through the shares for new music or movies. In other words my problem is that the disks are spinning up a lot more often than they should have to. Additionally the spin-up time delays the response time while browsing. Since the machine has a lot of unused RAM i want to tell the kernel that it should keep the directory structure in memory. That way the disks would not need to spin up every time someone browses through the directories.

View 2 Replies View Related

Ubuntu :: Natty Clears Memory Cache When Screensaver Launches

May 27, 2011

It seems that something is causing the memory cache to be cleared when the screensaver launches. Then when moving the mouse or hitting a key to stop the screensaver it takes forever for the already launched and minimized applications to redisplay. I am using the ubuntu classic desktop.

Now time to rant.

Natty has so many bugs it is impossible to work around them all. This memory issue is one of the worst. The dead zone caused by the static application switcher, and the application switcher plugins for compiz (and probably any override redirect window) is another big one. There is not a single window decorator for compiz that works right. I have encountered two packages with binaries that won't even run (emerald and xpdf). The nvidia drivers seem to have issues (this may be related to this memory clearing issue).

Without a doubt more bugs than any other release of Ubuntu to date. I have worked around most of them by recompiling, and etc., but this is a real pain.

This is all aside from the idiotic switch to unity, when it is clearly not ready for release. The general direction of Ubuntu is really starting to look bad. By the way, what is up with the decision to uglify the desktop with gray scale icons (I know this is an old one, but I haven't really ranted on Ubuntu yet).

Looks like I will have to revert to Maverick.

View 4 Replies View Related

Ubuntu :: Cannot Sync Nokia 5800 In Memory Card

Apr 3, 2010

sth happened and I cannot sync my nokia 5800 in memory card mode with my laptop. It says sth that certain files or folders are read only. Tried to change permissions but no luck. this is it:

koli@koli-laptop:~$ n
sending incremental file list
sent 7412 bytes received 21 bytes 14866.00 bytes/sec
total size is 11073190 speedup is 1489.73

[Code]....

View 9 Replies View Related

Fedora :: What Programs Can Be Used To Test And Find Faulty Memory (RAM, Video Adapters, Cache)

Oct 23, 2009

What programs can be used to test and find faulty memory (RAM, Video adapters, cache)?

View 2 Replies View Related

Software :: Monitoring Tools Which Can Listen Disks / Overhead / Cache / Memory / CPU Performance

Jul 13, 2011

I'm searching monitoring tools which can listen disks, overhead, cache, memory, cpu performance.Can anybody help me?

View 4 Replies View Related

General :: Good Software To Monitor Disk Io , Memory And Cpu Use?

Jul 19, 2011

I'm developing a website. I was wondering if there are any tools out there that make it easy to log and graph usage statistics of how my cpu, memory, disk and network are used.

I need to somehow know when I need to scale up / when I'm going to get problems soon.

I found http://sebastien.godard.pagesperso-orange.fr/
(command line thingy, not so useful because I'd have to write my own graphics and logging layer on top of it..)

View 3 Replies View Related

General :: Difference Between Hardware Cache And Slab Cache?

Nov 22, 2010

I am reading slab allocator, it defines slab cache, i am quite confuse is it same as hardware cache?

View 2 Replies View Related

General :: HD Spin Up Delay - Why Not More Pictures Held In Memory Until Disk Is Ready

Apr 13, 2010

I have a security cam watched by a computer with Motion. on that computer i use hdparm -S 240 /dev/hdb to stop the harddrive where the pictures go from running when not used (witch is usually 2 days in a row). Now when motion is detected, the first 3 or 4 pictures are saved, and then there is a pauze for about 5 sec. before the next picture. This has to do with the hard-disk spinning up (I tried without), but I don't understand why not more pictures are held in memory until the disk is ready. how to solve this without having the disk running all days?, or what i should start reading

motion (version 3.2.9)
Debian lenny
Kernel 2.6.26-2-686
hdparm v8.9
PII with 384mb ram

[code]...

View 6 Replies View Related

OpenSUSE Install :: Ubuntu In Dualboot Without Losing Any Data On Hard Disk?

Jul 2, 2010

How to install openSUSE with Ubuntu in dualboot? I have Ubuntu 10.04 and i want to install openSUSE 11.2 without losing any data on hard disk.

View 4 Replies View Related

General :: Disk Based Text Editor To Edit File Larger Than Memory?

Aug 2, 2010

I am using Ubuntu and looking for a good editor to edit a file that is > 4GB. I just need to put content at the end and beginning of the file. I suppose I could use something like

cat "text to add" >> huge_file

To append to the file. Is that the route to go? What about prepending? In general, what is the best route if I wanted to edit somewhere in the middle?

I've tried VIM and it fails miserably. I assume emacs and nano would be even worse. What else is there? I assume to accomplish what I am looking for, the editor would have to be specifically designed for this by not keeping the entirety of the file's contents in memory.

View 5 Replies View Related

General :: Set The Limits For Unlimited Virtual Memory With Limited Ram And Swap Space On Disk??

Apr 6, 2010

clarify me with ulimit output and memory limit?

ulimit -a output:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 1572864

[code]...

View 2 Replies View Related

General :: Sync Thunderbird Contacts Between Computers, Like Firefox Sync?

Sep 19, 2011

I've discovered Firefox Sync a while ago, and it's absolutely awesome. Now of course I'd like most of my software to work this way! So is there a way to get the same behavior with Thunderbird?

View 1 Replies View Related

Hardware :: Reduce The Amount Of Disk Cache?

Dec 26, 2010

Using KInfoCenter | Memory module it shows my 2GB of ram. Approx. 14% is used for Application Data and Disk Cache has been anywhere from 29-35%...leaving approx 1GB free. Can this 'Disk Cache" be reduced leaving more memory free or is this determined by the OS?

View 3 Replies View Related

Server :: Like To Understand Linux 'free' Command With Respect To Cache And Free Memory

Mar 27, 2010

Was wondering if anyone can explain briefly the relationship of "cache" and free memory in the "free" memory command.

View 2 Replies View Related

Ubuntu :: Solid State Disk As Cache For Regular Hard Drives?

Nov 16, 2010

60 gigabyte SSDs are down around US$100.

It seems like the optimal use would be as a cache for the regular hard drives in my computer. Eliminating the need for a fast hard drive, so I can just use a slow 2TB (~US$100) drive with a SSD cache.

Is there a good way to do this yet?

It seems like it would be nice to be able to exclude some files from caching, for things like bittorrent.

Reviews of some SSDs: [URL]

View 9 Replies View Related

Server :: Squid Servers Cache Size On Disk Automatically Increasing And Decreasing?

Jun 24, 2010

Squid servers cache size ondisk automatically increasing and decreasing,how I would resolve this issue

View 6 Replies View Related

Software :: Postfix - Cache - Logging - Taking Up All Disk Space - Limit / Turn Off?

Feb 22, 2010

I'm working on a few servers running centos and using postfix. I don't know what the exact problem is, but we are having problems with the disk space being maxed out at 100 gigs. What we think the problem is...is that postfix is either caching or logging all the emails we send out. We sent 250k emails (500kb apiece) over the weekend and we were having trouble with that quantity. It seems some of those email were queued up for retry sending...but we didn't have sufficient disk space for that? Something broke - I'm not sure what.

What I want to do is to find and change the config file that has to do with postfix email retrying - possibly limit this (not sure if this will fix my problem). Or, turn off /limit any way that postfix logs/caches emails so that it won't take up all the disk space when queued up for retry... Again, I'm totally lost here (on both what's going on, and how to fix it). I'm not sure what more information is needed to address this problem

View 1 Replies View Related

General :: Status Of Memory Usage And Disk Usage Using Sigar In Windows And Ubuntu

Mar 15, 2010

I was trying to get the status of memory usage and disk usage using sigar in windows and ubuntu. done this in windows by just copying the sigar library into jdk library. But i was unable to do so in ubuntu. I've copied the library to java-6-sun library but still can't run the program.

View 14 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved