General :: Windows - Using One Disk As Cache For Others
Jan 6, 2011
Given a PC with several hard drives:
Is it possible to use one fast disk as a giant file cache?
I.e. automatically copying frequently accessed data to that one disk, and transparently redirecting reads and writes to that disk, so that other drives would only have be accessed occassionally.
(writes would have to be forwarded to the other disks after a while of course)
Advantages:
The other drives could be powered down most of the time; reducing power, heat, noise speed of the other drives would not matter much. cache disk could be solid state.
How can I set such a system up?
What OS supports these options? Is this possible at all using Windows or Linux?
Example:
There are 3 disks with 1 Tb each. most of the files are only accessed very rarely, but about 5% of each disk is used frequently.
Which files are used frequently may change over time.
A solid state disk with 150GB should cache the currently frequently accessed files, so that access time is faster and the drives can be put into power saving mode.
View 3 Replies
ADVERTISEMENT
Oct 17, 2010
I've just bought a 6-core Phenom with 16G of RAM. I use it primarily for compiling and video encoding (and occassional web/db). I'm finding all activities get disk-bound and I just can't keep all 6 cores fed. I'm buying an SSD raid to sit between the HDD and tmpfs. I want to setup a "layered" filesystem where reads are cached on tmpfs but writes safely go through to the SSD. I want files (or blocks) that haven't been read lately on the SSD to then be written back to a HDD using a compressed FS or block layer.
So basically reads:
- Check tmpfs
- Check SSD
- Check HD
And writes: - Straight to SSD (for safety), then tmpfs (for speed) And periodically, or when space gets low: - Move least frequently accessed files down one layer. I've seen a few projects of interest. CacheFS, cachefsd, bcache seem pretty close but I'm having trouble determining which are practical. bcache seems a little risky (early adoption), cachefs seems tied to specific network filesystems. There are "union" projects unionfs and aufs that let you mount filesystems over each other (USB device over a DVD usually) but both are distributed as a patch and I get the impression this sort of "transparent" mounting was going to become a kernel feature rather than a FS.
I know the kernel has a built-in disk cache but it doesn't seem to work well with compiling. I see a 20x speed improvement when I move my source files to tmpfs. I think it's because the standard buffers are dedicated to a specific process and compiling creates and destroys thousands of processes during a build (just guessing there). It looks like I really want those files precached.....
View 1 Replies
View Related
Nov 3, 2010
(Ubuntu Linux server, 64-bits)I was troubleshooting a problem with a file (~3.0 GB) which I had just downloaded, but it was failing the integrity test, when I discovered something really unusual.First this is the MD5 of the file after download, which didn't match the expected value:
~% md5sum media.iso
5d74facb904cc1765a468354908a8f34 media.iso
Some time passes, nothing should have changed the file during this time, but when I went to check the file again:
~% md5sum media.iso
a5b97c5016afb39bd67ccfc3fa6ca59e media.iso
This was really unexpected. Since I have a lot of RAM, I suspected this was the effect of caching and something was going awry with it. I decided to retry with the whole file from disk, for my surprise:
~% sudo sysctl -w vm.drop_caches=3 # This linux command invalidates
vm.drop_caches = 3 # everything in the memory cache.
~% md5sum media.iso
2992aa6270f6e1de9154730ed3beedc1 media.iso
I redid it and now it seems to stay consistent, although this still isn't the value I was expecting. Certainly, the contents in memory cache were different from the contents on disk.This is the big problem.To fix the download, I created a torrent on the source machine and opened it in the target machine. Five 1MB chunks out of ~3.0GB failed integrity check. I used the torrent to fix these file chunks and how the file integrity is ok.The problem now is to determine where the data got out-of-sync.
I tested the memory with memtest86+, all but the bit fading test. I was expecting to see some failing memory module, but there wasn't anything. Everything is ok.Filesystem is Ext4, over LVM2, over a 3-disk RAID5 array.Ext4 is considered stable, and if data were inconsistent between disks, mdadm would have warned. But there is nothing in the logs. S.M.A.R.T. error logs are clean, the disks are new (have less than 30 days of "power-on-hours").I'm looking for information about any data-loss bugs in my current kernel (2.6.35), but there doesn't seem to be anything, as far as I looked.what else I could check, or where exactly could be the defect/bug?It is a Ubuntu 10.10 64-bit, Core i7 930, 6 GB non-ECC RAM.
Update: I confirmed that the files are being correctly written to the disk, the pages are being altered after they are read from disk, while in memory. I did a lot more memtests (I left it doing bit fade test overnight),and still nothing. All memory modules seem ok.Some more tests:
~% md5sum media.iso
cc8bcf1ce67ff7704eadc2222650c087 media.iso
~% cp media.iso tmp[code]....(direcat is a version of cat that reads with O_DIRECT, that is, bypassing page cache)There is a clear pattern: it always happens to the 2nd byte in a 16-byte alignment. In that byte, almost always the bit 4 (LSB) flips to one, but there was one instance where bit 2 flipped to zero.
View 1 Replies
View Related
Nov 22, 2010
I am reading slab allocator, it defines slab cache, i am quite confuse is it same as hardware cache?
View 2 Replies
View Related
Oct 10, 2010
I have a 2 TB disk in an external SATA dock, formatted with a single ext3 (Linux) partition, which doesn't show up in the Windows 7 Computer Management->Disk Management utility, even as a raw/blank disk. I've verified that there's nothing wrong with the disk by connecting it to my Linux machine and mounting it, and I've verified that the dock is functioning properly by connecting a different FAT32-formatted disk, which mounts flawlessly as expected.I realize that I can't actually read the ext3 partition without additional software (e.g., Ext3IFS), but why doesn't the disk show up at all? Is there some sort of stupid anti-Linux filter built in? Is there any way to force Windows to recognize the disk, so that I can at the very least use direct block access with it?
Background: I want to clone an identical 2 TB disk onto this one. Due to my hardware layout, it's much easier to have the source disk attached to one machine and the destination disk connected to another, and do the clone over the network (the network is not a bottleneck with switched gigabit ethernet), than it is to hook them both up to one machine.(1) I did this once before when both machines were running Linux, but I've since upgraded the destination machine and decided to switch back to Windows for regular desktop use. I've got Cygwin installed, and have verified that the same basic method (dd + nc) will work, but I can't do anything if Windows doesn't even consider the destination disk to exist.I only have one eSATA port on each machine. Opening them up just to do this clone is a rather large annoyance. Also, since this is my backup disk, I'd like to eventually automate the cloning from the active disk to another one that I regularly swap with a third disk that I store off-site.
View 1 Replies
View Related
Dec 26, 2010
Using KInfoCenter | Memory module it shows my 2GB of ram. Approx. 14% is used for Application Data and Disk Cache has been anywhere from 29-35%...leaving approx 1GB free. Can this 'Disk Cache" be reduced leaving more memory free or is this determined by the OS?
View 3 Replies
View Related
Nov 16, 2010
60 gigabyte SSDs are down around US$100.
It seems like the optimal use would be as a cache for the regular hard drives in my computer. Eliminating the need for a fast hard drive, so I can just use a slow 2TB (~US$100) drive with a SSD cache.
Is there a good way to do this yet?
It seems like it would be nice to be able to exclude some files from caching, for things like bittorrent.
Reviews of some SSDs: [URL]
View 9 Replies
View Related
Jun 24, 2010
Squid servers cache size ondisk automatically increasing and decreasing,how I would resolve this issue
View 6 Replies
View Related
Feb 22, 2010
I'm working on a few servers running centos and using postfix. I don't know what the exact problem is, but we are having problems with the disk space being maxed out at 100 gigs. What we think the problem is...is that postfix is either caching or logging all the emails we send out. We sent 250k emails (500kb apiece) over the weekend and we were having trouble with that quantity. It seems some of those email were queued up for retry sending...but we didn't have sufficient disk space for that? Something broke - I'm not sure what.
What I want to do is to find and change the config file that has to do with postfix email retrying - possibly limit this (not sure if this will fix my problem). Or, turn off /limit any way that postfix logs/caches emails so that it won't take up all the disk space when queued up for retry... Again, I'm totally lost here (on both what's going on, and how to fix it). I'm not sure what more information is needed to address this problem
View 1 Replies
View Related
Sep 24, 2009
I don't understand this error nor do I know how to solve the issue that is causing the error. Anyone care to comment?
Quote:
Error: Caching enabled but no local cache of //var/cache/yum/updates-newkey/filelists.sqlite.bz2 from updates-newkey
I know JohnVV. "Install a supported version of Fedora, like Fedora 11". This is on a box that has all 11 releases of Fedora installed. It's a toy and I like to play around with it.
View 12 Replies
View Related
Jan 5, 2011
I was laughing about klackenfus's post with the ancient RH install, and then work has me dig up an old server that has been out of use for some time. It has some proprietary binaries installed that intentionally tries to hide files to prevent copying (and we are no longer paying for support or have install binaries), so a clean install is not preferable.
Basically it has been out of commission for so long, that the apt-get upgrade DL is larger than the /var partition (apt caches to /var/cache/apt/archives).
I can upgrade the bigger packages manually until I get under the threshold, but then I learn nothing new. So I'm curious if I can redirect the cache of apt to a specified folder either on the command line or via a config setting?
View 2 Replies
View Related
Mar 6, 2011
I installed squid cache on my ubuntu server 10.10 and it is work fine but i want to know how to make it cache all files like .exe .mp3 .avi ....etc. and the other thing i want to know is how to make my client take the files from the cache in the full speed. since am using mikrotik system to use pppoe for clients and i match it with my ubuntu squid
View 1 Replies
View Related
Mar 24, 2010
Is it possible to boot from a disk image file(containing linux) file that resides inside windows and add a bootloader entry for booting from the disk image.?
View 2 Replies
View Related
Jan 9, 2011
I have a Centos 5.5 system that had 2 primary partitions (2nd is setup as LVM with multiple LVM partitions) and then installed Windows 7 as Dual Boot.
However, Windows 7 has installed a 200MB system partition which is GPT/EFI as partition 3 and the Win7 OS as a Primary Partition.
I have a heap of space undefined after this fourth primary partition.
However, as 4 primary partitions have been used, I can no longer create an extended partition to utilise this.
As such I would like to know what is the best and safest way to proceed, and if possible step by steps instructions for the best option eg:
1. Delete the Windows 7 System Partition and create the extended partition (I expect this will prevent Windows from booting)
2. Use something like partition magic to change the Win 7 OS Partition 4 to an extended partition (Not sure if this will work)
3. Make changes to the overall system including both Linux and Windows so that it will use GPT only (I have
had no experience with GPT so this is a bit scary)
4. Other?
View 2 Replies
View Related
May 27, 2011
I'm trying to help out a Windows Vista user by rescuing their data from a failing hard disk. When their laptop stopped booting, I immediately pulled the disk to get as much as I could off of it using another Windows box, but the process took days and ultimately choked on multiple bad sectors and stopped responding. I then hooked it up to my Ubuntu box via a USB disk dock and ran a ddrescue on the Windows partition. The operation took a week, then seemed to get stuck for another week on the "splitting failed blocks" phase. So I have an dd image and a log to go back to, but when I resume that process it still seems to use the disk and I don't see much progress.
I then tried a plain `dd` on the disk with `conv=noerror,sync` options, and that has been running for a few days now, but with input/ouput error messages every few seconds and seemingly no records going in or out. I think that's a bad sign.What's the best, and fastest, way to get the most data off the disk as possible and into an image file, and then perform any necessary operations on the image file so that the disk is no longer needed (since it seems to be just about dead)? Er, just realized I'd put "Windows 7" in the post title, but this is a Vista partition, and I can't change the title. They're pretty similar, with one OS being much less useful than the other, but I thought I'd better acknowledge my mistake.
View 14 Replies
View Related
Mar 17, 2010
i ahve to boot it from the command like an executable but i dont know how to exactly and when i thought i had it figured out i didnt know how to write the name of the file in the disk ubuntu just kept saying file or directory not found
View 14 Replies
View Related
Sep 29, 2010
I have 2 hard disks and all together I created 6 partion in it.On primary I installed Windows XP and REST 5 partions were having data.From Second hard disk ,I copied all the first partion data to another partions and installed Red Hat 7.It took only 3 GB all togethr for Linux Distribution.Now I need to fetch my Windows data back.I logged from the Windows XP disk and I could see the second disk through disk managemnt .But it was showing "Unhelathy".But I will not be able to retrive the data.So can Anyone give soln for this.
View 2 Replies
View Related
Mar 15, 2010
What is the most efficient way to match the partition tables on two different hard disks?
I have saved the partition tables using dd command in linux.
The partition tables are from a Windows system.
View 1 Replies
View Related
Jul 2, 2010
I'm using andLinux and for whatever reason, emacs seems to think that the file I'm editing has been changed every time I try to edit/save and keeps reprompting me. Very annoying. Is there a way to make emacs stop checking the file on the disk?
View 1 Replies
View Related
Oct 22, 2009
I need to install windows xp and linux in separate hard disk on my computer.Is it possible?
View 2 Replies
View Related
Apr 5, 2011
when i check disk partition in ubuntu 9.04
using #fdisk -l.my friend show me like this.i have 4 hard disks, but he didn't.. but only 3.Was there anything wrong? when i installed linux and maked config?(e.g> i have miss on setting disk partition matter) and when see below result, the device sda2, sda5 start same address, end too.is there anything wrong with my disk? i have seperated 4disks.. C:,D:, G:,H:
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe9ffe9ff[code].....
View 1 Replies
View Related
Jul 11, 2011
I was wondering if there is some way to determine when a file finishes writing to a directory on both Windows and Linux (obviously, they will probably be two different commands). This is mostly so that, instead of constantly polling a directory for new non-temp files, I can set up a program to simply listen for the completion of a write-to-disk (it seems better to do things that way).
View 3 Replies
View Related
Feb 17, 2010
i was looking for a way to stop my menus taking a few seconds to load my icons when i first open them and found a few guides suggesting using the gtk-upate-icon-cache command, but with the any colour you like icon theme i'm using (stored in my home folder .icons directory) i kept getting a "gtk-update-icon-cache: The generated cache was invalid." fault i used the inbuilt facility in the acyl script to copy the icons to the usr/share/icons directory and tried the command again, this time using sudo gtk-update-icon-cache --force --ignore-theme-index /usr/share/icons/ACYL_Icon_Theme_0.8.1/ but i still get the same error. i tried with several of the custom icon themes i've installed and only 1 of the first 7 or 8 i tried successfully created the cache.
View 4 Replies
View Related
May 11, 2011
error: unknown filesystem grub rescue>
Background: My mother's HP laptop had Ubuntu and Vista on it, Ubuntu my brother's doing. He decided he wanted to take off Ubuntu yesterday (he had forgotten the password), and deleted the partition that it was contained within. The computer now boots to this error.
Inventory: We no longer have the install disk for Windows Vista, he cannot tell me what version he used of Ubuntu, what partition it was on, any of the specs for the machine, or generally any information about the system. All I am aware of is that error on the boot-up screen. I have nothing else to work with.
I would like to remove Grub, and Ubuntu, and leave Windows intact (the request of the owner of the computer), but I have no idea what commands I could use to get rid of either when I can't access Windows, or how to properly remove them if I did access Windows.
View 10 Replies
View Related
Sep 28, 2010
Background: I'm a 20+ year Mac user, switched four years ago to Linux.I have NO experience with Windows to speak of. Whenever I've gotten a new box, it's either been home-built or a Mac.Now I have a new Acer Aspire AS7551 laptop, and although I have not yet booted into Windows, it comes with a Windows 7 Home Premium license. I've been running it with Ubuntu 10.04 off a flash drive and everything works perfectly.
What I want to do: I want to wipe the disk, install Lucid, and run Windows in Virtualbox (for the sole purpose of watching some Netflix and learning a little bit about Windows 7). For the purpose of this question, please accept that I DO NOT want to dual boot.The problem: From what I can find, I can't use the "recovery discs" that came with my laptop to create a virtual machine. This is confusing, because it seems like I paid for a legitimate Windows license for use with this laptop.
Questions:
1. Would MS exchange my recovery discs for a regular install disc to go with my license code if I explained this to them?
2. If not, is there a source for Windows ISOs that I can safely use with my license? (I am aware that the legitimacy of that approach is questionable, but I am quite comfortable with the ethics since I paid for Windows 7, own the license, and only want to use it on the same computer.
3.If not,is there any way in heck I can use my recovery disc to create something Virtualbox can work with?
4. If not, how are all of you people getting your Windows virtual machines going??
(I have read that Acer is a major hassle to get a refund from for the "Windows tax" so I might as well be able to use it--this is MS' chance to win over a fan if only they will let me!)
View 9 Replies
View Related
May 9, 2011
Keep getting the same error message trying to install via Wubi under Windows XP. When I download and run wubi.exe, I get an error message that I can't dismiss that reads: Windows - No Disk There is no disk in the drive. Please insert a disk into drive .(sic) I've tried downloading the appropriate .iso and putting it in the same directory as wubi
View 2 Replies
View Related
Jul 7, 2009
I have a sata 320 gb with mandriva linux 2009.1 on it.And it is what curently atached to my cpu. It is shown as 'sda' in the partition table.I also have another 40gb hard disk with windows xp installed on it.It is shown as 'hda' in the partition table . Now what i want to do is attach this 40gb hard disk to my pc and configure grub on my 320gb hard disk('sda') so as to boot windows xp(which is residing on the second hard disk,'hda')Can anyone tell me if what im doing is feasible or not? If it is feasible,can anyone suggest me how to get it working. I know i just need to add 2-3 lines to my grub.conf, but dont know what exactly i need to write.
View 3 Replies
View Related
Mar 7, 2010
when i type arp -n it shows nothingif i type arp -vit showsskipped :0 found :0I want to look at my ip and mac addresses in arp cacheif i change my ethernet ip address it is not reflected or stored in the cache.
View 4 Replies
View Related
Apr 25, 2011
I use Ubuntu, and Google Chrome mostly. How can I kill these HTTP headers, so that my browser caches this data? I believe it is XHR. Here are the relevant HTTP response headers:Cache-Control:no-cache, no-store, must-revalidate Pragma:no-cache.I also have Firefox, is there a plugin or something I can use to not respect "no-cache"?
View 1 Replies
View Related
Jun 28, 2011
I have a desktop and laptop at home, both running the same flavour of Debian Testing. How can I configure apt on the laptop so that it first tries to download the packages from the desktop before going to the Internet?
View 2 Replies
View Related
