General :: Write-through RAM Disk Or Massive Caching Of File System?
Feb 17, 2010
I have a program that is very heavily hitting the file system, reading and writing randomly to a set of working files. The files total several gigabytes in size, but I can spare the RAM to keep them all mostly in memory. The machines this program runs on are typically Ubuntu Linux boxes.
Is there a way to configure the file system to have a very very large cache, and even to cache writes so they hit the disk later? I understand the issues with power loss or such, and am prepared to accept that. Crashing aside, in normal operation the writes should eventually reach the disk!Or is there a way to create a RAM disk that writes-through to real disk?
I've surprisely recognized that it's possible to write a filesystem on a hard disk without any valid partition. Well, the general advantages of partitions are clearly. But what are possible disadvantages or limitaions if you don't use a partition (e.g. if you want to use the complete space as one volume for data mining or so)?
I was wondering if there is some way to determine when a file finishes writing to a directory on both Windows and Linux (obviously, they will probably be two different commands). This is mostly so that, instead of constantly polling a directory for new non-temp files, I can set up a program to simply listen for the completion of a write-to-disk (it seems better to do things that way).
I understand that the linux pipe is a buffer and that any data written to it will stay there until it is read, and if the max capacity of the buffer is reached, any additional writes will block (by default).
HOWEVER, the behavior of the pipeline below suggest that the write operations are buffered/cached before ever being written to the pipe on the client side here is write.sh, which creates 1000 byte string and writes it 100 times to stdout... the idea being that it'll block as soon as the 64kb linux pipe size is reached:
[Code]...
This is not what I was expecting: I was expected that once the capacity was reached, any reads would be followed immediately by a write to take advantage of the freed space. Instead, the blocked write operation seems to wait for some random amount of time/space to free until it unblocks and writes.
how to make removable media (e.g. USB sticks) not have any write caching. I want to prevent data loss when they are removed after file copying appears done but before write caches are written. I'm using Gnome on Squeeze.
I've found suggestions of adding the 'sync' mount option to /system/storage/default_options/vfat/mount_options in the Gnome configuration. However this doesn't seem to completely eliminate write buffering, as the drive activity light continues for several seconds after file copying appears done, and unmounting drives produces a dialog box which says to wait whilst data is written to disk.
The problem arises when I try to create a sub-directory inside the mounted directory. All the newly created sub directories become write protected.
I am accessing this file system from R software and it needs to write/create directories in side this mounted directory.
how can newly created sub-directories will become automatically writable, so that R can create new sub-directories and write data inside those directories.
I want to simply mount an ext4 file-system onto a normal mount point in Ubuntu (/media/whereever), as read-writable for the current logged-in user, i.e. me.
I don't want to add anything into /etc/fstab, I just want to do it now, manually. I need super-user privileges to mount a device, but then only root can read-write that mount. I've tried various of the mount options, added it into fstab, but with no luck.
Information on the net seems very sparse or outdated for how to go about booting to a RAM disk. I need to be be able to boot a PC without a hard drive in it. I want to be able to PXE boot a PC and supply it with a RAM disk image that also contains the contents of the root file system (obviously stripped down enough to keep the file size small and the boot up time fast).What I have gathered so far is that I need to extract the contents of the initrd.img file, add files as necessary, and repackage the initrd.img file. What I get confused on is how to configure the kernel line parameters to tell it to boot to RAM and not the hard drive and how to go about modifying the init script in the initrd.img to not switch to the hard drive for the root file system. I can't find anything on the net that describes concrete steps on how to go about accomplishing all of this. I'm aware of the existence of Live CD's, but I need to be able to boot the PC without relying on a hard drive, CD, or any other external media. It needs to get all of its contents from the PXE boot server and boot to RAM only. I have the PXE boot side configured successfully. Also, putting the root file system on a NFS share is also out of the question.
I have been running Ubuntu as my sole OS for about a month now, and aside from Computer Janitor, I have found no drive/fs cleaning tools, or defragging tools. I know this isn't Windoze, but surely there must be a way of maintaining your hard disk. I mean installs/uninstalls must leave detritus, and file moving must cause SOME fragmentation. Are there any apps, systems or methods in place that provide this functionality?
I run Ubuntu Netbook 10.04 on my EeePC 1005HA. I'm going to get a SSD for it eventually, but I can't afford one right now so it's running from a 200GB hard disk I scavenged off a dead laptop.
I went in power management and set the option that says "spin down hard drives whenever possible", but this accomplished a whole lot of nothing - whenever the computer is on, the drive's spinning. I ran hdparm -y and the drive clicked off, and then promptly spun back up after a few seconds. Iotop shows occasional tiny bursts of activity from "jdb2/sda1-8", which I don't really know how to interpret, but I don't have anything weird installed so I'm assuming this is normal system operation.
Now, what I need is some sort of application, utility, command - anything - that forces the computer to keep all filesystem changes in RAM with the drive shut down; every five/ten minutes or so (this would hopefully be configurable) it spins up the drive, dumps the filesystem changes to it, and spins it down again.
I realize this presents data loss risks related to crashing and poweroffs when the cache hasn't been dumped to disk, but I'm willing to risk it as Linux never really crashes at all, and since it's a netbook power failures won't cause unexpected shutdowns.
I am using kernel 2.6.32.21, and my hard disk is West digital WD10EARS-00Y, 1TB. This disk is just for data, I made 2 partitions on it, each has half. And I have another small disk for system. I am using ext3.
this is my fdisk /dev/sdc Device Boot Start End Blocks Id System /dev/sdc1 1 60800 488375968 83 Linux /dev/sdc2 60801 121601 488384032+ 83 Linux
[Code]....
I ran some dd to test the write throughput on /dev/sdc2. If I run it in /data2, I got around 70MB/s. If I create some directory, say /data2/dir/, and run dd again, I might get 60MB/s. Sometimes I still get 70MB/s, sometimes I get 60MB/s, differs for directories.
I wonder if this is because the allocation policy of file system, ext3, or this is from my hard drive?
I was wondering if a standard Debian 8 system with Gnome desktop does any kind of local dns caching, and if so, what the command is for clearing it. (Assuming I haven't purposely installed any DNS server software.)
I found multiple posts on the Web about unix DNS caching, but with widely different answers across distributions and across time.
I have been using Wheezy for two years, and everything was OK, so I decided to upgrade to Jessie. In fact i decided to perform a clean install of jessie, so I formatted the partition where Wheezy was installed, and the /boot partition, and I installed jessie in those partitions.
As I had done with Wheezy, I installed jessie in an encrypted LVM, and the installation was ok (well, almost everything was OK, because grub and plymouth were not working, but I will open another topic about it).
First thing I did after installing Jessie was editting sources.list in order to download a few programs (plymouth, firmware linux non free, libdvdcss2, gufw, menulibre). Moreover, I downloaded a few progrmas from the Debian install DVD (flashplugin, VLC, chromium, clamav). i did not perform a dist upgrade because i was not at home, and where I was I did not have a Wifi connection (so I was using my mobile connecion -USB ethernet with an android phone-. BTW I had just performed a hard reset I had not installed any apps after that, so that mobile phone was "clean").
After that, I created a desktop user account, and I rebooted the laptop . When i rebooted, I started to tweak my user account: I edited dconf, and the gnome shell theme; and I started my mobile connection to download three extensions for gnome shell (window list, simple dock and activities configurator. I had used those extensions with Wheezy, and I had never had any problems). Ten minutes after I started the mentioned mobile connection, I reveived a SMS as I had used over 300 MB. Gnome monitor showed that I have downloaded 300 MB, and the android native data usage app showed the same. I did not download any video or music neither watch any videos on youtube, dailymotion..., I did not visit any suspicious web
I had a look at the apt logs and I din not find anything significant (I was using a no sudo user account, so i was not able to perform a dist.upgrade), I had a look at the download, video, music, picture folders and i din not find anything. I tried to check the iceweasel cache folder, but there were so many subfolders I could not check everything.
Well appears to be that i have two partitions..one of 300gb i did on windows time ago and one of 100gb where i place my DEBIAN LENNY 5.0, the problem is that i cannot write into it..only i can copy stuff from it.
My system decided to crash on me, hard. It was humming along happily for about 2 months and now doesn't boot. If I boot from hard-disk, I get grub. Launching the first kernel choice hangs. I thought maybe the install was corrupt, so I booted from usb install disk. The usb hdd didn't boot; something about an error trying to access /dev/sda . Unplugging the internal disk and plugging in the usb install disk does result in the system booting. Plugging in the internal disk in a running system usb-booted system does not result in the system detecting the disk.
How do I know if the disk is physically broken? This seems unlikely since it does manage to launch grub consistently. Or is this still possible? How can I try to mount whatever is left? The usb install disk doesn't even list the /dev/sd*. Any pointers on how to reformat the drive if it's not being mounted?
I just found that I could perform write operation using a normal user account to a file system I mounted with the commands as followed:
sudo mount -t ntfs /dev/sda1 /mnt/disk/
This is the corresponding entry in the output of "mount" command: /dev/sda1 on /mnt/disk type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)
As far as I remember, when using a normal user account, I had to use "sudo" to perform any write operations (mkdir, rm, etc) to a device mounted using "sudo". But now it seems to be changed.
Do I remember wrong, or did Karmic have any updates change this setting? (I never manually changed user settings, except that I added a root user, but I never used it.)
OS: Karmic(up2dated) Kernel: Linux stephen-laptop 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 16:20:31 UTC 2009 i686 GNU/Linux
1. What can I use to read/write to my ext4 file system in Win7 x64? 2. I use Macbuntu. Is there any way to get a translucent top bar 3. My computer seems to be running hot while on Ubuntu. The fan speed seems increased. It goes back to normal on Windows though.
I have a netbook I'm not using and which I transformed into a server with Apache, Tomcat6, Netatalk, Webmin, BIND9 and Tor.
Problem is, the disks never stop spinning because all of the programs write a few kb at least every few seconds to disk, even when nobody is connected to it.
My question is: Is there a way to have the computer boot from disk like normal (maybe even a squashfs), keep ALL CHANGES to ram and then save to disk when either the ram is full (unlikely because the server is rebooted every few days) or at shutdown?
I thought about a mixture of ramfs and unionfs but I'm not good enough yet...
My problem is extremely slow write on hard disk and 100% cpu usage and it happens when I want to write something on the hard derive not any other external derive.
Tried a fresh ubuntu install. No change. I am not even sure if it is a software or hardware problem.
When I ls -l /etc/passwd, -rw-r--r-- 1 root root /etc/passwd When I login as myself, and rm /etc/passwd, it asks: rm: remove write-protected file '/etc/passwd'? If I say yes, will it actually delete the passwd file?
I have windows and linux distros on my /dev/sda. I tried to install fedora 13, but after reboot I cannot boot up to any of installed systems. I'm getting: Non-System disk or disk errorreplace and striky any key when readyWhen I use ubuntu 10, boot from first hard disk, I'm able to get GNU GRUB version 1.98...nd boot up to any system. but without ubuntu in a cd-rom I'm getting this error.I tried: grub-install /dev/sdX in one of my installed linux distros, but without any success.
I need to write a script to report useful information on disk utilization for each user's home directory.For each directory I need to show: 1. the long listing of that directory entry (but not the files in the directory), so that I can see the rights and owners of the directory.2. The amount of disk used by that directory, in human-readable format, including subdirectories. I need to have two lines for each user one after the other. For example:
/home/user1 directory info /home/user1 disk usage /home/user2 directory info /home/user2 disk usage
The script will assume that all users, except user root, have their home directories in the /home directory (no need to do anything with the /etc/passwd file). And if the administrator adds or removes users, the script should still work correctly (so the script shows the information for all current users).
Here's what I do know. The command "ls -ld /home/user's_name" will give me the info I need for #1. And the command "du -hs" will give me the info I need for #2. What I don't know is how to grab each individual directory in order to apply the above commands to each of them in order. ???
I'm using Ubuntu 10.0.4. I downloaded an old script for starting/shutting down a service I have, and evidently "initlog" doesn't exist anymore. What is the correct way to write to the boot (system?) log?
Booted up system and it stated please insert system disk press any key.So it sounds like my startup got corrupted? I running centos 4.5 on it (old yes) Is there a repair I can perform by a disk? Would it matter if I chose i386 or x86? I cant remeber which version was installed on it.
i just installed ubuntu 9.10 onto my windows vista laptop. i ran ubuntu update manager but it tells me i'm low on disk space. system monitor tells me that i still have 50.2 GB of space but the problem is that i only have 68 MB left in my / File System. how can i increase disk space in / File System?
I was trying to install Ubuntu desktop and laptop edition on a Sony Vaio netbook from a USB drive, but after I select the entire disk to be used and hit enter I get this message No root file system is defined. correct this from the partitioning menu. If I try to start windows I just get s black screen.
There isn't any separate file system for /home and we have only one (/) root file system for everything else on the system. Is there any way that we can still implement quotas for users through their home directories was mounted on (/) root file system. Do we need to have a separate file system (/home) compulsory for implementing disk quotas?
