General :: Why Is JFS Faster Than A Raw Device For 1.5TB Disk
Aug 19, 2010
I have a 4 * 1.5TB RAID5 disk array (software linux RAID, formatted with jfs) on my Fedora 12 system and want to expand it by adding another 1.5TB disk. I have added a drive to the system and conducted a simple performance check on it to make sure it was functioning properly:
Code:
# dd if=/tmp/bigfile.dat of=/dev/sdg1
5478774+1 records in
5478774+1 records out
2805132609 bytes (2.8 GB) copied, 168.77 s, 16.6 MB/s
But 16.6 MB/s is lousy. I ran an iostat -dmx 2 on this drive at the time of this lousy performance, and typical output was:
(note that sda and sdb are a linux raid mirror set for the / filesystem that holds /tmp). I formatted the new drive (/dev/sdg1) with jfs and mounted it under /mnt2:
Code:
# jfs_mkfs /dev/sdg1
jfs_mkfs version 1.1.13, 17-Jul-2008
Warning! All data on device /dev/sdg1 will be lost!
Continue? (Y/N) y
[code].....
I would like to know what the performance will be like before I add the disk to the array because I don't want to wait for the whole array to be rebuilt before finding out my array is performing badly. The array is used as part of a mythtv system and has up to 6 simultaneous recordings running on it, so it needs to perform well.
Here is my brief hardware and software detail in my production environment : AMD Phenom X4 3.4gHZ (Over clock to 4gHZ, 8G of Memory, 1TB 7200rpm Harddrive, Running Ubuntu server 10.10.My web production environment were pieced together 3 weeks ago.Here is my dilemma. started out with less that 40 users and now hitting 4,000 unique users per day.Now I am thinking I need faster write to disk and backup of data so I am thinking about putting together a Raid5.
I preparation for this.I have bought a new motherboard, AMD Phenom X4 3.6, and 2 more 2TB 7200rpm (Currently, I have a 2TB 7200rpm not used much)Been digging around this forum for posts with raid setup but still not sure how to seamlessly moving the some 10Gig of data from my current running prod environment once I have RAID5 installed on this new machine via the LIVE Ubuntu Server CD.
I have a Corsair R60 ssd disk which is a disk with both sata and usb connectors. But the usb thing seems to be a bit non-standard, or maybe its just my fedora linux.When I insert the disk using a usb cabel to a running Fedora 14 linux system, a device called /dev/sg3 is added but that is all. No new /dev/sd* device is created so I can't mount the disk.
So the disk is there. (The last entry) but my linux will for some reason not see it as a usb hard disk. When I insert other usb disks they work fine. It is only this specific disk which causes problems. I have tried on 3 different computers with the same result.
A hint to the problem may be that if I add the disk to a windows system(With usb) the disk is called "A fixed disk" and not a portable disk as expected. The disk works fine with linux If i connect it with the sata cabel, but I would really like to have it working with usb too. (To mount it on computers without sata).
Added:I did try to mount /dev/sg3 but mount say that its not a block device. (File say Its a character special device).
Added output from dmesg:
[ 97.454073] usb 7-1: USB disconnect, address 2 [ 105.913055] hub 2-0:1.0: unable to enumerate USB device on port 3 [ 107.048054] usb 2-3: new high speed USB device using ehci_hcd and address 5 [ 107.162900] usb 2-3: New USB device found, idVendor=1b1c, idProduct=1ab8
[code]....
I found an other guy with exactly the same problem [URL] so I think its beginning to look like a bug in the drives firmware or in the linux kernel.
Final update:Corsair have said that the disk design is broken and there does not seem to be any way to make it work.
After 2-3 partition an extended partition automatically created in which I am not able to create specified capacity i.e., say I want 150g of /photos partition, the /videos partition is automatically reduced and a free space at the end appears. Some free space is always there which i am not able to understand. Nevertheless i clicked to create, but I get an error viz. 'device not created'.
I am having problem with the root disk / fill up. So now that i have the disk information provided from SAN team Id 0:3, how would i do to add this device to / to increase space on Redhat Enterprise server 4?
I have one hard disk (call her HDA) that contains nothing but a single ext4 partition containing a backup of all my important data. Last night I did a clean install of Ubuntu 10.10 on my primary hard disk (call her HDB) and from there proceeded to upgrade directly to Ubuntu 11.04 upgrade. In 10.10, I was able to read HDA just fine. However after the upgrade, I can no longer mount this drive. When mounting from file browser:
Code:
Error mounting: mount: wrong fs type, bad option, bad superblock on /dev/sda,missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so The end of dmesg said the following:
Code:
dmesg | tail [ 82.130904] EXT4-fs (sda): bad geometry: block count 122096646 exceeds size of device (122096381 blocks)
my hard disk has a block count greater than the size of my device. I've done my background searching on this and tried a command line utility I've never heard of before:
Code:
# sudo e2fsck /dev/sda e2fsck 1.41.14 (22-Dec-2010) The filesystem size (according to the superblock) is 122096646 blocks The physical size of the device is 122096381 blocks
[code]....
this is as far as I've gotten. This drive holds over a decade's worth of work for me and is extremely valuable. I really didn't think that the Ubuntu upgrade process would mess with this drive, seeing as the Ubuntu install was contained on an entirely different drive. What is it that I need to do to restore my drive to working status?
I have an old laptop (PIII 800MHz, with 256 RAM) that I wish to use as my home server: it'll have to serve just two people, so I think that I'll be more than ok as for the RAM and the CPU.The issue is about data, because the internal hard disk is a 12GB, that is...ridicolous! I have more than 60GB of mixed storage and counting (images, videos and music) in an external usb hd. I could put the hd in my desktop pc just to serve the big files through ethernet or let it inside its usb box attached to the laptop.
The question is: which of these solutions will be the fastest? USB 1.0 attached to the server (laptop) or internal hard disk serving files via 10/100 ethernet to the laptop on demand?what about your experience? Is the difference based on a human notable scale?
Why is it faster to have more primary partitions when using Linux? Please give some real examples; I know some theoretical reasons but I don't understand them well.
I often have to transfer huge data over our LAN from one computer to another. The size of the files varies and can be somewhere from 2 GB to 50GB or more! The only accepted connection protocol between machines are ssh; and rsync and scp are the only options available for copying over network (unison is not installed). I usually use rsync with "-z" option to copy over network. if "rsync -z" is faster than "scp" for data transfer?
I find myself grepping the same codebase over and over. While it works great, each command takes about 10 seconds, so I am thinking about ways to make it faster.So can grep use some sort of index? I understand an index probably won't help for complicated regexps, but I use mostly very simple patters. Does an indexer exist for this case?EDIT: I know about ctags and the like, but I would like to do full-text search.
I'm using Linux Mint 8. However, internet browser speed is very slow as compared to using the same computer with Windows. In Linux, I've used Firefox 3.5 and Epiphany 2.28. Epiphany runs somewhat faster than Firefox so I mainly use it for the internet.
What I would like to know is if Epiphany browser 2.30 is noticably faster than 2.28? Should I go thru the trouble of upgrading to Mint 9 inorder to be able to use Epiphany 2.3 - as well as Chromium 8?
I am using Red Hat Linux enterpriser 4 using two physical LAN cards.There are two different ISP internet lines coming into a single gateway computer. Is it possible that if in our one network we are using 2 MB bandwidth and from another network we are getting 2 MB bandwidth.(Two different Networks) and combine it into 4 MB. In bonding it combine the bandwidth or it do loadbalancing ?
The question is can we accumulate or add these two bandwidth in Linux machine and it sent total bandwidth 4 to the end users ?
Lan Card A IP Address: 192.168.1.250 2MB Bandwidth l l l
Is there any C function that will translate UUIDs into device names? I have a little graphical mount tool that can read user-mountable device names from /etc/fstab and lets you cycle through the list and mount or unmount them. But it doesn't work with UUIDs, which are preferred these days. Is there any way around this?
2 days ago i have tried to install fedora 11 ...but after i did achieved installation successfully ...and rebooted system ,the device hasn't booted from hard disk ,alternatively it has booted from the network (Broadcom)- the third booting choice in bios - . i have tried three different DVDs ....but no thing changes ...!
Sometimes when I restart, my NTFS drives will mount in incorrect directories. It seems to only happen when I plug in USB devices such as flash drives, and keep them in when I boot. I have the fstab file configured correctly, but it still resorts to some odd default mounting points. Edit suposedly the device name changes whenever I boot with a flash drive plugged in. Is there anyway to mount a disk to a dir without pointing to its changing device name.
I've mounted Ubuntu 10.04 ISO using Daemon Tools and Virtual CloneDrive, but I end with thisI've disabled my DVD drive in device manager, but I still get it no matter how many times I try.
I am developing for a Linux based device for which the HOT PLUG option is deactivated. As part of optimizing the code, we also don't want to create device files for unused devices. We understand that both USB attached and fixec SCSI hard disks would create device files like /dev/sda,/dev/sda1 /dev/sdb, /dev/sdb1 etc. Is this understanding correct?
In the case of USB attached SCSI devices, would driver create this device file entry? How is it created? Can somebody please tell me how it is being created automatically. In case I attach a fixed SCSI hard disk before boot up(and create device file /dev/sda1), would USB SCSI device driver create device files starting from /dev/sdb, automatically.
For backup purposes, I built a RAID6 array on USB. When during the night the backup filesystem is being mounted, every so many days, a random disk fails:
Code: sd 9:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 9:0:0:0: [sdb] Sense Key : Illegal Request [current] sd 9:0:0:0: [sdb] Add. Sense: Invalid field in cdb[code].....
In other words, my RAID set fails every so many mounts, but I know how to fix it.What I want to find out is:
- Why is a random drive failed every now and then?
- How can I prevent that drive from being failed?
BTW: I have a second RAID set that has been functioning for years without error, so the setup I use must be correct. The only difference between both RAID arrays is the different vendor of the disks.
I have installed vmware in windows server 2003 enterprise edition. When I installed I changed the default installation folder from c: drive to i: drive, as I do not have space in c: drive. After I captured red hat linux cd in vmware and reached partitioning section and continued I got a message like "an error has occured no valid devices were find on which to create new file systems, please check the hardware for the cause of the problem". The machine is IBM server....
I'm kind of new to programming in Linux & c/c++. I'm currently writing a FileManager using Ubuntu Linux(10.10) for Learning Purposes. I've got started on this project by creating a loopback device to be used as my virtual hard disk. After creating the loop back hard disk and mounting it has the following configuration.
Disk /dev/loop0 doesn't contain a valid partition table Now what I want to do is develop a c++ program to read & write files to this loop back device,which I'm using to simulate an actual hard disk,at the blocks & sectors level. So far I've come up with the following code. But I'm still unable to read files from the hard disk one block at a time.
Code: #include <iostream> #include <stdio.h> using namespace std; int main() { char block[512]; int length=0; cout<<"Implementation of the File Handler Read Method..."<<endl; FILE *f = fopen("/dev/loop0", "r"); if(f == NULL) { cout<<"Error In Opening the HardDisk File Retuning Error..."<<endl; return -1; } //Read One Block of Data to Buffer length = fread(block, 1, sizeof(block), f); /* Do something with the data */ cout<<"Length : "<<length<<endl; return 0; }
When I run this Program All what I get is the message for NULL. "Error In Opening the HardDisk File Retuning Error...". So I could open the loopback device as a file an access it at the sectors & block level.
I don't understand the results of a simple performance test I ran using two basic scripts (running on a high end server):
perfVar.zsh :
#!/bin/zsh -f MYVAR=`cat $1` for i in {1..10}
[code]...
Performance test result:
> time ./perfVar.zsh BigTextFile > /dev/null ./perfVar.zsh FE > /dev/null 6.86s user 0.32s system 100% cpu 7.177 total > time ./perfCat.zsh BigTextFile > /dev/null ./perfCat.zsh FE > /dev/null 0.01s user 0.10s system 91% cpu 0.118 total
I would have thought that accessing a VARIABLE was way faster than reading a FILE on the file system... Why this result ?Is there a way to optimize the perfCat.zsh script by reducing the number of accesses to the file system ?
I have a 2 TB disk in an external SATA dock, formatted with a single ext3 (Linux) partition, which doesn't show up in the Windows 7 Computer Management->Disk Management utility, even as a raw/blank disk. I've verified that there's nothing wrong with the disk by connecting it to my Linux machine and mounting it, and I've verified that the dock is functioning properly by connecting a different FAT32-formatted disk, which mounts flawlessly as expected.I realize that I can't actually read the ext3 partition without additional software (e.g., Ext3IFS), but why doesn't the disk show up at all? Is there some sort of stupid anti-Linux filter built in? Is there any way to force Windows to recognize the disk, so that I can at the very least use direct block access with it?
Background: I want to clone an identical 2 TB disk onto this one. Due to my hardware layout, it's much easier to have the source disk attached to one machine and the destination disk connected to another, and do the clone over the network (the network is not a bottleneck with switched gigabit ethernet), than it is to hook them both up to one machine.(1) I did this once before when both machines were running Linux, but I've since upgraded the destination machine and decided to switch back to Windows for regular desktop use. I've got Cygwin installed, and have verified that the same basic method (dd + nc) will work, but I can't do anything if Windows doesn't even consider the destination disk to exist.I only have one eSATA port on each machine. Opening them up just to do this clone is a rather large annoyance. Also, since this is my backup disk, I'd like to eventually automate the cloning from the active disk to another one that I regularly swap with a third disk that I store off-site.
I just compiled my first own kernel (I'm using Arch Linux), following the tutorial on the german site. Now I tried to boot it, I ended up failing with this message: Code: Waiting 10 seconds for device /dev/sda1 ... Root device '/dev/sda1' doesn't exist, Attempting to create it. ERROR: Unable to determine major/minor number of root device '/dev/sda1' Here is the important part of my menu.lst:
[Code]....
I simply copy&pasted the Arch-entry, i.e. I also had the disk by uuid there. The failure message was the same, just the root device name was the different name Also, at first I did not have the initrd line in my menu.lst (as written in my tutorial that I may not need it). In this case I had this error message:
OS: Debian unstable 32bit, kernel 2.6.32-2, grub 1.98 from late january 2010 (only have working net-access from work now, so I am grabbing information from memory). EXT3 and EXT4 support is compiled into the kernel along with chipset/scsi/sata support (not as modules), and I have tested to boot ext3 with it before proceeding. Prereq: my old disk started to have too much S.M.A.R.T errors, so I bought another one, put in a USB cabinet, added swap and ext4 partition/filesystem to it, and copied over all data from the old system to the new that was mounted at /dest using the command "find ./ -xdev -print0 | cpio -paV0 /dest". Swiched disks, so I now have the ext4 disk sitting at /dev/sda (partitions: sda1 => ext4, sda2 => swap), and booted into rescue-mode from cdrom, using /dev/sda1 as root with a shell on. After doing this, I performed the following commands:
mount --bind /dev /dest/dev chroot /dest
modified the /etc/default/grub to instruct the kernel to boot using ext4, ran grub-install --recheck /dev/sda ran update-grub to modify /boot/grub/grub.cfg (which looks as it should) After doing this, grub finds my partition and mounts it. It however stalls with the message: "warning: unable to open an initial console" and does nothing after this point. I have no ramdisk, but my old kernel booted fine from ext3 (and still does if I copy it to a ext3 partition), and since the ext4 support is compiled into the kernel - should I really need a ramdisk?
