General :: Share A Large Number Of Files Into Chroot Env?

Aug 17, 2010

I understand that chroot is usually used to provide security, however, for my issue, security is a big don't care. I am very new to using chroot and don't fully understand how the chroot'd env works.

problem: Trying to use a vendor supplied cross compile environment. The environment runs as a chroot'd env and works just fine. I have a large number of additional modules that I wish to compile in the chroot'd environment. FYI, these modules are also (succesfully) compiled for other targets not using chroot'd env's. Copying the source files into the the chroot environment is not an option (don't have hours to wait for copies to finish and it would break the make system). Having them live in the environment is also not an option (the chroot build is a tiny part of the build process and we cannot revamp our entire source tree to accommodate it).

I am looking for a way to have the compiler in the chroot'd env have access to a path that is outside of the env and typically higher up in the same path that holds the chroot'd env. I have tried soft links (they don't work as expected). Hard links only work for single files and there are 10's of thousands of files that would need to be linked. I am not sure how I would go about exporting the additional files and then mounting the exported files in the chroot'd env (or if that would even work).

View 2 Replies


ADVERTISEMENT

General :: Tar Not Working With Large Number Of Files?

Dec 6, 2010

in my middle of script i am using tar command to tar some 1000 images and size of each image is 5MB.

all the images are provided as argument as tar -cvf file.tar <all images as argument>

but my tar file file.tar does not contain all the images.

View 4 Replies View Related

General :: Copying Large Number Of Files From One Directory To Another

Feb 10, 2010

I've a directory containing around 2.8 lacs of files. I want to move them to another directory.If I use cp or mv then I get an error 'argument list too long'. If I write a script like

for file in ls *; do
cp {source} to {destination}
done

then because of ls command , its performance degrades.How can I do this?

View 7 Replies View Related

General :: Copying Large Number Of Files In Windows?

Mar 15, 2011

I am facing problem in copying a large number of file 18 lakh (18,000,000) files from my personal hardisk to another hardisk each file is very small and size of folder is around 3.95 GB copying files using copy given by Windows is frustrating and I am not even able to compress file its giving me error that its not readable.And problem is I am not able to open this drive in Linux it showing me error there saying do diskchk in Windows and Windows disk check is also not able to repair this drive and goes into some mode unsolvable.Is there any way to open disk with error to open in Windows and if not any way I can copy data faster?ERROR: Disk labled EDU is corrupt go to windows and chkdsk /f there and reboot into window 2 times.

View 3 Replies View Related

Ubuntu :: Command With The -r Option To Compare A Large Number Of Files And Files In Subdirectories

Jun 16, 2011

I am using the diff command with the -r option, to compare a large number of files and files in subdirectories. My main interest is to find out which files have been changed, and not what the actual changes are, and since a lot of files has been changed, it would be a lot easier to view the file names only. Is there and option for diff that might do this, or does there exist a similar tool/command that could do the job?

View 1 Replies View Related

General :: Transfer Large Number Of Files Host To Host

Oct 20, 2010

I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.

I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.

View 3 Replies View Related

Ubuntu :: Data Loss When Transferring Large Number Of Files?

Jul 20, 2010

This problem is not exclusive to Ubuntu, I've experienced it in Windows and OSX as well, but it seems that almost every time I transfer a large number of files (i.e. my music collection) between my desktop computer and laptop via my external hard drive, I end up losing files for no reason. I usually don't notice the files are missing until later on, because I am never informed of any data loss. Now, every time I make a large transfer of files, I just do it two or three times to ensure that I don't lose any files.

View 2 Replies View Related

Ubuntu :: Find And Move Commands On Large Number Of Files ?

Feb 21, 2011

We recovered a large number of files from a HD I messed up. I am attempting to move large numbers of files of a type e.g. .txt .jpg , into a folder by type to more easily sort through them.

Here are the commands I have mainly been trying with various edits:

Code:

Code:

So far the most common complaint I have gotten "missing arguments to execdir".

This is on Ubuntu 10.04

View 6 Replies View Related

Applications :: Extract The Sender Id From A Fairly Large Number Of Files?

Nov 18, 2010

I'm trying to extract the sender id from a fairly large number of files and am having trouble assigning variables from a file. Here is what I have so far, (which is fairly kludgy I know, but it's been some years since I've done any scripting or programming, and I find that I have lost the knack to a large degree).

[Code]...

View 1 Replies View Related

Ubuntu :: Large Number Of Files To Count - Argument List Too Long

May 28, 2011

I have the standard problem of trying to count a large number of files in a directory (>100k)

I have tried: ls ~/user/images/* -l | wc -l and find ~/user/images/* -maxdepth 1 -type f | wc -l

In both cases, I get the argument list too long error message.

I have tried using xargs but I can't seem to get it to work right.

The command

returns a valid answer but it includes all the subdirectories in the file count.

View 4 Replies View Related

Programming :: BASH Script Optimization For Testing Large Number Of Files

Sep 18, 2010

I want to move files from a $SOURCEDIR to a $DESTBASE/$DESTDIR. Under $DESTBASE there are many directories, and I need to test beforehand if a file from $SOURCEDIR already exists in any of them.

This is obviously extremely slow, and the real use case involves dozens of dirs and thousands of files. Creating a temporary "index" file for the find command (instead of running it every iteration) speeds it up a little, but it's still very clumsy.

View 14 Replies View Related

General :: Using Sed To Replace A *large* Number Of Variables In A File?

Jul 28, 2011

I have a large number of log files, on a linux box, I need to cleanse sensitive data from before sending to a third party. I have used the below script on previous occasions to perform this task, and it has worked brilliantly (script was built with some help from here :-)

#!/bin/bash
help_text () {
cat <<EOF
Usage: $0 [log_directory] [client_name(s)]
EOF

[Code]...

However, now one of our departments has sent me a CLIENT_FILE.txt with 425000+ variables! I think I may have hit some internal limit. I have tried splitting the client file into 4 with around 100000 variables in each, this still doesn't work. I'm loathe to keep splitting though as I have 20 directories with up to 190 files in each directory to run through. The more client files I make, the more passes I have to do.

View 2 Replies View Related

General :: Convert A Large Number Of File Types From None Standard To Text?

Sep 28, 2009

I have on my windows machine several hundred files that are a format of .nc .ncs for a CNC machine. I need to convert them to txt which is something as easy as opening in notepad and then saving as .txt but there are so many that this kind of action would take way too long.

The reason I am writing the linuxquestions is because I would feel more comfortable in loading a live CD and using some sort of terminal command to do this than I would to download one of the many "freeware" type programs I have found for windows (even more so since I have had a root kit before and had to start all the way over to get rid of it).

I need to know:

1. Is this possible to do with the terminal without super advanced knowledge.

2. Can one please point me in the right direction; something to read or an example

View 2 Replies View Related

Software :: Filter A Large Document By Line Number?

Feb 17, 2010

I have a 50000 line(ish) set of records in a file. I have another file where I have filtered out all the line numbers for those which have an error of various types. e.g column count, field type etc. I want to get all those lines into a separate file so I can sanitise them. There are abt 3-4000 of them.

How can I access those lines which I want to isolate into a single file? I have all the usual linux stuff available and a bit of understanding of regexps.

View 5 Replies View Related

CentOS 5 Networking :: Configuring For Large Number Of Tcp Connections?

Jan 24, 2011

i've got a select based application that wants to support a large number of mostly idle connections. the code is java and works on windows, suse enterprise linux, mac os x. it does not work on centos 5.5 (32-bit, 2.6.18 kernel, 1G of memory).

i've read and followed the directions in various articles about tuning linux for large numbers of connections (including the C10K problem), and gotten the number of sockets up to 3200.

these didn't make any apparent difference:

[URL]

on windows, i can get up to around 78,000.

on suse enterprise linux (a few years ago), i got up to 90,000. that's where i got bored and stopped.

on my mac laptop with os x (snow leopard), i got up to 10,500.

i have used ulimit -n 10240

my current goal is 10k sockets.

the test is that i'm opening one socket at a time until it fails. when it fails, many of the sockets which have already been opened also fail, in one giant cascade. sounds like a buffer / memory problem.

each group of 64 sockets gets a thread to manage select calls for them. thus i'm only using around 61 threads total when it fails.

View 3 Replies View Related

Ubuntu :: Can't Find Any Process That Consume Such Large Number Of Memory

Feb 23, 2010

I used 9.04 for months and it work fine before restarting my PC. After I restarted my PC, the memory consumption takes up to 4.2 GB after login. However, I cannot find any process that consume such large number of memory.

[code]....

View 3 Replies View Related

Ubuntu :: Adding Additional Users Causes Large Number Of Errors?

Jun 29, 2010

I'm running 64-bit 10.04, upgraded from 9.10. The problem I am experiencing is that any user accounts aside from my main account are problematic. This includes any accounts I add, as well as the GDM guest session.The specific problems that I have thus far experienced are as follows:

1. The desktop loads often improperly. In the latest instance of this the graphics on the right side top panel were randomly chopped-up, leaving parts of my clock on either side of the volume control, among other things. 2. If I make ANY customizations to the desktop at all, the desktop takes nearly a full minute to load on log-in. 3. Flash videos don't work properly on Firefox. Sometimes they only play after refreshing a page, often they will not load at all. Also, attempting to load or play a flash video will sometimes causes Gnome or Firefox to crash. 4. (And this is the one that REALLY has me stumped) Whenever I log into my main accountant after logging out of another account, the IBus control appears in my system tray.However, when I open the IBus preferences the associated check box is (and has always remained) unchecked.Not sure where to go with this one. More than anything, the IBus bug makes me unsure of where to even begin looking for the problem.

View 1 Replies View Related

CentOS 5 :: 5.2 Installation Hangs On Server With Large Number Of Drives?

Apr 6, 2009

I am attempting to upgrade a system from 4.7 to 5.2 using a (now) DVD drive attached to the onboard IDE. Originally I had tried using a remote NFS image and a USB stick but I thought maybe there was a problem with the image. I can get up to the point of the installation of selecting the keyboard for the system and then it freezes and never goes any further. It doesn't appear to be a kernel panic since I can still switch between consoles.

I've got an MSI K9NGM2-FID with 14 drives in it. It serves as a file server for our backup server. It's got a secondary 4 port Silicon Image SII 3114 SATA card using the sata_sil module, and an old IDE Promise FastTrak TX2000. Technically I could have 16 drives but the 750W PS is walking the fine line on tripping it's self-breaker with the 14 drives and 7 fans. I would like to NOT have to disconnect all of this to do the upgrade.

I thought maybe that running the install using the "noprobe" option would help so it didn't detect and load the modules for the Silicon Image or the Promise cards and detect all of the drives but it still gets stuck on the step after selecting the keyboard. The installation info console and the dmesg console don't really provide any useful information. The installation console says:

INFO : moving (1) to step welcome
INFO : moving (1) to step language
INFO : moving (1) to step keyboard
INFO : moving (1) to step findrootparts

And the last lines of the dmesg console says:

<6>device-mapper: multipath: version 1.0.5 loaded
<6>device-mapper: multipath round-robin: version 1.0.0 loaded
<6>device-mapper: multipath emc: version 0.0.3 loaded

Is there a hidden "debug" option that will turn on a lot of extra logging?

View 7 Replies View Related

General :: Downloading Very Large Files Via SFTP

Apr 1, 2011

I need to download some very large files (circa 75 GB) from a remote server via SFTP. I've been using SFTP via the command line on my Linux netbook. Around halfway through, the transfer stops and says "stalled." Can anybody recommend a reliable way to download these files?

View 1 Replies View Related

General :: Creating Random Large Files?

Aug 27, 2010

how I can randomly write / create a 1 GB file in bash to test disk / network i/o? I was told I could use the 'dd' command but I don't know if there are some better ways and or what the 'dd' command looks like.

View 7 Replies View Related

General :: Transferring Large Files Using Scp With CPU And Memory Considerations?

Oct 5, 2010

I want to transfer an arbitrarily large file (say >20GB) between 2 servers. I have several considerations:

Must use port 22 (ssh) because of firewall restrictions
Cannot tax the CPU (production server)
Memory efficiency
Would prefer a checksum check but that could be done manually
Time is not of the essence

Server A and Server B are on the same private network (sharing a switch) and data security is not a concern, Server A and Server B are not on the same network and transfer will be via the public internet so data security is a concern, My first thought was using nice on an scp command with a non-CPU-intensive cypher (blowfish?). But I thought I'll refer to the SU community for recommendations.

View 1 Replies View Related

General :: Searching Text Files For Large Numbers?

Dec 31, 2010

I am looking for a way to search for large numbers in text files and print the nearby lines.

For example if I had a text file like:

Event: 11
blah: 3
blah: 41 bleh: 19
Event: 2
blah: 31

[Code].....

View 1 Replies View Related

General :: Uploading And Downloading Large Files Between Clients?

Jun 5, 2011

I am looking for a file sharing program to install on my dedicated server that will allow me to upload large MP3 files and allow my clients to download them. these files are recordings of counseling sessions for families who are seeking help for their children.

What I am looking for is similar to the system this company uses [URL].

View 4 Replies View Related

General :: Nas - Most Effective Backup Software -> When Dealing With Large Numbers Of Files?

Jul 18, 2010

I have two NASes. I work off of one, and the other is used as a backup. As I have it set up now, it's slow. Running a backup takes a week. Even for 7 TB, with 1,979,407 files, this seems a bit outlandish,particularly as both systems are RAID-5 and the network is all gigabit. I've been digging about in the rsync man pages, and I really don't understand what differentiates the various topologies.Right now, all the processing is being done on the backup NAS, which has the main volume from the main NAS mounted locally over SMB. I suspect that the SMB overhead is killing me, particularly when dealing with lots of files.

I think what I need is to set up rsync on the main nas as a daemon, and then run a local rsync client to connect to it, which would hopefully allow me to completely avoid the whole SMB-in-the-middle affair, but aside from mentioning that it's there, I can find very little information on why one would want to use the daemon mode for rsync.

Here's my current rsync command line: rsync -r -progress --delete /cifs/Thecus/ /mnt/Storage/input? Is there a better way/tool to do this? Edit:Ok, to address the additional questions: The "Main" NAS is a Thecus N7700. I have additional modules installed that give me SSH, and it has rsync, but it's not in the $PATH, and I havn't figured out how to edit the local $PATH in a way that persists between reboots. The "Backup" NAS is a DIY affair, built around a 1.6Ghz Via Mobo with a Adaptec Hardware RAID card. It's running CentOS 5 with a full desktop environment. It's the hardware I'm running rsync from. (Gigabit is through a additional PCI card).

Further Edit: Ok, got rsync over SSH working (thanks, lajuette!).I had to do a bit of tweaking on my command line, I'm running rsync with the args:rsync -rum --inplace --progress --delete --rsync-path=/opt/bin/rsync sys@10.1.1.10:/raid/data/Storage /mnt/Storage (Note: I'm specifically not using -a, because I want to change the ownership to the local account, to not freak-out SELinux)

View 5 Replies View Related

General :: Cp Adds Exclamation Points When Copying Very Large Text Files?

Jul 13, 2009

For my research I have some very large files that are basically millions of lines of ten columns of numbers. These files can be up to 5 GB in size. Recently I noticed that when I made a copy of one of my files, some exclamation points appeared in it where there should not be any: in front of random numbers throughout the file. Making another copy of the file would result in exclamation points in front of different numbers in different parts of the file. Doing this many times has given me up to four exclamation points in different parts of the file. Sometimes the file copies just fine without producing any extraneous exclamation points.Additionally, I have occasionally seen a "^K" where there should be a newline (the data that should have been on the next line was instead on the previous line with a ^K in front of it) in copies that I have made of my files. I don't know if this is related or not.

View 7 Replies View Related

Hardware :: Mem - 'input Output Error On Sr0 Logical Block (a Large Number) After A Long Time The Booting Proceeds To A Point

Aug 29, 2010

I have been having a lot of trouble lately with installing from CD/DVD. The DVD reader/writer on this laptop is new. Nevertheless, trying to install Ubuntu onto an exernal HD, I get 'input output error on sr0 logical block (a large number) After a long time the booting proceeds to a point, but I never get the actual installation started, and have to shut down manually.

The CD is fine, says the Ubuntu-checker. I just installed using my sons laptop, and there was no trouble. Question: does this indicate a motherboard failure? A memory block damaged? Do you know of a diagnostic tool I can use to check the reading of a CD/DVD?

View 7 Replies View Related

Ubuntu Servers :: Completely Chroot Enviornoment Where Every Single Service Is In Chroot Mode (bind,mysql, Postfix)?

Jul 12, 2011

recently we decided to make our own panel (like Plesk or cPanel) but for Ubuntu and it will be licenced under GPL (like any other professional sofware).want to make a panel not only that fits our needs but also the needs of other system administrators and domain owners. We researched other panels and found out that non of them has security/look/ease of use in one package. Bad codig is another problem found in other panels.I made a short overwiev of what I think we have to have in the beginning.I Security :1. Completely chroot enviornoment where every single service is in chroot mode (bind,mysql, postfix, .... )2. Easily managed IPtables trough web-based interface. 3. Coding rules has to be strict.

II Software selection :
1. MTA - Postfix
2. POP - dovecot

[code]....

View 7 Replies View Related

General :: Maximum Number Of Files Per Folder?

Apr 28, 2011

I would like to ask you if there is any maximum allowed number of files per folder in linux (without risking it to lose everything). I am using openuse 11.4 with latest kde (4.6?).

I am trying something fast and dirty and it might be that one folder will contain like 10^6 files.

Is there is anything I should be warned about that?

View 10 Replies View Related

General :: Find Number Of Files In A Directory?

Feb 22, 2010

i need to know how to find number of files in a directory? is there any system calls in fedora 12.And i need to know how to perform a operation if the that count increases by one?

View 14 Replies View Related

General :: Merging Files With Different Number Of Rows Using Awk?

Apr 18, 2011

Does anyone have a solution for merging files if the number of rows in the two (or more) files is non-equivalent.To exemplify, how about merging the following 3 files:

file1:
1
2

[code]...

View 8 Replies View Related







Copyrights 2005-15 www.BigResource.com, All rights reserved