Programming :: Synchronization Of Two Processes With Share Memory?
Feb 27, 2011
I have two processes that share a piece of memory, and i want to use the shared memory to send data from one process to the other. it's like a simple consumer-producer problem. when the producer fills the shared memory, it waits until the consumer can consume some data in the memory; the consumer needs to wait if there is no data in the memory. The thing gets complicated when both threads are allowed to sleep and wait for the other to wake it up.
i wanted to use condition variable of pthread for synchronization, but it doesn't work in multiple processes. i tried semaphore, but it's quite complicated and i still cannot make it right. I believe it's a common problem and someone should have written similar code before, or maybe the code is even wrapped in a library, but when I search for it on Internet, I only found information about how to share memory between processes. Does anyone know where I can find this kind of code or library?
I work on Linux for ARM processor for cable modem. There is a tool that I have written (as the job demands) that sends/storms customized UDP packets using raw sockets. I form the packet from scratch so that we have the flexibility to play with different options. This tool is mainly for stress testing routers.
The details are here.
I actually have multiple interfaces created. Each interface will obtain IP addresses using DHCP. This is done in order to make the modem behave as virtual customer premises equipment (vcpe).
When the system comes up, I start those processes that are asked to. Every process that I start will continuously send packets. So process 0 will send packets using interface 0 and so on. Each of these processes that send packets would allow configuration (change in UDP parameters and other options at run time). Thats the reason I decide to have separate processes.
I start these processes using fork and excec from the provisioning processes of the modem.
The problem now is that each process takes up a lot of memory. Starting just 3 such processes, causes the system to crash and reboot.
I have tried the following:- 1-I have always assumed that pushing more code to the Shared Libraries will help. So when I tried moving many functions into shared library and keeping minimum code in the processes, it made no difference to my surprise.
2-I also removed all arrays and made them use the heap. However it made no difference. This maybe because the processes runs continuously and it makes no difference if it is stack or heap?
3-I suspect the process from I where I call the fork is huge and that is the reason for the processes that I make result being huge. I am not sure how else I could go about. say process A is huge -> I start process B by forking and excec. B inherits A's memory area. So now I do this -> A starts C which inturn starts B will also not help as C still inherits A?. I used vfork as an alternative which did not help either. I do wonder why.
reduce the memory used by each independent child processes.
I found from command 'top' that 8GB memory are used. However, using command 'ps' with some options to grep the running processes and then summing up the memory used by the running processes are less than 2 GB. Where has the used memory gone ?
The following code is for monitoring the memory used by apache processes. But I got a problem that the data I got by this script is much larger than the physical memory. It was said that there are some libraries used simultaneously by many processes, so the data I got has some double counted part. Because apache has many httpd processes.
Anyone have an idea of getting the multi-processes memory used?
I recently built a new server, running 64 bit Slackware 13.0 with the following specs:
MSI 785GTM-E45 AMD Phenom II X2 550 2GB DDR2 Onboard video from AMD 785G chipset 2x 80GB IDE system drives using software RAID with 2GB swap partition
I only include these because I'm not convinced my problem is not hardware related at some level. Basically, when I first start up the system, the memory usage is anywhere from 60 to 200MB. Then it starts to gradually climb until there is only 12-15MB free. This can take anywhere from a few hours to a few days.
The only thing I really use this for is to serve Samba shares and the occasional SSH login. I've set up Samba shares to be accessed by my Windows machines as well as a Mac. Initially I just explored the network if I wanted to see the share from these machines, but that was too slow and unreliable (the server would not always show up in Windows), so now I automatically mount the share as a network drive at startup (from Windows). So I don't know if this would have anything to do with the steadily increasing memory usage. These systems are not on/connected all the time, but the memory usage seems to rise anyway.
When I run top, it reports that nearly all of the physical memory has been consumed (after a while of uptime), but none of the swap space has even been touched. This is a typical output of the first several lines, sorted by swap size:
I can't be sure, but I believe the message just repeated itself before this part. MS-7549 is one of the components of the mainboard (MSI), maybe the northbridge or something (it seems to be associated with more than one model), so I thought for sure this meant a hardware problem, maybe with the storage controller or memory. But Memtest86 passes for more than a day without any errors, and I loaded Windows on one of the disks and ran it as hard as I could with prime95 and nothing so much as flinched. So I've got to think this is some kind of issue with linux and/or how it is handling my hardware configuration.
I am doing a test to get the memory used by apache`s apache2 processes. I used a script to get VmSize and VmRss from /proc/pid/status, and loop through that to get the sum of VmSize and VmRss of all the apache2 processes.
I found the VmSize (about 4GB) and VmRss (about 3.4GB) are much larger than the physical memory (1GB) when apache server was saturated. It was said because of the multi-counted libiraries size used by many processes simultaneously. Then , how to get the physical memory used by apache2 processes? Or how to get a more reasonable memory data?
I work on Linux for ARM processor for cable modem. There is a tool that I have written (as the job demands) that sends/storms customized UDP packets using raw sockets. I form the packet from scratch so that we have the flexibility to play with different options. This tool is mainly for stress testing routers.I actually have multiple interfaces created. Each interface will obtain IP addresses using DHCP. This is done in order to make the modem behave as virtual customer premises equipment (vcpe).
When the system comes up, I start those processes that are asked to. Every process that I start will continuously send packets. So process 0 will send packets using interface 0 and so on. Each of these processes that send packets would allow configuration (change in UDP parameters and other options at run time). Thats the reason I decide to have separate processes. I start these processes using fork and excec from the provisioning processes of the modem. The problem now is that each process takes up a lot of memory. Starting just 3 such processes, causes the system to crash and reboot.
I have tried the following:-
1-I have always assumed that pushing more code to the Shared Libraries will. when I tried moving many functions into shared library and keeping minimum code in the processes, it made no difference to my surprise.
2-I also removed all arrays and made them use the heap. However it made no difference. This maybe because the processes runs continuously and it makes no difference if it is stack or heap?
3-I suspect the process from I where I call the fork is huge and that is the reason for the processes that I make result being huge. I am not sure how else I could go about. say process A is huge -> I start process B by forking and excec. B inherits A's memory area. So now I do this -> A starts C which inturn starts B will also not help as C still inherits A?. I used vfork as an alternative which did not help either. I do wonder why.
I'm running several SHOUTcast server instances and a WowzaMediaServer instance on a CentOS machine. I'm experiencing a memory leak problem, but I can't figure out which processes are eating memory.
TOP command reports as follows:
[Code]...
Something misterious to me (I'm still a Linux newbie) is that TOP reports a total of 7.5GB used ram but very small percentage for single process (0-1%). Memory consumption starts at 1GB/8GB after reboot and in three days running gradually increases up to 8GB. I'm practising with Linux, but I still miss a lot to understand what's happening on my system. For instance, are there linux kernel logs saved somewhere that I can look at?
I am working with a C++ program consisting of two threads. The first threads receives packets through an UDP unicast connection and stores them in a buffer. The second thread reads the packets from the buffer and sends them through an UDP multicast connection. Both use blocking sockets and share a common buffer and a linked list L1, which are protected by mutexes. The program seemed to work just fine, receiving a packet and sending it almost immediately, but started giving some trouble recently. The synchronization between both thread started failing, and I decided to use a non-blocking socket in the sending thread. As a consequence, sendto() doesn't work in some cases, causing an errno 11 (Resource unavailable).
I know of /etc/security/limits.conf and that can be used to limit all sorts of good things, but I haven't found anything that talks about using this when the users come from LDAP. Would I be able to do something like
@"Domain Users" soft nproc 25 @"Domain Users" hard nproc 40
where Domain Users is the group all users belong to in our system.
I am facing an issue where the process starts hanging. When I closely look at the logs I come to know that some of the child processes that are forked by the parent process are not finished.
1) Is it possible that the child processes that are not finished occupy the socket memory of the parent process and ultimately a point is reached where no socket memory is available to fork new child processes.
2) What is the standard limit of socket memory in linux?
3) What is the fate of such child processes (as I have mentioned above)?
4) How to debug such cases so that the exact problematic area is identified?
I wrote a program that multiplies 2 matrices using multi-threads and another one using multiple processes and shared memory. Both in C.I need to find the total memory usage of these programs. I know of the top command, but when my matrices are relatively small they don't even show up on top because they complete so fast, how can I find the memory usage for these instances?Also, how can I find the total turnaround time of my programs
I am new to C and linux. My code below does arbitary writes but I cant figure out where or how it does it.
I am calling the insertNode() function with seq = 'MISSISSPPI$' and alphabets = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ$'
Code:
Weird behaviour I should mention is that when I check for NULL pointer in node->child[index], the unassigned values are not null anymore, they point to arbitary memory.
Is that possible that SHM shared memory is counted as cache memory on Linux with kernel 2.6.18?If find it really odd since this memory is not file backed, but I have a piece of code that loads data using shm_open+mmap, and it generates an amount of cache memory in /proc/meminfo that corresponds exactly to the amount of shared memory (I load that data from a file but I am using posix_fadvise(fd,0,0,POSIX_FADV_DONTNEED) to ensure this file is not cached and I made sure that it is working as expected). As far as I know SHM memory was not tagged as cache memory with kernel 2.6.9.If it is the case it is really unfortunate since normally cache memory can be considered to be part of the "available" memory since it can be flushed promptly but this is clearly not the case with SHM memory... Is there an easy way to get the total amount of used SHM memory on a system?
I am doing real time programming in C++, under Linux. I have two processes, let me say A and B. A process is being started periodically, every 5ms. B process is being started every 10ms. The process A is doing data processing. The process B is reading that data and displays it. I am confused about how to run periodically processes. The problem is that the period of process A should be as much as it is possible accurate (5ms). For the process B it isn't so important. I have created independent processes, each in one .cpp file, and I am starting them from bash file. Is that OK? I don't have to make child processes in order to have parallel processes?
how can I measure time of N processes and N threads and then compare this time to prove that threads are faster than processes. understanding C code, or also for some good way to measure time of N processes and N threads for C.
I have 2 completely different processes A and B (they do not have any relationship) suppose A opens a file with file descriptor 4 and B opens another file with file descriptor 5
Can process A use the fd 5 (i.e using the file object of processs B) by any means..if suppose i let the fd 4 of process A to point to the file object of process B(fd 5) will it work? is it safe?
I need a way to kill off the pids 2819, 2820 because they do not have a process tied to them like pids 2918, 2922 and 6657. The way it works is peek shell (pid 2918)is opened then it starts a ksh (pid 2922) session then from there the end user runs a command (pid 6657).
the wrong part of the forum but basically im working on a project and getting no where what so ever and i was wondering if i could get your help. Basically i have to create two scripts do:
The parent script which is going to: o spawn several child processes. o keep track of the progress of the child processes.
I am building a queue system (in C on CentOS 5) where each directory queue has a number of process instances 'watching' it. I have a root-permission queue supervisor process which does all the forking, and the instance processes give up root as their first action. For max security each queue is accessible to a different set of users and groups.
I need to keep the RAM requirements as low as possible, so I am relying heavily on copy-on-write (since all memory writes are to shmget() shared memory)
Each process presently dlopen's the libraries it requires after fork, but I'm wondering if I would gain by performing the dlopen before forking a number of instances which all require the same libraries.
Any pointers to mid-level tutorials on dlopen and fork internals would be most appreciated: this is about my third post anywhere since I converted to Linux, but I just can't find the answer to this anywhere online.
If a process forks its child and communicate with the child using pipe, do closing the write end of the pipe and terminating the writing process have the same effect?
The ipcs command shows the facilities of IPC in Linux, but it does not shows the processes that communicating to each others. How can I list the processes, which is currently communicating in IPC ?
which in theory should pull *only* the PID of "mono user1.exe" and kill only that. The problem: It kills any and every single instance of mono that is running on my system, every userx.exe thats open. I am confused, as a simple "ps aux | grep 'mono user1.exe'" does only return the mono user1.exe process and not the others. "ps aux | grep 'mono'" returns them all though. how I can modify that script so that it only kills the specific process? Would "pkill -9 -f 'mono MCuser1.exe'" work as well - or would it too kill every instance of mono? I cant do a lot more of trial and error, its not good I am killing those instances accidently...
