Ubuntu :: Automating Website Data Capture & Spreadsheet Entry?
Feb 3, 2010
I have a spreadsheet which I update regularly. This involves downloading CSV files from a couple of websites, copying data from those files into an OO.org spreadsheet and copying formulas within the spreadsheet (the number of rows of data varies, so when the number of rows of data increases, I need to copy more lines of formulas).
I'd like to automate this process, so that with a single command I can get the data from the websites and feed it into my spreadsheet without having to copy rows of formulas. Although I've done lots of numerically-oriented programming, I have no idea how to undertake this automation task. I don't even know which tools I need to learn to do it.
I have set up a MySQL database which has only three fields for user-name, password and index number. But there is a LOT of data (thousands of columns) which would take too long to enter manually. I'm sure MySQL must have a file format whereby a single large file containing different data types can be successfully automatically imported and the various data elements distributed to their intended fields. Does anyone know which characters MySQL uses as field seperators for this formatting of data, or is there a program already written which can take raw data and organize it into a SQL compatible file?
I've been using Excel from OpenOffice with moderate success for the last year. Now I've just tried to open a new spreadsheet and I'm suddenly getting a highly condensed view. The cells are about 1mm X 4mm. If I expand them by selecting and dragging, then try to copy data from another spreadsheet onto the new one, the data shrinks to the small scale -- unreadable.
I was recently turned on to grooveshark by a friend and I would like to capture a couple of the songs from there. I was told about cplayer by someone but I'm not sure if it does that/ has that feature. Also, I read that cplayer uses some other way of using the hardware on my computer (there's a name for it but I don't remember). I'm worried about putting something on here and having problems with sound with other things on my computer too.
Basically I'd just like to find something I can use to do this though. Something easy to use.
Is there any way to record/capture live streaming video/audio from a website?
Code: www.ww.com I think the file will be .swf something to do with Flash and Macromedia i have tried all kinds of downloader to no available.Thanks for your help....
I have a toshiba phone system that has a feature where it spits raw data and i can direct that data to a serial port on the toshiba phone system or to an IP/PORT the data is just ascii data - tells me calls that are made/time/etc
I would like to setup an existing linux box to capture this data and store the data for me
then later i can build reports based on that data (and for me a report could be just a few cat's/greps/etc so basic stuff) - but i can figure this out later.
for now i would just like to GRAB the data on a port and dump it into a file (and recycle files every 24 hours)
I'm using tcpdump and tcptrace to track all incoming and outgoing data packets through my network interfaces. But I fail to monitor the voip data for skype that way, although it works well with http port 80, for example.
I want to track the ip address of the data packets for skype, i.e. know the ip address of the other one speaking at the other end of skype. How can I achieve this?
I've checked the port setting in my skype and I'm sure I'm listening on the right port. But nothing is showing up while I'm in connection with skype.
I have a weird problem I am experiencing with my Apache 2.2 w/ ISPConfig 3 running on Debian 5.0.4. It is not a website html issue.***PROBLEM***Images are not beind displayed,javascripts are not being run, however when the site is viewed from a local source as a file(like on a USB Flash Drive) they work fine and display properly.Here is the website URL:I have removed a lot of the images and content or changed it in order to troubleshoot it myself, so the site is stripped pretty bare. There is supposed to be a logo inside the "header" <div>, which is set in the CSS file "style.css". There are supposed to be images for navigation, but they show up as red "x"'s. I removed the javascript elements and just trying to work with barebone html/css and can't seem to get it to display properly.
identify the right network structure for a data intensive website, built on LAMP. I'm thinking of a load balanced website, and that it should have a mysql master/slave setup server. I'm no expert in this area, so any online resources are welcome. You can check out the website [URL] - but it will have 10 million items once the hardware can support it.
Im trying to figure out how to display parts of a .db file created by the scorch2000 server to display a player name, games played, score and maybe more...
I dont want to display everything, of course ^^, but how do i get the player name, the number of games he played and his score to display it in a webpage in this fasion:
Name Games Played Score
joe blow1 25 9876890 joe blow2 31 8989767 joe blow2 26 7989767 joe blow2 17 5989767 joe blow2 13 4989767
and by highest score because the log doesin't put them in in score order....
please help, i asked the maker because he has one runing already but no answer back, well the game is pretty old so i didn't really expect an answer anyways and tryed to figure it out but i dont know functions in php, this is to include in a php-nuke block (this i know how to do
here is an example of a working page at the developper website:url
Data flow has become nowadays really huge. Although how hard one can try to tide folders, hard-disk, it can always be some days you are looking for something. What could be interesting to make available, e.g. to install for users, on all Debian stations. In some universities, with Linux-machines network, it is often kind of obliged to use emacs and 'office' KDE applications, as a standard at first.
for organizing better, what could be the solution as 'data flow organizer'? Many solutions, many methods, many ways are offered but none will be perfect
I am trying to set up clonezilla on pxe so that it will start up and show the drones in the workplace a list of images, they choose the one named after the model of computer, and image it. Which means all clonezilla settings but image need to be set and it should automatically choose default settings at 1024x768 when grub pops up. I have looked for command-line options, scripts to run, etc... so far this is what I have...
I don't need the bash for selecting the image, but you might as well add that seeing as there's almost no info online for people who want a pxe server...
I use Evolution Mail Client everyday on my fedora machine. It has become indispensable somewhat. I am wondering if Evolution can be made to run in the background without the GUI, so it can check mails (IMAP/POP) and post delivery reminders through notify-send. I don't mean evolution-notify-alarm ... I know evolution-notify-alarm runs like that but I think that is limited to calender events and tasks (using data mostly stored locally)
I find that I get notified via notify-send interface only when Evolution GUI is on or minimized. Is there a way to get a real background version of this activity - the GUI can then appear when the Panel Icon is clicked
For a certain task, everytime I have to open 10 bash terminals and write different commands on them. Any way to automate this? I am running Fedora linux.
I have a large music directory that I'd like to somehow acquire, or generate, a list of each sub-folder within it, and then somehow get the list into a spreadsheet format. Is there a way to do this?
I'm trying to create a simple times table on OpenOffice.org Spreadsheet
I'm trying to use the "$" in the top tab (input line) i.e.
Quote: =A3*$B2$ Quote: =A3*$B3$
[Code].....
To make the question clearer. When I used to use Microsoft Excel in Windows I would do just as I said in the above and I would be able to construct simple multiplication tables to more bigger/harder to memorize kind of type. And that would be by dragging the rectangle from it's bottom-right corner
Screenshot-5.jpg Screenshot-4.jpg
So my question is what's the alternative OpenOffice.org Spreadsheet symbol for the "$" symbol used in the formula bar in Microsoft Excel?
I have used the words 'Open Challenge'.I have attached a spreadsheet 'Sample file for filter.ods'.TASK: In a presentation, I would like to show what are the Topics of India on Day-1 , Day-2 , Day-3 , Day-4. That means I am going to Filter India and (Day-1, Day-2, Day-3, Day-4).
I had a really important spreadsheet with my student's grades in it. I wanted to rename it so I right clicked and renamed (at least that's what I thought I did). After I looked again, the file somehow became a zip folder. I inside the zip folder are a bunch of files but I can't the spreadsheet anywhere! Here are some screens:
here is the thing. I've deployed BackupPC in a server at work, and everything is working fine. Now, what I need to do this thing. I have a remote server with a website and a postgres db running. I've been able to set up everything to be backed up using rsync. But I would like to make the same process to restore the backup immediately in a local server trough rsync, that has to be ready just in case of failure of the remote server. What I've tried to do is to run the DumpPostUserCmd, so whenever a dump is performed in the remote server, I can have my local server updated. What I've find out is that when you perform a restore through the web interface, the command that performs it is (using ps ax in the backuppc server):
So, what I could find out is: /usr/bin/rsync is the command that the process runs in the server where I want to put the restore. Then the options, and finally, source and destination of the restoration. But, as you can see, the source is '.' but I cannot guess where to point that!
Basically, I go through certain logs on my linux hosts checking for anything out of place. You know, the usual, /var/log/messages, lastlog, etc. What I've been using is just a simplistic grep statement that looks for keywords in certain log files, such as warn, crit, etc., with the -i flag to ensure no case searches. I thought, well, surely there are others out there doing the same thing, but more automated. Perhaps something that can check files for keywords and then notify, via email, if anything out of the ordinary is seen.
I need to ask a question on many forums/mailing lists.In such a situation I need to open browser and login to many forums at the same time.Then go and format it according to forum guidelines and then paste it.So I want to know if I can automate this process some way by some alias or command line utility and just type the question in a text file and give URLs on command line and the question gets posted to those forums where I request?
I am volunteering to assist a non-profit org with their technology, I will be presenting Ubuntu to the teachers this week and I hope to have a positive response to using Ubuntu!ith that said, the org being a bilingual school we will most likely focus on using Edubuntu. While I personally use Ubuntu and have installed it over the years on multiple laptops & deskptops, I am now facing the task of installing and maintaining on a dozen laptops and another dozen deskptops!While the install is very straight forward, installing on so many systems will be very time consuming! In addition, we would like to install additional apps such as Skype, Dropbox, French Language support, Possibly Ailurus and a few other apps.
I am looking for suggestions how to best handle this? What would be the best method to automate the install across the systems?Could the best approach be to install Edubuntu and all the apps we require on one system and then create an image from the "master system" to be used for the other systems?It seems that creating a master image would be ideal as it would include everything we need but I have never done that before
I'm having some trouble with hyperlinks in openoffice 3.2 spreadsheet, though it may be a begginers problem.
I wrote a script to execute a certain task (in my case, to open an DNA alignment file with bioedit) and then, in openoffice, I assigned a hyperlink button to the path needed to execute the script, which is in my /home ("./script_file"). When I do this the first time, clicking the hyperlink button runs the script perfectly. However, whenever I close openoffice and re-open it afterwards, the hyperlink doesn't work anymore. I've checked and this happens because the original URL, which was "./script_file" changes to /home/user/etc... or /temp/etc... and, obviously the file isn't found. If I manually correct the URL, the hyperlink is functional again... how to fix the URL of an hyperlink so that I won't have to correct it every time I close and re-open the program?
Is there any way to make an OpenOffice spread sheet open to the last cell that was edited? I have a friend that has a very large spreadsheet, and would like to have it open to where he left off, rather than cell A1. He says excel does, or can be made to do this.
I know this is more of the Openoffice question but I felt I would find some solution here for this. At my work I am using lot of PDF files which are created using Openoffice Calc (Spreadsheet). I want to convert those PDFs to Calc file again for editing. Currently I am using a windows Machine with able2extract pdf converter for converting PDFs to .xls file. But it is not ideal solution for Linux and Openoffice users here.
I'm in an Aussie rules fantasy football competition and someone posted a zip file with relevant stats for players in the under 18 comp, as a guide for selecting possible players in next years team. Never used any of the office applications. I've attached the zip file and this is what the poster said...
In the actual spreadsheet though, there are two sheets. The first is all the raw data which you can order by position (based on me watching the games, and the line-ups listed on the AFL websites), K: H ratio, state etc. The second is a sheet which you can use to compare players.
I find that sometimes it is easier for me to make decisions about which players to pick when the stats are more obvious, so if you are the same, you can use this sheet as well. Anyway, I hope you guys enjoy the spreadsheet and I'll update it after each round. Fixed, I had to change the file from .zip to .xlsx
