Software :: Automating Access To Interactive Sites Which Use HTML And Javascript
Mar 5, 2011
Is there a modern equivalent of "Expect", which was developed for the UNIX environment to automate access to programs which had been designed to interact only with a human?
Now with many facilities only accessible by HTML and javascript, what solutions are there to access such websites from a Perl script for example?
The need to access interactive systems by automated agents is the same now as when Don Libes originally wrote Expect, described well in his paper to the Summer 1990 Usenix Conference, Expect: Curing Those Uncontrollable Fits of Interaction.
I want my computer to access the web - I don't believe I should have to be chained to it by mouse and keyboard before it can interact with web sites. Surely there are some tools for achieving this. I've searched, but haven't been able to find any. Although I'd like to do it from a Perl script, I'd consider any language: C, PHP, anything.
How can I verify JavaScript on a site (not on the same server) is working from command line access of the URL?
On checking some of what was written in 2005 I find most of it isn't still relevant (if it was then - I noted the w3m-js stuff mentioned in 2005 hadn't been updated since 2003).
Since then a couple of times I've looked into it but didn't make much headway but also didn't spend a lot of time on it. So far as I could tell from other posts I'd seen others hadn't either.
Recently I found out about Mozilla's SpiderMoneky js library and that it could be linked into elinks text browser and did same on my RHEL5 system.
I found Spidermonkey rpms in the EPEL (Fedora sponsored) repository: js-1.70-8.el5.i386.rpm js-1.70-8.el5.x86_64.rpm js-devel-1.70-8.el5.i386.rpm js-devel-1.70-8.el5.x86_64.rpm
I was able to download and install using rpm then compile elinks from source to include the above. Running elinks version after that shows the ECMAScript support is compiled into the new elinks binary. Also setting various ecmas* flags in the elinks.conf does have effect so clearly it is using them. (As opposed to another system where I use the original RHEL5 provided elinks RPM it complains about the ecmas* stuff being invalid options.)
It appears that by default it has this enabled but just for good measure I added:
set ecmascript.enable = 1.
Despite that when I pull up the page I see:
This product requires use of a browser that supports JavaScript
As a test I set the above to 0 on the off chance it was backwards and saw the same thing.
I then set the following:
set ecmascript.error_reporting = 1
On pulling up the page I now see a popup box:
Quote:
JavaScript Error:
A script embedded in the current document raised the
following exception:
TypeError: Window.Focus is not a function
OK
That seems to suggest that despite the earlier message about needing a browser that supports JavaScript that it is in fact executing at least one JS but that it is getting an error. Can anyone confirm that?
Does anyone have any idea what I�d need to do to deal with the Window.Focus message?
Essentially the site is a login page and even though I am able to input username and password after opening in browser when I tell it to POST it simply returns to the same login/password page.
And of course it doesn't have to be elinks/spidermonkey. I'd be interested in finding out if anyone knows a way I can verify a site that is running JavaScript is actually responding properly from command line rather than via a browser?
The SpiderMonkey stuff from Mozilla when compiled into elinks is supposed to do that but given my results Im not sure if it is or not.
Most of the documentation I can find on JavaScript stuff appears to be aimed at developers testing their own JS code from command line rather than accessing a web page with JavaScript.
I'm looking for an existing linux distro with a pure HTML(5) client interface. Sort of like m0n0wall, but then featuring all functionality a modern OS includes, implemented in HTML/Javascript/CSS. Kind of a Skylight clone, but then using only free and OSS software.
I am trying to set up a webpager that contains a background of the skylouds uperimposedin the center will be an image of an airplane with a transparent background.I want the sky to scroll continuously to the left behind the airplane. I am having problems with the background not being visible, not moving as expected, the airplane not being visible, and just disappointment in general.I think the background of a div will not be visible if there are no internal contents because it will be sized to zero. So I introduced another transparent image to force the size of the main div. Still no joy.Here is my code. Note that I don't have my timers set up yet, just a button to test background movement.
free program that can convert my webpages containing javascript and html code into a flowchart. also i dont mind if the program works on either windows or ubuntu as i have both.
I have created which uses Gnome Seed which is in the Ubuntu repositories (although the newest version needs to checked out from Git to allow it to work) and SeedKit which unfortunately isn't in synaptic but needs to be downloaded. The links to get the source for both projects are as follows:
[URL]
The proof of concept is a JavaScript/HTML based frontend of Gnome's Log Viewer. It doesn't replicate 100% of Log Viewers functionality but it produces the basics of it which is to display the contents of Linux log files. The tutorial and link to download the source files are as follows:
[URL]
For more information about SeedKit and HTML based User Interfaces are in my blog, link for that post is as follows:
I managed to configure my W890i phone to get access to internet through an ubuntu-based computer. It's very easy to use the phone to give internet access to the computer, but the opposite is quite more tricky. For that I've done the following
----On the phone---
-Set the USB network option to "through computer", so that the phone uses the computer's internet connection and not the opposite.
-Decide and set "Shared Network" parameters: user, pasword and workgroup.
-In "conectivity-> internet connection" set "allow local network" to "yes"
----On Ubuntu 10.04---
-Install samba, samba-client, smbfs, smbclient, firestarter and dhcp3-server
-Configure Samba (System-> Administration-> Shared folders): same workgroup as in the phone, add new user (the phone), passwd this new user. In my case the user was called "w890i" and the password given was the same.
-Once the phone is connected to the computer through USB (then select "phone mode"), a new connection appears in NetworkManager: usb0.The aim is to create a shared network that gives internet access to this device. Edit the IPv4 parameters of this new connection, set them to Manual and give an IP adress (192.168.0.1) and a subnet mask (255.255.255.0); the rest of the fields are left empty.Connect this network.
-Set firestarter to use dhcp3: sudo ln -sf /etc/init.d/dhcp3-server /etc/init.d/dhcpd
-Launch firestarter and follow the wizard. Set "allow internet shared connection", choose the device for the primary internet access, and then the device for the shared network (usb0). Then change the settings for firestarter: activate DHCP for local network, set IP to the one we gave before (192.168.0.1).
-Open dhcp3-server config file sudo gedit /etc/default/dhcp3-server And set INTERFACES="usb0"
-Set the policies of firestarter: in incoming connections, allow connections from the IP adress given to the phone (192.168.0.1). Then add rules for the ports that need to be open for this connection. I opened HTTP, HTTPS, SMB, SMTP, POP3, IMAP, IMAPS, DHCP for all the connections in the local network.
-Apply policies and start the firewall.
------------
After all this, the phone can access the internet through the computer. Two problems appeared:
1. I couldn't get access to https sites, like webmails. The phone gave a "communication error". But then I tried with Opera instead of the browser built in the phone's firmware, and I could finally get to https sites.
2. I couldn't retrieve mail, neither POP nor IMAP nor IMAPS. I thought it was a firmware problem again, and I tried out several mobile phone email clients written in java, but none of them worked.
So this is at the moment the problem. If I connect from the phone to the internet directly through 3G, the email clients work for all my accounts. I don't think it's a firewall problem, because the ports are opened for this connection
Im having trouble using my wireless usb dongle. I've installed the software and connected to the web, but i can only access a number of websites.
I can access ....., and watch vids and perform searches on google, bing, yahoo and the like. However when i try and access a site that i have searched for it takes me to the internet providers homescreen and tells me to activate my account or top up my account.
I know i have enough GB left as i recently activated my account and i'm able to use the wireless stick on another computer without any problems on all websites. So this leads me to believe that there is maybe a problem with my browser or security settings, but im not sure what to do.
How do I get rid of my wubi partition since Ubuntu doesn't work on a Vista laptop. Sorry, it was a gift for Christmas. I never realized Ubuntu was so defective.I am running Windows Vista 32 bit with 3 gigs of ram and one partition from the factory (HP).
I have a VPS - and I was trying to install gd support for php on to it. Once I had done this I had a 500 Server Error. With the following errors in the log
[Sun Dec 12 07:34:29 2010] [error] [client 2.125.204.56] SoftException in Application.cpp:574: Could not execute script "/home/perfectp/public_html/phptest.php" [Sun Dec 12 07:34:29 2010] [error] [client 2.125.204.56] Caused by SystemException in API_Linux.cpp:444: execve() for program "/usr/bin/php" failed: No such file or directory
How to add parental control like { block web site access, restricting login access for particular time, restrict web access} for particular user accounts in Linux.
I am using Ubuntu 10.04, when connected to the ethernet through , Atheros Communications AR8152 v1.1.I can successfully access the internet. But when I try to use the wireless I can connect to the network using Atheros Communications AR9285, but can't access any internet sites. Or ping any other devices on the network.I'm using an Acer Aspire One D260
Strangest thing...yesterday my 11.04 box lost the ability to access anything external. I can ssh to the box from outside my network, and all other boxes on my network are fine, but this one box can't hit anything external. If I attempt to ping a site it's doing the same resolution, but no packets are delivered.
Up until yesterday it was running fine, and I didn't make any changes to my home network or install any updates. Totally stumped.
Edit: this is a wired connection with a static internal IP.
I need a textbrowser with Javascript Support. I have compiled the elinks stable 0.11x and the unstable 0.12x version mit JavaScript Support (js moz dev). Both browsers can be started and showed websites normaly. But no javascript, even simple tests like [URL] failed. I started elink with ./elink and typed then the url.
I have a question which it believe it is quite simple but I have no clue how to do it... I'm using Fedora 12 and I'm the only Linux machine in my office's LAN... I have Apache (httpd) setup and my co-workers can access the sites perfectly by using my IP address, for example [URL]... So here's the question, how do I setup a domain name so everyone in the LAN can access the sites by typing [URL]..
I'd like to ask a question on installing software.
I cannot install software using "yum" but I can with the GUI program "Add/Remove Software".
For example, I tried:
It will exhaust all the mirror sites without success.
All the error messages look like:
But I can install "screen.i686" easily from the "Add/Remove Software" GUI and it is very fast.
However, what's puzzling is that I could install some certain packages although I still see the same kind of error messages.
For example,
will work. But in general, "yum" is running very slow and is rejected at every mirror sites.
My settings: I just installed "Fedora 14" (Linux 2.6.35.11-83.fc14.i686) as a virtual machine (using VirtualBox) on a Windows XP (Professional, SP3) host on my laptop (CPU: Intel Core2 Duo), which accesses internet through a HTTP Proxy server since it is behind a corporate firewall.
Unable to access our site via eth0. Can access the same site, via same ISP on wireless laptop, or via eth0, but using ip annonymizer service. Have a feel that it could be something obvious, but after digging through Modems firmware I am running out of ideas
I installed Apache2 and I added a second virtual host, now can i separate the 2 hosts and let them access 2 different internal test sites? For Example lets say"
User1 uses one IP User2 uses second IP
How can I separate it in Apache2 and in the /var/www ?
I am in China, where some web sites(....., facebook, etc) are blocked. So I have to access these blocked site by VPN. But how can I access blocked sites by VPN and unblocked sites by my direct ADSL?
In the past week or so I've noticed some weird network behaviour. I find accessing some sites such as Amazon, Paypal, and Bigstockphoto really slow. Sometimes the page will not load at all. Other sites are fine. The problem sites are not a problem for others on my LAN at home. When I try to open the problem sites, I can see in Firestarter blocked connections coming from 2.1(8/9).xxx.xxx on various ports such as 36007. This only happens for the problem sites. I attached a typical output from firestarter.
This happens with Firfeox or Chrome. Using Ubuntu 10.10
Alright this works fine to pull the directories/files in the /var/Store/2010/ directory.
But when you click on of the links it tries to http://'serveradress'/$filename
note that $filename in the url is the filename clicked on so the php script is working. but I need it to change to that dir so that you can see the folder/files there and work your way up/down/side wise thru the folder tree to where you need to go. Not try and pop it as a direct url which doesn't work.
I have a local network that has three LANs and each LAN has several machines. I created a folder named myweb that contains html files under /var/www/html/. Let's say the folder is on A machine. From B machine, which is in different LANs, I want to access myweb folder using a web browser. I can ping both machines nut I can't access the folder. I tried [URL] where sxa.org is the hostname and the http server (Apache) name.
I have a situation where I am in a non-interactive shell. I have tried from within my non-interative shell to spawn an interactive shell but my output still does not goto me. Isn't there a way I can somehow go into /proc or somwhere and make the output my /dev/tty1? Or some way else to remedy this?The situation arises because I drop from my restricted shell environment (a sort of CLI interface), into the actual Linux shell. I cannot change the code of the CLI environment I am just faced with being in the linux shell environment and its non-interactive. Its very annoying to have to put > /dev/tty1 after every command I type.
Not to mention it seems damn near impossible to get pagers like more and less to work properly when your in a non-interactive shell.
Basicly just installed a fresh version of Cent OS 5.4 with apache httpd installed automaticly during the installation. The http daemon is running and when navigating to localhost i get the welcome cent os apache page thingy. The problem is when i put an index.html file in /var/http/www then try and navigation to localhost I get a 403 forbidden error.
