I want to automate form filling on a website with information of certain parameters that will return products based on the parameters I enter. I tried using mechanize in python but it does not support javascript and it seems like in order to navigate the entire process of filling in parameters requires pressing some buttons that seem like javascript objects. For instance the Guided Selection button:
<a onclick="_gaq.push(['_trackEvent', 'Navigation Menu', 'Click', 'Guided Selection Link']);" id="ctl00_NavigationMenu_ConfigureLink" href="javascript:__doPostBack('ctl00$NavigationMenu$ConfigureLink','')">Guided Selection</a></li>
I also tried using selenium but I do not want to create a new instance of a browser. Any python based suggestions? Perhaps jython? Thanks a lot!
My recommendation would be to simply capture the form information using a tool like firebug. And then "replay" the request (whether it's GET or POST) through Python. Here is a snippet for a POST request.
import urllib
import urllib2
url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
In fire bug you can get the url and values from under the console tab.
This probably won't work if you are doing automated form discovery.
Related
I'm trying create a simple UI here on my iOS app to test a thing or two out but I'm having some issues here. My app is set up with a UITextField and UIButton. I'm trying to replace a string on my .js file which is saved on my virtual server. In my .js file I have below:
// Prepare a new notification
var notification = new apn.Notification();
// Display the following message (the actual notification text, supports emoji)
notification.alert = 'Hi James';
I basically would like to replace "Hi James" with whatever I typed in the UITextField in my Swift 3 project but not too sure where to start. This would be my first time sending data to .js file so anything would help. I'm thinking so far that it'd be something along the lines to below. Node.js would be similar to Javascript since it's cross platform.
func sendSomething(stringToSend : String) {
appController?.evaluateInJavaScriptContext({ (context) -> Void in
//Get a reference to the "myJSFunction" method that you've implemented in JavaScript
let myJSFunction = evaluation.objectForKeyedSubscript("myJSFunction")
//Call your JavaScript method with an array of arguments
myJSFunction.callWithArguments([stringToSend]) }, completion: { (evaluated) -> Void in
print("we have completed: \(evaluated)")
})
}
Found that on a relevant StackOverflow post so I feel like I'm getting close. Any assistant would be appreciated in advanced. Have a good one!
I recommend using the Node HTTP or ExpressJS server reading the POST fields and posting a document from your iOS app with the desired field
See
https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/
I am trying to scrape the resulting page from this page:
http://data.philly.com/philly/property/
I am using 254 W Ashdale St as my trial entry, when I do that in my browser it directs me to the result I'm looking for in the HTML (same URL though).
Python requests is successfully putting the address I put in in the results page, but I am not able to get the owner information, which is what I am trying to scrape. I have been trying with Selenium and phantomjs, nothing I am doing is working.
I was also confused about the form action, it seemed to just be the same URL as the page the form is on.
I appreciate any and all advice or help!
Selenium takes care of virtually everything, just find the elements, enter the information, find the button, click on it, then go to the owner, click on it and get scrap the info you need.
import selenium
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('http://data.philly.com/philly/property/')
#enter the street address
driver.find_element_by_name('LOC').send_keys('254 W Ashdale St')
#click on the submit button
driver.find_element_by_name('sendForm').click()
#find the owner
owner_tag = driver.find_elements_by_tag_name('td')[2]
owner = driver.find_elements_by_tag_name('td')[2].text
print(owner)
#click on the owner
owner_tag.find_element_by_tag_name('a').click()
#get the table with the relevant info
rows = driver.find_element_by_tag_name('tbody').find_elements_by_tag_name('tr')
#get the row with the sale prices
sale_prices = list()
for row in rows:
sale_prices.append(row.find_elements_by_tag_name('td')[4].text)
print('\n'.join(sale_prices))
Output:
FIRSTNAME LASTNAME
$123,600.00
$346,100.00
[..]
$789,500.00
i developed a simply.js app that fetches bus arrival time from a webservice, problem is that as of now it work only for one stop.
i want to create a configuration page with a multiselect where i could choose multiple stops , sending them to the pebble as an array and at the press of up/down buttons i want to cycle the array to show different bus stops.
Im not good in C, i prefere javascript thats because i used simply.js.
id like to know and learn how to do it, because i think online there isnt much documentation and examples.
Found a similar question/ issue at simply.js github page https://github.com/Meiguro/simplyjs/issues/11. The code example below comes from Meiguros first answer. The code sends the user to your configuration website, which you should configure to send json back.
You can probably copy the code example for enabling the configuration window and paste it in the begining of your main pebble app.js file. Do not forget to add "capabilities": [ "configurable" ], in your appinfo.json file. If you are using cloudpebble you should go to the settings page of your app and make sure the configurable box is checked.
var initialized = false;
Pebble.addEventListener("ready", function() {
console.log("ready called!");
initialized = true;
});
Pebble.addEventListener("showConfiguration", function() {
console.log("showing configuration");
//change this url to yours
Pebble.openURL('http://assets.getpebble.com.s3-website-us-east-1.amazonaws.com/pebble-js/configurable.html');
});
Pebble.addEventListener("webviewclosed", function(e) {
console.log("configuration closed");
// webview closed
var options = JSON.parse(decodeURIComponent(e.response));
console.log("Options = " + JSON.stringify(options));
});
(https:// github.com/pebble-hacks/js-configure-demo/blob/master/src/js/pebble-js-app.js - remove space after https://)
To then push the settings back to the pebble i think you need to add
Pebble.sendAppMessage(options);
just before
console.log("configuration closed");
// webview closed
I found this out at the last post on this pebble forum thread http://forums.getpebble.com/discussion/12854/appmessage-inbox-handlers-not-getting-triggered-by-javascript-configuration-data
You can aslo find a configuration website example named configurable.html in the same git as the code example at https:// github.com/pebble-hacks/js-configure-demo remove space after https://
Hope this helps a bit on the way to achieving your goal
So the configuration page is a web page, and you can host it and provide your URL as mentioned by Ankan above.
Like this:
Pebble.openURL('http://assets.getpebble.com.s3-website-us-east-1.amazonaws.com/pebble-js/configurable.html');
Lets say you decide to take the name and age of the user in the configuration page, you would have two text fields for them to enter their information, then you would have a submit button. For the submit button write a javascript function which uses jQuery to take the values of the text fields onclick, then save those values to a variable, and use JSON to send them to the phone. Here is an example of a fully created configuration web page: https://github.com/pebble-hacks/js-configure-demo
Enjoy.
The reason I am specifying Google Script and not just javascript in general is because the examples I have found for javascript all use html, which Google Script does not appear to play well with outside of basic formatting commands.
I have one function formSubmitReply() which sends an email based on a form submission, and another addToCalendar() that automatically populates the form submission into a Calendar event. Both of these have been tested independently and function well.
What I am attempting to do is to include an "Add to Calendar" button in the email that will execute addToCalendar() when clicked.
The difficulty I keep encountering is the error "TypeError: Cannot find function addToCalendar in object Generic." How do I get around this?
Here are the relevant pieces of my code:
// Button setup
function doGet()
{
var app = UiApp.createApplication();
// Create button and give it a click handler
var button = app.createButton("Add to Calendar").setId("button");
button.addClickHandler(app.createServerHandler("formSubmitReply"));
app.add(button);
return app;
}
// Email function
function formSubmitReply(e)
{
// Call "button" application
var app = UiApp.getActiveApplication();
/* Additional function code; excluded here */
// Send email
GmailApp.sendEmail(userEmail,
"Event Support Request Submitted",
message +
app.getElementById("button").addToCalendar(e),
{name:"Media Services"});
}
I don't think it is posible to call an apps script function like that from an email button(or any js though i might be mistaken on that one). What you could do is serialize the information from e.Then embed a url to a apps script that will handle the encoded information passed as parameters. Url lengths are limited so you might want to embed a form that submits the data to a apps script if the data you are sending is very large.
I'm trying to use python / mechanize to login to this webpage:
http://www.solaradata.com/cgi-bin/mainProgram.cgi
The login form uses a Javascript function that produces an MD5 hash from several field values before submitting the results for authentication. Since mechanize can't do javascript, I tried to replicate the same functionality inside of python and then submit the resulting values. However, I'm still getting "invalid user / password" errors.
Here's my current code, can anybody point me towards where I went wrong? Thanks!
url_login = 'http://www.solaradata.com/cgi-bin/mainProgram.cgi'
import mechanize
import md5
username = 'superfly' #not my real user/pass
password = 'stickyguy' #not my real user/pass
br = mechanize.Browser()
br.open(url_login)
br.select_form(nr=0)
br.set_all_readonly(False)
session = br['session']
br['user'] = username
br['password'] = password
m1 = md5.new()
m1.update(password + username)
br['password'] = m1.digest()
m2 = md5.new()
m2.update(password + session)
br['hash'] = m2.digest()
for form in br.forms():
#print form
request2 = form.click() # mechanize.Request object
try:
response2 = mechanize.urlopen(request2)
except mechanize.HTTPError, response2:
pass
print response2.geturl()
# headers
for name, value in response2.info().items():
if name != "date":
print "%s: %s" % (name.title(), value)
print response2.read() # body
response2.close()
Use m1.hexdigest() instead of m1.digest()
I'm not familiar with python, but it appears the they are returning the hex value of the MD5 hash in the javascript version of the algorithm. Does the python MD5 do the same?
You should be able to test this without going through the submission process and testing for success. Instead, using a JavaScript developer tool such as Firebug or Chrome developer tools, calculate the result you get in-page. Then, using the same inputs, see what you get from your program. They should match, character for character.
It way be overkill but if you really need to script access a javascript heavy site you can look at selenium-rc or source labs.
These tools allow you to script an actual browser the same as a user.
Selenium