I am looking for a way to manually submit a Post request to a server, without using the website's UI. I can see the request headers and the post parameters in Firebug when I perform the action manually (clicking the UI's "submit" button). I am hoping there is a way to reverse engineer some Javascript using these headers and parameters so that we can automate this process.
Reason: My company recently purchased some process automation software that enables us to write automation bots that access out business partner's portal site and automatically adjust our digital marketing bids. For one of our partner sites, front-end manipulation doesn't appear to work, because the Post request is submitted via AJAX.
The software does allow us to execute custom javascript within the environment, so I am trying to construct some Javascript using the headers and request parameters.
Is there a standard template into which I can plug these parameters to execute Javascript that will send the Post request to the server?
Thank you
UPDATE:
Thank you all for your help! I've made some progress but am still having difficulty implementing the solution within the software.
The following request works when I run the code in Firebug in Firefox:
$.ajax({
type: "POST",
url: "http://acp.example.com/campaigns/122828",
data: "data-string"
});
However, the software we're using might be a little out of date and I'm not sure it recognizes the AJAX syntax.
Is there a way to effectively write the same statement above, but in Javascript rather than AJAX? Then I think it would work.
You can use AJAX to post data to a server without any direct UI interaction. I will break down a simple jQuery example below:
$.ajax({
type: "POST",
url: url,
data: data,
success: success,
});
$.ajax Is a method offered by the jQuery framework to make AJAX requests simple and cross browser compatible. As you can see I have passed in a JSON object containing various values:
type - This is the first key I have specified, in this instance you'll want this to be of the value POST as this determines the HTTP Request Method.
url - This specifies the server end point, for example: post/data/here.php would post the data to that url so that it can be picked up and handled correctly.
data - This key expects a JSON object, string or array of data to send in the POST request.
success - This key expects a function, it is called on the server's response to the request, with any relevant data passed through.
More documentation is available at: http://api.jquery.com/jquery.ajax/
If all you want to do is POST data, no JavaScript needed.
You should be able to use a browser extension for this. I have one called REST Console that is similar to what you describe. I believe REST Console is for Chrome only, but a quick Google search yielded a similar looking extension for FireFox called RESTClient.
Related
I have a client-side web-app which is all static html and javascript, and I use JQuery. I am more of a back-end/middle-ware guy with Spring, and so I am not really good withthe front-end stuff.
I have a RESTful API from my client UI to my middle-ware web app written in Spring as a Controller. I can call this successfully, and there is middle-ware business logic where I can a third-party API and it returns HTML. I return this HTML to my client and I was trying to load it in an iframe or within a div tag. Of course, I got all kinds of javascript errors and a lot of CORS issues. So, if this isn't a feasible option to get HTML from another web-site and deploy it on my page, then I am willing to let that go.
I created another RESTful API on my back-end that now just returns a URL this third-party RESTful API. Specifically, I am calling the authorization page for Coinbase, but it could easily be any other third-party authorization page. I have the full URL that they recommend, and I am ready to do something with it.
So, I get a URL (dataType is 'text') like this, and I can see it with an alert:
https://www.coinbase.com/oauth/authorize?
response_type=code&client_id=MY_CLIENT_ID
&redirect_uri=MY_REDIRECT_URL
&state=SECURE_RANDOM
&scope=wallet:accounts:read
I figure I would do a redirect as follows:
window.location.replace(response);
but, what happens is I get an HTTP 404 because my system is pre-pending my UI locahost to that url. So, it looks like:
http://localhost:8080/my_web_ui_app/https://www.coinbase.com/oauth/authorize?
response_type=code&client_id=MY_CLIENT_ID
&redirect_uri=MY_REDIRECT_URL
&state=SECURE_RANDOM
&scope=wallet:accounts:read
And obviously, that is not going to work as I get the HTTP 404 error.
Anything else I should try? Like I said, I am not a good UI guy, but I just need something that works ok.
Thanks!
It was the Ajax call itself.
$.ajax({
url : 'http://localhost:8080/myapp-be/api/myrestapi/authorize',
headers : {'token' : token},
dataType: 'text',
cache: false,
success: function(response) {
console.log(response)
$(location).attr('href',response);
}
});
The dataType: 'text' was the culprit.
The URL was the response, and as 'text', it added another layer of quotes around the response.
Once I removed the dataType from the Ajax call, then the default dataType worked and I no longer got a URL with extra quotes. On success, I was able to do a simple redirect and that worked. Was a simple fix.
Thank goodness for Chrome and the debugging tools so I could see that the response coming back had that extra quotes around it.
I got a URL from a ecommerce website and when i access it i get all the last 5 products that i've visited in their site. I don't know how it works, i guess it's because of the cookie that this ecommerce website have left in my browser.
I would like to use this URL to show in my website something like this: "The Last 5 Products You Have Seen at X Ecommerce Website".
But to do that this URL must be executed in somehow in the client side and i will still need to get the JSON content returned by this URL.
Is there exist anyway to do that by using PHP or any other web technology?
Thank you!
It might be cookies, localStorage (there are other APIs to save data on local computer, imo they are unused or deprecated e.g. openDatabase) or last views could be connected with account and saved on internal database.
You should use AJAX, but by default in browser mechanism called CORS blocks all requests coming from other domain than resource.
In PHP you can download external page using file_get_contents function or cURL library, but without localStorage/cookies (which can be accessed from JS executed on domain, where that cookies are saved).
AJAX is your option for client side requests. Here's the jQuery guide for it.
https://api.jquery.com/jquery.ajax/
Here's a quick example:
$.ajax({
url: "http://ecommerce.com/your/url/here",
method: 'get',
dataType: 'json', //if you're sure its returning json you can set this
success: function(data) {
//handle success json here
//be sure that you're going to receive json though, possibly could receive some other data type and you should handle appropriately
},
error: function(error) {
//handle error json here
}
});
I am trying to post a JSON data struct to a REST API service from a webpage from. On submitting the form, javascript will capture form field values, make JSON data, then POST the data to my REST API. I am not entirely sure how to do this with JavaScript. Can anyone point me in the right direction or drop some sample code.
Thanks
Try jQuery, it's got some nice helpers for this kind of stuff.
Check out the jQuery.post documentation, you probably want the postJSON method.
As for getting started with jQuery, check out their getting started page
You can get a form's data with
var data = $('#formid').serializeArray();
Then you can use that in the postJSON method:
$.post(url, data, callback, "json");
Or for even easier solution, use my jQuery plugin: https://github.com/jpillora/jquery.rest
I need to POST to a different domain than my webpage will be on. Can I do this?
If you don't care about older IE's you can use headers to allow cross domain requests: https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS
the url is input by end users as string on my page, so may point to any domains.
JavaScript in current page needs to sniff the url, verify whether it's still valid, and return the types as image, or video, or audio, even considering html5 video audio tag and existent flash embed. And No need to wait for the complete file transfer.
Can someone help, from concept? thanks very much.
i'm aware the cross domain problem on ajax. So no idea on basic how-to.
If what you're asking, is:
Given any URL -> lookup given URL using a javascript ajax request, and determine if it is a video/audio/image - then, once detected, use the URL accordingly, then you can do something like this:
jQuery and AJAX response header
However, you'll not be able to make a request using client-side JavaScript to another domain, as it will require a cross-domain request (where your alternatives are JsonP, or weird headers in the response).
You're better off passing the URL to your own server, and performing the logic there (Via some kind of server-side web request) and passing a payload back to the client, with the required information in JSON or something - e.g.
{payload: 'video'}
Old question, but I recently wrote a utility that might help you out. It's a CORS-enabled MIME-type checker. See the API doc at lecoq.herokuapp.com
Use it like so: example
I am building a Spider in Perl and have a problem:
The Site I want to spider uses a JavaScript for Age-Verification and I don't know how to get past this in Perl...?
The Script looks like this:
<script type = "text/javascript">
function set_age_verified(){
new Request({
method: "post",
url: "/user/set_age_verified"
}).send();
$('age_verification').setStyles({visibility: 'hidden', display: 'none'});
$('page_after_verification').setStyles({visibility: 'visible', display: 'block'});
return false;
}
</script>
And here the OnClick Event :
<img src="http://example.com/age-verification-enter.gif" alt="ENTER">
The function has two effects. One is to POST a request to the URL "/user/set_age_verified" and the other is to alter the display visibility of some HTML.
Your spider can easily ignore the second effect, but presumably the first effect, by going to the server, sets some cookie or server variable which the server will require.
You do not have to actually run the javascript, so long as the server sees the same POST data.
The answer is for your Perl script to detect pages which have this javascript, and to call a Perl function to POST the data to the age verification URL.
Any cookie or similar which is returned will have to be recorded by you - your HTTP library may take care of this for you though.
What Perl modules are you using? WWW::Mechanize has an AJAX plugin, although it hasn't been updated in a while. I guess you could also look at something like WWW::Selenium.
But I bet that AJAX request is going to inject some HTML that requires the user to input some data, then submit a form. Pretty tricky to cover all bases for that general case...
Take a look at the WWW::Mechanize::Firefox module. It allows you handle some JavaScript.
Also, in Firefox HTTPHeaders is your best friend.
Turn it on, manually click what ever you need to in order for the Javascript to run and submit to the server, then go back to the HTTPHeaders window. It will show you exactly what that Javascript event sent to the server (GET or POST + the data, even if it is HTTPS) - as well as the server response.