I am building a Spider in Perl and have a problem:
The Site I want to spider uses a JavaScript for Age-Verification and I don't know how to get past this in Perl...?
The Script looks like this:
<script type = "text/javascript">
function set_age_verified(){
new Request({
method: "post",
url: "/user/set_age_verified"
}).send();
$('age_verification').setStyles({visibility: 'hidden', display: 'none'});
$('page_after_verification').setStyles({visibility: 'visible', display: 'block'});
return false;
}
</script>
And here the OnClick Event :
<img src="http://example.com/age-verification-enter.gif" alt="ENTER">
The function has two effects. One is to POST a request to the URL "/user/set_age_verified" and the other is to alter the display visibility of some HTML.
Your spider can easily ignore the second effect, but presumably the first effect, by going to the server, sets some cookie or server variable which the server will require.
You do not have to actually run the javascript, so long as the server sees the same POST data.
The answer is for your Perl script to detect pages which have this javascript, and to call a Perl function to POST the data to the age verification URL.
Any cookie or similar which is returned will have to be recorded by you - your HTTP library may take care of this for you though.
What Perl modules are you using? WWW::Mechanize has an AJAX plugin, although it hasn't been updated in a while. I guess you could also look at something like WWW::Selenium.
But I bet that AJAX request is going to inject some HTML that requires the user to input some data, then submit a form. Pretty tricky to cover all bases for that general case...
Take a look at the WWW::Mechanize::Firefox module. It allows you handle some JavaScript.
Also, in Firefox HTTPHeaders is your best friend.
Turn it on, manually click what ever you need to in order for the Javascript to run and submit to the server, then go back to the HTTPHeaders window. It will show you exactly what that Javascript event sent to the server (GET or POST + the data, even if it is HTTPS) - as well as the server response.
Related
Ok here's my problem. I'm working on this little site called 10winstreak and I'm trying to detect if a stream is live or not with javascript because our server that we run the site off of cant handle processing every single request with PHP. The basis of detecting if a stream is live or not is you go to their XML file and in one of their tags (if it's live) it will say something along the lines of true and often time the XML file on their site will be empty if a particular stream isn't live. for example if you have a twitch.tv stream for gamespot you go to http://api.justin.tv/api/stream/list.xml?channel=gamespot and if it's got stuff in it then it's live if not then it's not.
so basically my code looks like this:
function check (URL, term){
$.get(URL , function(data){
console.log(data);
//data is whatever the server returns from the request, do whatever is needed with it to show who is live.
var number = data.search(term);
if (number > -1)
{
document.write("Live");
}
else
{
document.write("Offline");
}
});
}
and URL is a url that gets passed in and term is the term to search for in the xml file (usually "true" or "True"). but before anything happens I end up with "XMLHttpRequest cannot load http://api.own3d.tv/liveCheck.php?live_id=6815. Origin (my server's URL) is not allowed by Access-Control-Allow-Origin."
I've looked into it all over the net and i dont seem to be able to find anything that I can use. there's alot of theory stuff but not enough actual code and i dont understand the theory stuff to be able to start typing code out. from what i've seen you have 2 ways to go, use JSONP or add a line somewhere in your sever to allow for cross-domain accessing. neither of which i understand fully nor know how or what to do. It would be alot of help for someone to show me what needs to be done to get rid of this error. of course if you can explain it to a non-coder like me it would be even more awesome but at my current point, as long as the code works for all I care it might as well be magic lol.
You can solve it :)
Take a look at xReader
<script src="http://kincrew.github.com/xReader/xReader.full.js"></script>
<script type="text/javascript">
xReader("http://api.own3d.tv/liveCheck.php?live_id=6815", function(data) {
alert(data.content);
})
</script>
I think you need cacheburst option. but you can be banned from YQL.
I think its because the path is not relative. You may be calling this from a different domain/sub-domain. You can potentially allow other origins to access, which may open up a security hole or you can create a proxy locally.
In PHP creating a proxy is easy: http://blog.proxybonanza.com/programming/php-curl-with-proxy/
Now, instead of directing your request straight to that URL send the request from jQuery to your own local url and have it access it on the server side.
Another option would be to use YQL: http://www.parrisstudios.com/?p=333 (I wrote an article about this a while ago)... In that way you can turn the response into JSON, which can be accessed cross-domain (as can javascript).
You could ask for the API responses to all be returned using a JSONP server and in JSON.
You aren't going to be able to do this via client-side javascript unless they've enabled some way to retrieve their data cross-domain (CORS, JSONP, some flash widgety thing getting read permissions from crossdomain.xml file(s) located on their server...)
Short answer: unless 10winstreak offers a JSONP service, you'll have to do things on the server-side.
Slightly longer answer:
For security reasons browsers won't let you make AJAX requests from www.example.com to www.example2.com (or any other domain except www.example.com). There isn't much you can do about this except use JSONP (and you can only do that if the remote webservice offers it).
Therefore, what you end up needing to do is ask your server "hey what's on that other server?" and (since it's not limited the way a browser is) it can go get the XML from that other server. There are various ways of doing this, either with code or Apache config; not sure what's right for you, but hopefully now you understand the general principle.
P.S. See this question: Wouldn't have been simpler to just discard cookies for cross-domain XHR? if you are curious why browsers do this.
* EDIT *
I just checked out JustinTV's site, and it appears that they already have a PHP library for you to use:
https://github.com/jtvapi/jtv_php_api
This is very likely your best bet (if you want to keep using PHP that is; if not they have libraries for other languages: http://www.justin.tv/p/api).
the url is input by end users as string on my page, so may point to any domains.
JavaScript in current page needs to sniff the url, verify whether it's still valid, and return the types as image, or video, or audio, even considering html5 video audio tag and existent flash embed. And No need to wait for the complete file transfer.
Can someone help, from concept? thanks very much.
i'm aware the cross domain problem on ajax. So no idea on basic how-to.
If what you're asking, is:
Given any URL -> lookup given URL using a javascript ajax request, and determine if it is a video/audio/image - then, once detected, use the URL accordingly, then you can do something like this:
jQuery and AJAX response header
However, you'll not be able to make a request using client-side JavaScript to another domain, as it will require a cross-domain request (where your alternatives are JsonP, or weird headers in the response).
You're better off passing the URL to your own server, and performing the logic there (Via some kind of server-side web request) and passing a payload back to the client, with the required information in JSON or something - e.g.
{payload: 'video'}
Old question, but I recently wrote a utility that might help you out. It's a CORS-enabled MIME-type checker. See the API doc at lecoq.herokuapp.com
Use it like so: example
I am using jquery for all my ajax thing, I don't know if that is fine but I use that for now.
I have one text input when user type characters in it I call server side get some values and add them on the view.
Code that I use bellow works fine but I want to improve it a little.
How can I make this ajax call so that users that want to investigate my page source code can't see what I call here?
So basically I want to hide from page source what url, what type and data send I use here, is it possible?
$(function () {
$("#txtSearch").keyup(function (evt) {
$.ajax({
url: "/Prethors/Users/SearchUsers",
type: "POST",
data: "text=" + this.value,
success: function (result) {
$("#searchResult").prepend("<p>" + result + "</p>");
}
});
});
});
No, a user will always be able to figure out what calls you are making if you include it in javascript.
You can compress and minify the javascript, but a determined person will always be able to find your url calls.
Here's a js compression site, for example.
http://jscompress.com/
overall, you shouldn't worry about this. there is no way I'm aware of to hide your ajax calls, but you shouldn't need to.
-you could encrypt the info.
-you could use comet to stream the data on a persistent connection. (super complicated).
-follow good server security practices and not worry about it.
source: here
If you are really worried about this, you could set up kind of an anonymous URL, which will then redirect to where you really want to go based on some variable which is arbitrary.
for example, instead of going to "/Prethors/Users/SearchUsers"
go to "/AnonymousCall?code=5"
from which you could execute the code you want for searchusers
You can't hide client-side code. You can disguise it with minification but sensitive data should always be stored and processed on the server-side.
Use console.clear(); after you ajax calls :P
It just clears the reqs from the console but you still cannot hide client side calls.
i've got a little problem with AJAX, CouchDB and JavaScript.
I can open the following URL from CouchDB in my browser: http://192.168.1.58:5984/mydb/name
new Ajax.Request('http://192.168.1.58:5984/mydb/namee', {
method: 'POST',
onComplete: function(transport) {
alert(transport.responseText);
}
});
I always get empty alert.
Can you help me?
The problem here is, that your browser doesn't allow you to make a query on an other web server than the one where you're script originates. (Google for: Same Origin Policy)
But there is a kind of a common technique which is a workaround for this use case. It's called JSONP. Since version 1.0 you have to activate this functionality first in CouchDB. In the section [httpd] of your CouchDB configuration file (.ini) you have to add an
allow_jsonp = true
After this is done you can produce JSONP queries on your CouchDB. Basically adding dynamically lines like this:
<script type="text/javascript"
src="http://server2.example.com/getjson?callback=parseResponse">
</script>
But for details refer to the article linked above.
Anyway I propose on the JavaScript side of things to use a Framework as jQuery, DojoToolKit, ect. In jQuery e.g. it is enough to add "?callback=?" at the end of the URL.
AJAX doesn't support cross domain scripting. all calls need to be to a URL with the same domain as the one of the current document.
a good solution would be to build a proxy service on the server side, that will take the local
request, make an HTTP call to the couchDB server, and return it's response.
Can we do client side validation using AJAX in struts 2 application ?
If yes, then please let me know the procedure.
Answer is right here.
http://struts.apache.org/2.x/docs/ajax-validation.html
even provides prototype example instead of dojo
As Sergio said, you could use this, for example, in a registration form where you have fields like:
Username
E-mail
Password
Repeat Password
You can set up some javascript to validate the form before the user submits it, or better yet, don't enable the submit form until all required fields are complete, avoiding user frustration (you tell him specifically where the error is and how to correct it) and adding another layer of validation to it.
In this case you can pre-validate Username and E-mail against the server to see they are not taken yet. You can do something like this (in jQuery, from memory):
$("#email").change(function(e) {
var $elem = $(this);
var val = $elem.val();
if (!val) {
myMarkField($elem, "not-valid");
myValidateForm("myForm");
} else if (!/^[a-z0-9_+.-]+\#([a-z0-9-]+\.)+[a-z0-9]{2,7}$/i.test(val)) {
myMarkField($elem, "not-valid");
myValidateForm("myForm");
} else {
$.getJSON("isEmailAvailable.php", {email:val}, function(result){
if (result.available) {
myMarkField($elem, "valid");
myValidateForm("myForm");
} else {
myMarkField($elem, "not-valid");
myValidateForm("myForm");
}
});
}
});
Where isEmailAvailable.php should return something like { available: true }
AJAX is the mechanism by which you could invoke the server-side validation. The "J" in AJAX stands for Javascript.
You would typically make use of AJAX (and hence Javascript) to invoke some sort server-side validation that would be inappropriate for client-side validation. For example, the validation of a zip code against a table of acceptable values.
How would you do this? You use JS for client-side programming (including validation), it has nothing to do with the server side.
Additionally: server-side-validation via the client-side JS would be a bad idea, as all your validation code would be exposed to the user.
If your going to do server-side input validation then what do you need to use JavaScript for?
I'll make the assumption that you are using PHP on the server-side in which case you'll want to validate it there instead of making another unnesicary request.
If you want to use AJAX to do some server-side validation (and you'r using JQuery) this should at least get you started:
function validate(fieldName)
{
$.get('/validation.php',
{'value' : fieldName.value, 'name' : fieldName.name},
function(data, textStatus)
{
if(data)
{
fieldName.style.class = 'validFormInput';
}
else
{
fieldName.style.class = 'invalidFormInput';
}
})
}
I'm no JS/JQuery expert and I didn't test this so it's just to get you started.
I think you need to read a bit about AJAX.
I recommend this link: http://rajshekhar.net/blog/archives/85-Rasmus-30-second-AJAX-Tutorial.html
I think it's simple and easy to understand.
Basically you have server and client side code. JavaScript (client side) can invoke a URL using Ajax(Asynchronous JavaScript And XML) witch is very similar to invoke any other URL using a browser. The main difference is that there is no page reload.
Your JavaScript will need to be waiting for the reply in order to handle it (in your example show or hide an error message).
Basically it allows you to execute some code at the server an update your page without a page refresh.
In short, the AJAX application architecture enables the client (i.e. a piece of JavaScript running in a web browser) to actively ask for information from a server. Most often, the JavaScript client will use the received information to update small portions of the browser's document using the DOM.
Also, the JavaScript client code can do some obvious validation of e.g. a form to be sent to a server (as shown so nicely by Danita's answer).
The server can be anything. It may just return a plain xhtml file, or may respond to the request by calling a complex PHP script that generates a response. Heck, it may even be a dedicated mobile device entirely implemented in bare C. As long as it implements the http protocol the client's request was transmitted in. This might be JavaScript. Improbable but true.
So Yes: it can be done using 'server-side' JavaScript, if you have a server running JavaScript. But No: that's not probable.
And Yes: you apparently need to read upon AJAX. (I'm not a webdesigner, and even I seem to know more than you ;) Don't leave it that way!)