Develop Reference

Can i scrape this site using just node?

im very new to JavaScript so be patient.
I've been trying to scrape a site and get all the product URLs in a list that i will use later in other function like this:
url='https://www.fromuthtennis.com/frm/c-10-mens-tops.aspx'
var http = require('http-get');
var request = require("request");
var cheerio = require("cheerio");
function getURLS(url) {
request(url, function(err, resp, body){
var linklist = [];
$ = cheerio.load(body);
var links = $('#productResults a');
for(valor in links) {
if(links[valor].attribs && links[valor].attribs.href && linklist.indexOf(links[valor].attribs.href) == -1){
linklist.push(links[valor].attribs.href);
}
}
var extended_links = [];
linklist.forEach(function(link){
extended_link = 'https://www.fromuthtennis.com/frm/' + link;
extended_links.push(extended_link);
})
console.log(extended_links);
})
};
This does work unless you go to the second page of items like this:
url='https://www.fromuthtennis.com/frm/c-10-mens-tops.aspx#Filter=[pagenum=2*ava=1]'
var http = require('http-get');
var request = require("request");
var cheerio = require("cheerio"); //etc...
As far as i know this happens because the content on the page is loaded dynamically.
To get the contents of the page i believe i need to use PhantomJS because that would allow me to get the html code after the page has been fully loaded, so i installed the phantomjs-node module. I want to use NodeJS to get the URL list because the rest of my code is written on it.
I've been reading a lot about PhantomJS but using the phantomjs-node is tricky and i still don't understand how could i get the URL list using it because i'm very new to JavaScript or coding in general.
If someone could guide me a little bit i'd appreciate it a lot.

Yes, you can. That page looks like it implements Google's Ajax Crawling URL.
Basically it allows websites to generate crawler friendly content for Google. Whenever you see a URL like this:
https://www.fromuthtennis.com/frm/c-10-mens-tops.aspx#Filter=[pagenum=2*ava=1]
You need to convert it to this:
https://www.fromuthtennis.com/frm/c-10-mens-tops.aspx?_escaped_fragment_=Filter%3D%5Bpagenum%3D2*ava%3D1%5D
The conversion is simply take the base path: https://www.fromuthtennis.com/frm/c-10-mens-tops.aspx, add a query param _escaped_fragment_ who's value is URL fragment Filter=[pagenum=2*ava=1] encoded into Filter%3D%5Bpagenum%3D2*ava%3D1%5D using standard URI encoding.
You can read the full specification here: https://developers.google.com/webmasters/ajax-crawling/docs/specification
Note: This does not apply to all websites, only websites that implement Google's Ajax Crawling URL. But you're in luck in this case

You can see any product you want without using dynmic content using this url:
https://www.fromuthtennis.com/frm/showproduct.aspx?ProductID={product_id}
For example to see product 37023:
https://www.fromuthtennis.com/frm/showproduct.aspx?ProductID=37023
All you have to do is for(var productid=0;prodcutid<40000;productid++) {request...}.
Another approach is to use phantom module. (https://www.npmjs.com/package/phantom). It will let you run phantom command directly from your NodeJS app

Is there a way to AutoFormat (Javascript) code in TestComplete?

So similar to ALt-Shift-F in Netbeans, is there a to do this right in the ide in TestComplete? Not sure if this is possible or if anyone can think of a workaround to autoFormat without leaving the TestComplete window.
I'm trying to get the below solution to work with http://jsbeautifier.org/ for javascript / Jscript code in TestComplete.
Thanks

Great question!
There is no built-in function for that. So, we should not expect any solution to be 100% convenient - it is just not a simple task to modify the current script editor contents (if at all possible). So, whatever you do, it will still be some kind of compromise.
In general, the task is three-fold:
Get the current unit code.
Format the code.
Put the code back to the unit.
According to my understanding, items 1 and 3 can be accomplished only by creating a TestComplete plug-in - accessing editors for project nodes is not an easy thing.
UPDATE: silly me! There is a way to access the script editor code - I've updated the below part.
What will help us avoid switching to a different app, are the Script Extensions:
We create a custom Checkpoint in the form of a Script Extension, and install it to TestComplete. As a result, we get a button on the toolbar that we can click to invoke our code.
In the design time action, we call some code that reads the editor contents, then uses external code formatting functionality, and replaces the editor contents with the formatted code.
It would extremely interesting to see the implementations other TestComplete users can suggest! As a start, I am posting a solution that includes using an external web site to format VBScript code (http://www.vbindent.com/). I know that the starter of the post is probably using JScript, but I have not found a JScript formatter yet.
My solution is a simple Script Extension. I can't post a file here, so I will post the code of the two Script Extension files:
Description file:
<!-- Description.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<ScriptExtensionGroup>
<Category Name="Checkpoints">
<ScriptExtension Name="VBScript Code Indent" Author="SmartBear Software" Version="0.1" HomePage="smartbear.com">
<Script Name="VBIndent.js">
<DesignTimeAction Name="Indent Current VBScript Unit" Routine="DesignTimeExecute"/>
</Script>
<Description>
Indents VBScript code in the currently active unit.
</Description>
</ScriptExtension>
</Category>
</ScriptExtensionGroup>
Code file:
// VBIndent.js
function DesignTimeExecute()
{
if (CodeEditor.IsEditorActive)
{
var newCode = IndentVBSCode_Through_VBIndent(CodeEditor.Text);
if (null == newCode)
return;
CodeEditor.Text = newCode;
}
}
function IndentVBSCode_Through_VBIndent(codeToIndent)
{
var URL_VBIndent = "http://www.vbindent.com/?indent";
var httpObj = Sys.OleObject("MSXML2.XMLHTTP");
httpObj.open("POST", URL_VBIndent, false);
httpObj.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
httpObj.send("thecode=" + escape(codeToIndent));
var responseText = httpObj.responseText;
// Extract the indented code from the response
var rx = /<textarea name=\"thecode\".*?>((.*\n)*?)<\/textarea>/;
matches = rx.exec(responseText);
if (null == matches)
{
return null;
}
codeIndented = matches[1];
return codeIndented;
}
After you create these files, and put them to something like "\Bin\Extensions\ScriptExtensions\VBIndent", and click "File | Install Script Extensions | Reload", you will see a new "Indent Current VBScript Unit" item in the custom checkpoints drop-down button on the Tools toolbar. Clicking the element will format the VBScript code in the currently active editor.
So, this is to give a clear idea of what a solution can look like. Better suggestions are welcome! Share your thoughts!

FYI
I've done. Based on your posts.
JSFormat.tcx
https://drive.google.com/uc?export=download&id=0B1x_73bHRc2Jcm8wbTJ2dUpZQTQ
To install the extension copy attached file JSFormat.tcx to C:\Program Files (x86)\SmartBear\TestComplete 10\Bin\Extensions\ScriptExtensions
To use view next image:
https://drive.google.com/uc?export=download&id=0B1x_73bHRc2Jc3RuLXFpTnlCSnc
Regards

Issues in developing web scraper

I want to develop a platform where users can enter a URL and then my website will open the webpage in an iframe. Now the user can modify his website by simply right clicking and I will provide him options like "remove this element", "copy this element". I am almost through. Many of the websites are opening perfectly in iframe but for a few websites some errors have shown up. I could not identify the reason so asking for your help.
I have solved other issues like XSS problem.
Here is the procedure I have followed :-
Used JavaScript and sent the request to my Java server which makes connection to the URL specified by the user and fetches the HTML and then use Jsoup HTML parser to convert relative URLs into absolute URLs and then save the HTML to my disk in Java. And then I render the saved HTML into my iframe.
Is somewhere wrong ?
A few websites are working perfectly but a few are not.
For example:-
When I tried to open http://www.snapdeal.com it gave me the
Uncaught TypeError: Cannot read property 'paddingTop' of undefined
error. I don't understand why this is happening..
Update
I really wonder how this is implemented? # http://www.proxywebsites.in/browse.php?u=Oi8vd3d3LnNuYXBkZWFsLmNvbQ%3D%3D&b=13&f=norefer

2 issues, pick any you like:
your server side proxy code contains bugs
plenty of sites have either explicit frame-break code or at least expect to be top level frame.

You can try one more thing. In your proxy script you are saving your webpage on your disk and then loading into iframe. I think instead of loading the page you saved on disk in iframe try to open that page in browser. All those sites that restirct their page to be loaded into iframe will now get opened without any error.
Try this I think it an work

My Proxy Server side code :-
DateFormat df = new SimpleDateFormat("ddMMyyyyHHmmss");
String dirName = df.format(new Date());
String dirPath = "C:/apache-tomcat-7.0.23/webapps/offlineWeb/" + dirName;
String serverName = "http://localhost:8080/offlineWeb/" + dirName;
boolean directoryCreated = new File(dirPath).mkdir();
if (!directoryCreated)
log.error("Error in creating directory");
String html = Jsoup.connect(url.toString()).get().html();
doc = Jsoup.parse(html, url);
links = doc.select("link");
scripts = doc.select("script");
images = doc.select("img");
for (Element element : links) {
String linkHref = element.attr("abs:href");
if (linkHref != "") {
element.attr("href", linkHref);
}
}
for (Element element : scripts) {
String scriptSrc = element.attr("abs:src");
if (scriptSrc != "") {
element.attr("src", scriptSrc);
}
}
for (Element element : images) {
String imgSrc = element.attr("abs:src");
if (imgSrc != "") {
element.attr("src", imgSrc);
log.info(imgSrc);
}
}
And Now i am just returning the path where i saved my html file
That's it about my server code

Get webpage and read throug it using javascript

Hi i have a quick question, say that you would like to connect to a website and search it for what links it contains, how do you do this with javascript?
I would like to do something like this
Var everythingAdiffrentPageContains = //Go to some link ex www.msn.se and store it in this variable
var pageLinks = []; var anchors = everythingAdiffrentPageContains.getElementsByTagName('a');
var numAnchors = anchors.length;
for(var i = 0; i < numAnchors; i++) {
pageLinks.push(anchors[i].href);
}
We can assume here that we have acces rights to the site so this is not of a concern.
In other words I would like to go to some site and store all that sites Hyperlinks in an array, how would you do this in javascript?
Thanks
EDIT since pointed out Im not trying to connect to another domain. Im trying to connect to another apache webserver inside my lan that hosts a website that I would like to scan for links.
Unfornuatley I do not have PHP on my webserver :/ But a simple javascript would do it
for example go to X:/folder/example.html
Read it, and store the links

Unfortunately - You can't do this. "We can assume here that we have acces rights to the site"...that's a false assumption from a JavaScript point of view, if the page is on another domain. You simply can't access content on another domain (not HTML content anyway) via JavaScript. It's prevented by the same-origin policy, in place for several security reasons.

I suggest you to use a JS framework that helps you to retrieve elements and do stuff with DOM easily.
For example using mootools you could achieve this writing some code like this:
var req = new Request.HTML({
url:'./retrieve.php?url=YOURURL', //create a server script to "retrieve" the html of another domain page
onSuccess: function(tree,DOMelements) {
var links = [];
DOMelements.getElements('a').each(function(element){
links.push(element.get('href'));
});
}
});
req.send();
The retrieve.php page should be written for example in this way:
<?php
$url = $_GET['url'];
header('Content-type: application/xml');
echo file_get_contents($url);
?>

Is robust javascript-only upload of file possible

I want a robust way to upload a file. That means that I want to be able to handle interruptions, error and pauses.
So my question is: Is something like the following possible using javascript only on the client.
If so I would like pointers to libraries, tutorials, books or implementations.
If not I would like an explanation to why it's not possible.
Scenario:
Open a large file
Split it into parts
For each part I would like to
Create checksum and append to data
Post data to server (the server would check if data uploaded correctly)
Check a web page on server to see if upload is ok
If yes upload next part if no retry
Assume all posts to server is accompanied by relevant meta data (sessionid and whatnot).

No. You can, through a certain amount of hackery, begin a file upload with AJAX, in which case you'll be able to tell when it's finished uploading. That's it.
JavaScript does not have any direct access to files on the visitor's computer for security reasons. The most you'll be able to see from within your script is the filename.

Firefox 3.5 adds support for DOM progress event monitoring of XMLHttpRequest transfers which allow you to keep track of at least upload status as well as completion and cancellation of uploads.
It's also possible to simulate progress tracking with iframes in clients that don't support this newer XMLHTTPRequest additions.
For an example of script that does just this, take a look at NoSWFUpload. I've been using it succesfully for about few months now.

It's possible in Firefox 3 to open a local file as chosen by a file upload field and read it into a JavaScript variable using the field's files array. That would allow you to do your own chunking, hashing and sending by AJAX.
There is some talk of getting something like this standardised by W3, but for the immediate future no other browser supports this.

Yes. Please look at the following file -
function Upload() {
var self = this;
this.btnUpload;
this.frmUpload;
this.inputFile;
this.divUploadArea;
this.upload = function(event, target) {
event.stopPropagation();
if (!$('.upload-button').length) {
return false;
}
if (!$('.form').length) {
return false;
}
self.btnUpload = target;
self.frmUpload = $(self.btnUpload).parents('form:first');
self.inputFile = $(self.btnUpload).prev('.upload-input');
self.divUploadArea = $(self.btnUpload).next('.uploaded-area');
var target = $(self.frmUpload).attr('target');
var action = $(self.frmUpload).attr('action');
$(self.frmUpload).attr('target', 'upload_target'); //change the form's target to the iframe's id
$(self.frmUpload).attr('action', '/trnUpload/upload'); //change the form's action to the upload iframe function page
$(self.frmUpload).parent("div").prepend(self.iframe);
$('#upload_target').load(function(event){
if (!$("#upload_target").contents().find('.upload-success:first').length) {
$('#upload_target').remove();
return false;
} else if($("#upload_target").contents().find('.upload-success:first') == 'false') {
$('#upload_target').remove();
return false;
}
var fid = $("#upload_target").contents().find('.fid:first').html();
var filename = $("#upload_target").contents().find('.filename:first').html();
var filetype = $("#upload_target").contents().find('.filetype:first').html();
var filesize = $("#upload_target").contents().find('.filesize:first').html();
$(self.frmUpload).attr('target', target); //change the form's target to the iframe's id
$(self.frmUpload).attr('action', action); //change the form's
$('#upload_target').remove();
self.insertUploadLink(fid, filename, filetype, filesize);
});
};
this.iframe = '' +
'false' +
'';
this.insertUploadLink = function (fid, filename, filetype, filesize) {
$('#upload-value').attr('value', fid);
}
}
$(document).ready(event) {
var myupload = new Upload();
myupload.upload(event, event.target);
}
With also using PHP's APC to query the status of how much of the file has been uploaded, you can do a progress bar with a periodical updater (I would use jQuery, which the above class requires also). You can use PHP to output both the periodical results, and the results of the upload in the iframe that is temporarily created.
This is hackish. You will need to spend a lot of time to get it to work. You will need admin access to whatever server you want to run it on so you can install APC. You will also need to setup the HTML form to correspond to the js Upload class. A reference on how to do this can be found here http://www.ultramegatech.com/blog/2008/12/creating-upload-progress-bar-php/