Develop Reference

get source code of whole website - which loads additional content after scrolling down

I want to fetch this site
https://www.film-fish.com/modern-mindless-action
to fetch the IMDB IDs of all movies listed there.
The problem is that the page loads all movies listed there just after scrolling down. So, a simple wget doesn't work.
Even if I scroll to the bottom of the page and view the source code, I do not see the last movie in the list (Hard Kill (2020)).
So the problem seems to be that the content is being created via JavaScript.
Has anybody a tip on how to achieve that?

So the problem seems to be that the content is being created via a js
script. Has anybody a tip on how to achieve that?
Indeed, executing JavaScript code is beyond scope of GNU Wget. You would need browser automation tool. If you know some Node.js or JavaScript I suggest taking look at PhantomJS Quick Start, Page Automation. Please take look at first example in 2nd link, you should be probably able to rework to your needs, i.e. instruct page to scroll down using JavaScript then extract what you need using JavaScript.

load external site and change its visualization

I'm trying to create a web page able to change a site visualization (.css or / and .js) in order to recreate the same live change capability offred by Firebug for Firefox or the Inspector of Chrome.
Here an image to better explain my task:
I have been able to visualize the other site inside my page using the iframe, but unfortunately it is not possible to change its visualization and access its elements due to the "same origin policy".
Is there a way to do this using the iframe or loading the external site inside another element?
Update:
Considering the answers the options should be:
create a php proxy page to load the target site and change visualization on it.
create a browser extention.
I've tried the first, even if it requires to install a web server (xampp), with a simple page calling the function file_get_contents('http://www.site.com');
The page is loaded but unfortunately missed some elements (like images) and it is only a static copy; it is not possible to go further in the site navigation.
Update 2:
Load the entire page via javascript could be the better solution (I don't konw how) if it is possible to live change the code but what about the possibility to interact with this "page copy" and transfer the interaction to the original one?
Scheme:
Explanation:
I've noticed Firebug extention can select and live edit any page element, even if they belong to the iframe which loads an external domain page.
What I'm looking for is a way to act like Firebug, get an element and change its style.
I'm trying to load the site into the iframe beacuse I wanted to create a toolbar above it to select my "visualization styles"; for example a button to makes titles bigger and red.
Anyway I'm open to any other methods suggestions.
Update 3:
I have found an extention for both FireFox and Chrome which is really close to my aim: "Stylish"
This add on allows to live change any site css proprerty and save it in order to reload them every time you'll visit the page.
Now my question is: How can I do the same creating a dedicated page to load and change visualization of a specific site?
FINAL EDIT:
In order to continue this question with a more relevant arguments I decided to ask a new one: create a php proxy page

No. Your solutions may be
to let your own site act as proxy so the same origin policy isn't triggered
to build an extension, which will be browser dependent (Firefox or Chrome) and which will require authorization and installation

I'm not sure if I understand what you want very well, but my feeling to ''trick'' this easier would probably to give very specific height and width to your first site (the iframe) and do a jQuery condition
If ($('body').width() == 500 && $('body').height() == 400 {
$('body').addClass('isiFrame');
}
Then, you only have to do your css .isiFrame .myCoolDivs {....}
You might have to use it on a document ready also, but that could be one way to trick it and since you're not doing it on resize (exepect if somebody's having his screen at this exact width and height at start)
The safer way would probably to create a master session using PHP but I cannot give you an example since it've been to long and echo the body class if the master_session or variable is equal to true
Hope it helped!

If you try to fight Same_origin_policy and try to fight it I am sure you won't get much success their.
Server Side
I would suggest you Handle this on server-side, grab the web-page and apply whatever styling and scripts you want, should be very easy!!
If you use Ruby on rails - Nokogiri gem can help you to parse html. And you can use standard library to 'get' a webpage.
Client Side
If you want to do this on client side, you need to write some jquery/javascript code, you can take following steps:
Get the webpage you want to display.
Grab the element's which include js/css files, remove them and your own.
Display the page in new Iframe present in your page.

jQuery+ AJAX + Help Hiding This Content

First of all: I hope the following question is not too generic.
I have a small problem and I cant think of a good solution and I was hoping some1 here is able to help me.
This is my situation:
I am using AJAX to dynamically load pages. My main site is index.php and once I click on a navigation link, the AJAX script replaces the content of index.php with new content and adds a hash tag to the URL. For instance:
I click on the link to about.php, the script adds #about.php to the URL and loads content from about.php into index.php. It works great :) However, there is a small issue that I would like to resolve:
Lets say we start by navigating to index.php#about.php directly - this means the content of index.php is visible for 2,3 seconds and than gets replaced with content from about.php. And I would like to avoid that.
I came up with a few ideas, but they are all not really great:
1) Hide content -> than make AJAX call -> on completed AJAX show content again
Downside: The content is still visible for a second.
2) Hide content with CSS and show it after AJAX call
Downside: This would work perfectly, but users without Javascript (and the GoogleBot) will see an exmpty page only.
3) Use an empty index.php and put the content of it in main.php and automatically load main.php via AJAX on page load.
Downside: Would work too, but again, users without JS and GoogleBot will just see an empty page when the visit index.php
Thats all I can think of and all three solutions are not good, because I am worried the SEO value will dramatically decrease when I have an empty index.php (I could accept that users with no JS get nothing to see).
p.s. I read somewhere that when you have display:none in an external css file and block it with robots.txt, GoogleBot wont know the difference, but I am worried thats maybe not the case? Any1 got some experience?
Edit: I guess my whole question comes down to this:
Do you think hiding the whole content of index.php with CSS (and than show it with JS), will be a huge no-go for SEO or will it be okay with GoogleBot (afterall the content is still in the source, but not visible to the user)?

If you used query strings instead of the hash you could have index.php load the correct content at the server level.
A plugin like history.js can help you push URLS to the browser so that you still get your ajax browsing.

Wow where to start...first of all the page 'blink' I'll call it is 2-3 seconds for you but it is completely dependent on the users computer, how fast it executes the javascript, and how fast the AJAX call returns so you could have a much larger delay.
Second I wouldn't worry about Googlebot seeing any of the ajax content. While it's true googlebot does try to fiddle with some javascript it won't make the ajax call like a normal browser would. I'd be very surprised if Googlebot ever saw any of your Ajax loaded data.
Googlebot does a fantastic job of figuring out what content is delivered via html/css to a user when they visit your page. It also figures out if something is displayed or not and does a good job of deciding if that content is just stuffing or is something that really matters.
You're worried about what someone without javascript will see when the entirety of navigating your site is based in javascript. This doesn't seem to reconcile.
You've got PHP available. My suggestion is to forgo the AJAX stuff you're trying to do and do it in PHP. You can just as easily script the same behavior in PHP as you can in AJAX.
SEO NOTES:
If you're looking for solid SEO results I suggest making the static (non-javascript version) page as SEO friendly as possible. I like to 'pick the low hanging fruit' like making sure the page has one and only on H1 and that it has the most important keywords in it. seomoz is one of the best sites I've found for seo information.

Showing a demo of my CSS on any website

I have developed a small component which can be put in to any website. Now, I want to develop a code that could demonstrate how would my component look like on any website.
So, the person would come to my page and put in his URL and then my code should embed my custom JS/CSS in to the downloaded HTML and display it. Something like this.
Here, like the feedback tab, I want to show my component any where on that page.

Try a bookmarklet.
Create a piece of javascript that adds your code into the page such as the following:
javascript:(function(){var%20script=document.createElement('script');script.src='http://www.example.org/js/example.js';document.getElementsByTagName('head')[0].appendChild(script);})()
Add it as the href of a link like so:
Link Text Here
Tell your users to drag the link to their bookmark toolbar and click on it on different websites to try your code out.
Some examples: http://www.reclaimprivacy.org/, http://www.readability.com/bookmarklets

In the example you linked, they are requesting the page specified in the url querystring parameter on the server, and then doing more or less the following steps:
In the <head> tag they are adding a <base href="url" /> tag to the document. The base tag will make any relative links in the document treat the value in the href attribute as their root. This is how they are getting around broken css / images. (The base tag is supported by all browsers)
At the end of the document (IE the </body> tag) they are injecting the javascript that runs their demos.
They serve the modified HTML requested to the browser.
All of this is pretty straight forward in implementation. You could use regular expressions to match the <head> and </body> tags for steps 1 and 2 respectively. Depending on the server platform how you actually request the page will vary, but here are some links to get you started:
C# - HttpWebRequest object documentation
PHP - HttpRequest::send

Nathan's answer is the closest to how we have done the demo feature at WebEngage. To make such a demo functional, you'll need to create a Javascript widget that can be embedded on third party sites. syserr0r's answer on creating a bookmarklet is the simplest approach to do so. Our's is a JAVA backend and we use HttpClient to fetch the responses. As Nathan suggested, we parse the response, sanitize it and add our widget Javascript to the response. The widget JS code takes it on from there to render the Feedback tab and load a demo short survey.
Disclosure: I am a co-founder and ceo at WebEngage.

You can not do this with JQuery due to cross site scripting restrictions.
I suggest you write a PHP script that downloads the URL specified by the user and includes your widget code and then echo it back to the user.

I recommend using bookmarklets. I've made a bookmarklet generator for adding jQuery-enabled bookmarklets to a page to make development easier.
There's a caliper bookmarklet on the page that you can mess around with just to show an example of it working.
Full disclosure, this is something I've made, I'm not trying to be spammy as I think it's relevant: zbooks

You could make an iframe page, which loads their page in the iframe, and uses javascript to inject your code into the iframe.

Here is my approach...
http://jsfiddle.net/L2kEf/
html
<iframe src="http://www.bing.com"></iframe>
<div>I am div</div>
css
div { background: red; position: absolute; top: 20px; width: 100px; left:20px;}
iframe{width: 100%; height: 500px;}
you can add javascript/jquery too, so you could do something like,
jQuery //not 100% sure it would work coz of cross browser thingy, but you know, worth a try.
$('div').click(function (){
$('iframe').contents().html('changed');///
});
if this can't change any of the contents, you can display a dialog, to say it would normally work if it was in your website, then use #syserr0r approach for bookmarked users, for better results, since you are offering this kinda services, to developers, im sure they would know about bookmarking, my approach would be rarely used :) so hope it helps.

I had a problem of a similiar nature, and the main obstacle is the cross-domain policy.
You have to ask the user to put your code in a <script src="..."> or create a proxy solution that would add your code for them.
I went for the proxy and here are my observations:
it's easy to create a basic proxy in php - there are some php proxies on sourceforge and Ben Alman has created a simple php proxy for AJAX. Based on those I was able to create a php proxy altering the content properly in one day.
after that I spent a lot of time making it work with more and more sites with issues. You can never create a perfect proxy.
As an alternative (sa long as you are non-commercial) you can use http://www.jmarshall.com/tools/cgiproxy/ and put the site in an iframe and then do whatever you want to do with the iframes document, as it's in your domain thanks to the proxy. You can access iframeDOMnode.contentWindow.document then, etc.

You can create a Crossrider extension which your users can download.
Then simply add this to your App/Extension code:
appAPI.dom.addRemoteJS("http://yourdomain.com/file.js")
Your users can then download the extension (it works cross-browser for Internet Explorer, Chrome and Firefox) and it will load your JS code on every page load.

You can get an approximation of what it will look like using a iframe. Take a look at that link for an example.
http://jsfiddle.net/jzaun/5PjRy/
The issue with this appoch is that you can't move your DIV(s) when the page scrolls, they are in effect just floating over the iframe. There is no way around this as cross-domain scripting wont let you access the iframe's document to monitor scroll events.
The only other option you have for a better fitting example would be to load the page from the server side in whatever scripting language you are using and load that into the iframe (or into a div, etc.) and you can use javascript all you want as the page is coming from your domain.
For your example of what will your widget look like I imagine floating your DIV(s) over an iframe would give enough of an idea.
Please note the example you gave is using the server side method, not the iframe method.

I agree with the bookmarklet strategy.
I'm a fan of http://bookmarklets.heroku.com/, which lets you generate bookmarklets easily, inject jQuery, etc.

Jquery / Javascript — How to optimise my site load visually?

I'm making finishing touches on my site and am struggling to make the load of the page look less jumpy. best way to show you what i mean is to show the site:
http://marckremers.com/2011 (Still not complete, in alpha phase)
As you can see the contents sticks to the left, loads a ton of jquery and images into the page, and only then click into place (centre).
I'm wondering if there is a way i can make it click into place first and then load the elements?
I tried putting the reposition script just after , not even that works.
Any ideas? Thanks

With all of the images you have, your page is 1.5mb, coupled with 70 http requests. No wonder your site behaves the way it does.
You should be using sprites on the smaller images to reduce http requests and as far as the large images go, you are loading all of the pictures at once. Even the ones that aren't displayed right away. The images that aren't displayed right away should be pulled in via AJAX after the page loads.
If you want to go further into optimization I would also:
Use one external javascript file. Yes
it increases size, but I favor that
over http requests.
Minify your html/javascript/css.
Don't host jQuery on your site, use a CDN such as Google APIS.
Check out a service similiar to Amazon S3.
I could reinvent the web site best practices wheel here, or I could send you to Yahoo best practices for web site optimization There is a ton of very important information there, read it and reference it.

You loaded jQuery twice, once from your own site and another time from Google's CDN. For starters, you should probably move all the JavaScript to the bottom of your HTML. Then you need to optimize your if ... else that handles how many columns to display and your Google Maps iframe.
To speed the visual up, instead of using jQuery, you should probably have some vanilla DOM scripting that dynamically creates some CSS styles for the projects and tb_tweets classes, so it doesn't have to wait for all your JavaScript to load before handling resizing of your projects and tb_tweets.

use http://mir.aculo.us/dom-monster/ and do everything it tells you to do. If you want tools to figure out what is going on during page load, the chrome developer tools are hands down the best out there for client side optimization.

A think you could do is put your javascript functions in the document.ready(function()), this way the functions will be loaded AFTER the page is loaded. I guess you don't need the functions for loading the site, just to interact with it?

Generally you only want to trigger your events after the page has rendered, i.e., using
$(document).ready(function()) {
//your javascript goes here
}
Then, in your HTML you have placeholders so the page doesn't "expand" or "jump" as you put, with some kind of indication that the element is still loading. Then, when your ajax requests complete, simply populate the placeholders with the ajax response.