Develop Reference

get source code of whole website - which loads additional content after scrolling down

I want to fetch this site
https://www.film-fish.com/modern-mindless-action
to fetch the IMDB IDs of all movies listed there.
The problem is that the page loads all movies listed there just after scrolling down. So, a simple wget doesn't work.
Even if I scroll to the bottom of the page and view the source code, I do not see the last movie in the list (Hard Kill (2020)).
So the problem seems to be that the content is being created via JavaScript.
Has anybody a tip on how to achieve that?

So the problem seems to be that the content is being created via a js
script. Has anybody a tip on how to achieve that?
Indeed, executing JavaScript code is beyond scope of GNU Wget. You would need browser automation tool. If you know some Node.js or JavaScript I suggest taking look at PhantomJS Quick Start, Page Automation. Please take look at first example in 2nd link, you should be probably able to rework to your needs, i.e. instruct page to scroll down using JavaScript then extract what you need using JavaScript.

How can I break a third party iframe on my site, with code?

I am not a programmer.
Someone has scraped my site home page source code and placed their iframe over it, so that when the page is fetched it displays their content.
The iframe is not immediately apparent but it's there, just well hidden. These sites are all hosted on hacked servers running WordPress. They still display our site links and architecture that is being delivered by our server. There are currently over 160 such sites built using the same method.
I believe that they have disabled js so that may not be an option.
I know that we can break out of an iframe if it's our site in the frame.
Is there any way, either on the server side or on the page to break their iframe and force our page to the top?
If we can break it, then our code becomes worthless and with a bit of luck they may stop using it.
Update:
Just wanted to add a few points to anyone who has any ideas.
1, They already have the code, only things being served are the images and CSS files because they have only left those links in the page.
2, They are showing their site by floating it with a z-index on top of everything, which is why when you view src you see the site above and not the site that is floated in the iframe.
3, The iframe is visible if you inspect element with firefox and scroll to the top of the page you can see the iframe they are using.

Based upon the additions (currently in an answer), since they have your code there's not much you can do about breaking out of the iframe.
Depending upon your server environment you could try determining what page is requesting your images and CSS, and then display modified versions to those accessing the scraped versions. The key word for your searches is 'hotlinking.'
Possible modifications could include not serving the assets (images/CSS), or returning a CSS file that just does a display:none; on HTML elements to hide.
It might be a fool's errand, but trying to contact the hosts of the hacked servers might be a good idea, but I can't honestly say that it will get you very far, and might be a waste of time for the majority of them.

method to include external url into existing page with overflow

Here is my situation. I have a new navigation menu built on an entirely different php framework than the actual site is running. Both the new navigation and the existing site are on the same domain. I tried removing the old bar and adding the new bar via an iframe:
<iframe src="/new/menu" width="100%" scrolling="no"></iframe>
This works perfectly fine except some of the menu elements have submenus. When the menu drops down, it's stuck inside the iframe and doesn't overflow the parent page.
I've looked into it and it makes sense this is a security issue with an iframe. But is there some other trick that would allow this external url to load AND overflow it's parent page?
Also, just so it's out there, this is a temporary visual upgrade the client is pushing me for. They want me to "splice" elements into place as they're completed so I definitely need to find some type of solution to make this work properly.
Any ideas will help!!

Some things you could do is First of all a php call something along the lines
$menu= file_get_contents('http://www.menuurl.com/');
echo $menu;
Second thing you could do is a javascript call preferable with jquery
$('#divToLoadInto').load('file.html');
Source:
http://api.jquery.com/load/
http://php.net/manual/en/function.file-get-contents.php
Here is some alternative solution:
- A Fully javascript solution using ajax

Need an alternative to HTML frames (I now know why they're evil)

Just to summarize, I'm trying to make a live page render (browse websites) while having an overlaying menu for functionality.
I may have made a mistake in how I designed my application and need an alternative to frames. Any advice is appreciated on this.
So the goal of the application is analyzing web content. Its got a global menu with the usual stuff: profile, sign out, settings. A global footer with the standard legal stuff and in the middle is where all the functionality comes in.
I knew that one of the functionality was a live render of the site with a menu overlaying at the bottom. With it, they could view elements and write up comments on what was there.
Each function was embedded inside a frame so if i clicked the page view button, the frame would shoot off to the frame page and relay it. Heres the problem. In order to get a live render of a site and using it this way, I had to put a frame inside a frame and thus duplicate scroll bars were born.
If anyones confused as to how the layout works there are 2 shots here.
http://s1180.photobucket.com/albums/x406/0vertone/?action=view&current=pageviewlayout.jpg
And a cleaner designer model (mainly just global menu)
http://s1180.photobucket.com/albums/x406/0vertone/?action=view&current=globalview.jpg
I've got the coding it and for the overlaying menus, I tried both an accordion and a css style thing that slides. You can take a look at the code here.
GLOBAL MENU:
save this as global.html.erb
http://paste.ubuntu.com/648135/
PAGE VIEW:
save this as pageview.html.erb
http://pastebin.ubuntu.com/648146/
Once you've done this, it should work as you'd expect.
The technology or way its done is option but the design is not.
Just looking now, on the page view at the bottom of live website but above the footer. There needs to be an overlaying menu that slides in an out. If anyone has any ideas on how to implement a live view of a site and an overlaying menu, I'm all ears.
Thank you
Quick Summary:
What I'm trying to achieve is an application that has a global menu that stays consistent across all pages. In the middle of this global menu is a frame that goes to other pages that I've created, e.g list view, statistics and page view.
Page view is the one I'm working on at the moment. In the page view there are 2 components,
Live view of website (whatever page they want) and an overlaying menu at the bottom. Think like youtubes playlist on bottom of page. The problem is that in order to put this live render of a website in the frame, you have an iframe within an iframe and I;m trying to resolve that.
But rather than treat it as a bug, I'm looking for alternatives to how I can achieve the above
My final solution available on pastebin:
page view:
http://paste.ubuntu.com/648246/
Global view:
http://paste.ubuntu.com/648250/
Basically this
jQuery('#iframeWrapper').height('61%');

This is a quick demo of CSS and divs - you might be able to do what you want using this method.
<style type="text/css">
#holder {position:absolute;left:0;top:0;width:425px;height:350px;z-index:1;}
#menu {position:absolute; left:0;top:300px;height:50px;background-color:#f0f;width:425px;z-index:10;}
</style>
<div id="holder">
<iframe width="425" height="350" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" src="http://maps.google.co.uk/maps?f=q&source=s_q&hl=en&geocode=&q=firebox&aq=&sll=53.800651,-4.064941&sspn=17.72791,44.736328&ie=UTF8&hq=firebox&hnear=&z=5&iwloc=A&cid=2836054255231781537&ll=51.386465,-0.138487&output=embed"></iframe><br /><small>View Larger Map</small>
</div>
<div id="menu">Nav / whatever goes here</div>
Also there is he option of using ajax to call content into the #holder div rather than using an iFrame too

Loading external content with jquery or iframe?

Hiho,
There's an existing website that i need to include into another site which goes like this:
a.mysite.com
and i need to fetch content from this site in my
www.mysite.com
website...
As i need to access the content of the iframe the Same origin policy produces a problem here.
What i did was to configure mod_proxy on Apache to proxy pass all requests from
www.mysite.com/a
to
a.mysite.com
This will work fine...but my problem is that im not sure what the best way would be to include those pages.
1. Idea
As the content of the iframe is a full featured site with a top navigation...left navigation etc....i would need to change the page template to only show the content box to be able to integrate that page in the iframe.
2. Idea
I could just load the DIV where the content lies through JQuery.load() and integrate it into my site.
What is the best way to accomplish such a task? How bad is both ideas from the SEO point of view?

Unless it involves significant rework, the best solution is to combine the two into a single HTML page on the server side (using server-side includes).
Advantages:
No problems with SEO as it's delivered as a single page. Content in iFrames and content loaded via AJAX (with an associated link in the HTML) are traversed, but only the link, not the content itself is associated with the main page. See: http://www.straightupsearch.com/search-marketing/best-practices/seo_iframes_a_g/
Faster page load - either of your suggestions will cause the main page to be loaded first before the other content is loaded.
No reliance on Javascript - your second method will fail completely if javascript is not supported / turned on.
Include all JS and CSS only once - your first method will require these to be duplicated in the <head> of each page. This is more of a long term advantage if you wish to achieve full integration of site "a". However, it can be a disadvantage initially, see below.
Disadvantage:
May cause conflicts with scripts and CSS between the two pages. However, this same problem exists with your second method.
If you must choose between either of the two options you proposed, I would not select the second as others have suggested. Significant amounts of static content should never be loaded via Ajax, and in this scenario gives you no additional benefits. At least iFrames guarantee no JS and CSS conflicts.

Use the 2nd approach (jQuery.load) and if you're working with HTML5, for browsers that support the History API you can change the URL to whatever the content is for that div.
Check out https://github.com/blog/760-the-tree-slider for an example of how github did it for their tree slider.
EDIT:
I am not sure how using an iFrame whose src points to your own domain affects search rankings but at best it's a grey area. I would assume that possibly some pagerank would trickle from the parent to the child but I have no clue how it would work for instance if a blogger linked to your page with the iframe that pointed to another page. This would be a pretty good question to ask at the Webmaster Help Forum

Always say no to iframes. jQuery+Ajax all the way.