This question already has an answer here:
Scroll down to bottom of infinite page with PhantomJS in Python
(1 answer)
Closed 5 years ago.
I am trying to scrape a (very long) webpage for some content.
I am using Python's BeatifulSoup library, but am not limited to Python. The issue I am facing is that the page only loads a small subset of the content, and, as soon as the user scrolls to the top/bottom of the page, more content is loaded.
There is a javascript function that is called when the scroll-bar hits the top/bottom, triggering the server to load more data. Is there a way for me to programmatically scroll through the page, or call the function that gets the page to load more data, so I have all the content available?
I know I could just use VBScript to control the window and scroll, but ideally I am looking for a more elegant solution that allows to me to extract the data and scroll the page at the same time.
You can check following answer OR you can check the net panel and call the same url or method which gets called when page hits the bottom of the page
Related
I'm using Selenium to scrape a webpage. Initially, when you navigate to the web page, the page contains about 10 items. As you scroll to bottom of the page, the page continuously loads more items (via AJAX / Javascript) until page has been fully loaded. Once the page is fully loaded I scrape data from each item in list.
Occasionally, I need to delete an item from the page. This causes the page to completely reload/refresh to it's original state with the initial 10 items - forcing me to repeatedly scroll to end of page which is very time consuming. In other words, if the page has a total of 500 items and I need to delete 50 item at various (and unknown) locations through the list I will need to reload entire list of approx. 500 item 50 times.
Since the page is being reloaded via ajax / js I thought I could solve this problem by disabling Javscript on this specific page. I was able to accomplish this by following this answer on stackoverflow. However, it is a catch 22 situation - since once I disable javascript the page will no longer load new items when I scroll to the bottom. I need to first load all items on page via Javascript and then disable js so the page does not automatically refresh itself when an item is deleted.
Initially, I thought I could scroll to end of page (while have js is enabled) and then disable js once I hit the bottom of page so that as I loop through items on page and delete item, page will not be forced to refresh. However, based on my testing I found that once js is enabled before page is loaded it can no longer be disabled before you navigate to a different page.
My question is: Is there a way for me to temporarily block (or disable) Javascript execution after a page has loaded so that the page is not refreshed (yet still allow me to scroll to end of page via js)?
I have seen a method on Youtube and various other sites, which, upon changing to another page on the same server make the browser not directly redirect the user to the new page, but
stays on the same page until the new one is loaded
dynamically loads the new pages content seemingly without any idle time inbetween page changes
shows a progess bar on the top of the screen
leaves any html headers or other fixed content unchanged
In this gif you can see the animation on top of the page, upon changing the page, there is a progress bar and the new page is displayed seamless.
Here is where I am a little helpless, my attempts of finding something useful in this manner brought me practically nowhere, i do not know if there is a library/framework for this use that i simply cannot find or there is some messing around with dynamical page loading i do not know about.
How is such an effect achieved and what techonolgies are requiered?
You need a single page application framework. For example look at AngularJS
If you want, you can do it with pure JS code using AJAX.
This question already has answers here:
Stopping a iframe from loading a page using javascript
(6 answers)
Closed 7 years ago.
Is there a way in javascript to simulate pressing the "stop" button the browser whenever a link or button is clicked?
I'm trying to load other websites while displaying a frame at the top of the page, the issue is that sometimes it takes a while to load all the content on a page (especially if there is something being streamed). Is there anyway to make it so I can place a "stop" button or link on my top frame that ceases the loading of anything else?
I checked around but couldn't find anything
Thank you!
You can use the window.stop() method, however note that depending on the location of your script, JavaScript is not executed until after the DOM (typical practice), so often times the page will already be loaded before this executes.
For an iframe, simply target the frame and use the stop method.
This question already has answers here:
Javascript back button for iframe parent window
(2 answers)
Closed 9 years ago.
I want to embed some third party content which will be an html page, inside my html website. I thought of using the iframe approach in which i will embed the third party html page inside my website as an iframe.
The motivation behind using iframes was that i dont know what libraries they might use, or what CSS they might write. I wanted their logic to be separate from mine. Iframe fulfills my this wish.
But now i have a problem, if i want the browser Back/Next Functionality for any navigation's inside the iframe, it will be very difficult to maintain because of the route. At the same time, maintaining the iframe state when the parent page is refreshed will be pretty tough.
So, is the iframe approach fine first of all? If it is, then how do i cater to the Browser Back/Next?
Well, you can manage back/forward button with JavaScript like this:
iframe.contentWindow.history.back();
iframe.contentWindow.history.forward();
Bigger problem is that included page can disallow possibility to be included into iframe (if you include page with this behaviour into iframe, it will load over your page - check for example this qeustion).
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Modify the URL without reloading the page
I just came across this site, http://www.so-design.co.uk/#/brand/1 , if you use the slider it changes the image just like any normal image slider but it also changes the url to /brand/1, /brand/2 ect which i suppose means that all the slides and their content are crawled by search engines as individual pages
any ideas how they are doing this ?
cheers
You might want to take a look at manipulating the browser's history as well as the history property. Also, reading the hash usingwindow.location.hash
First thing to do when you want to try and work out how a website has been developed is to view the source code.
The quickest way to do this is to right-click the page and look for "View page source" or something similar (depending on your browser)
Of course, in this instance, when you right-click it should become obvious that the website uses Adobe Flash. Therefore, this is not a typical JavaScript slider as you first seemed to assume.