This is going to sound pretty generic but I don't know how else to ask this question.
I am using Selenium Webdriver (on python) to automate the download of some images. I can access a list of links to those images after a search. The full list of links is of known length, but gets loaded after successive scroll downs.
For example, after launching the search, I may know there are 210 results, but only 20 of them get loaded. Scrolling down multiple times will load the other ones. Workflow example:
Insert search details, click search button
Read total number of elements
Scroll down until the number of displayed elements is the same as the total
Download stuff
Close download page, go back to point 3.
The scroll down is really painful because the image download is on a different page, and closing the new page will reload the original search results without all the scroll-downs. Meaning I need to scroll again.
In order to speed up this process, I tried using PhantomJS and upping the vertical resolution of the page. However, this does NOT load all elements and the process fails, probably because the vertical resolution is enough to load all 20 elements and cannot trigger the scroll-function to load the others.
So I'm guessing there is some function that gets triggered when scrolling down. But I can't find it in the page source.
What I know is that it loads a <div id="loader"> that I see every time new items are getting fetched (and every time the page hangs because of connection problems).
My question n.1 is whether calling the function directly without scrolling, (i.e. d.execute_script("some_magic_function") would save some time compared to d.execute_script("window.scrollTo(0, 1000);")). Perhaps not, so I should just use the actual scrolling behavior as I'm doing now.
But if there could be an advantage, then my question n.2 is how I can find the function that triggered the loading of new elements (and of the <div id="loader">). I've tried looking for onscroll or just scroll but got nowhere.
Related
Question: With Javascript, can you tell the web browser to begin loading a page, but don't begin rendering it yet?
Issue: A client wants his web page to show listings like a book. When you click on the next button, he doesn't want the next page to immediately load. He wants the book to close (a closing animation) and then load the next page.
Current status: All links go to Javascript. I show the closing animation. Then, I replace the window location. The issue is that there is a clear wait for the next page to load. It would be nice if I could load the following page into cache while the closing animation runs. In other words, I want to make that three-second animation useful time by loading all the HTML, CSS, Javascript, and images for the following page and then all that happens when I set the new location is that it renders.
Possible solution: I have the main page that had two full-screen iframes in it. One iframe is the current page. The other is hidden and is used to load the next page. After the animation, I flip which iframs is visible and which isn't. This is good except that the back button doesn't work properly. If you click back, you go to wherever you were before you went to the website. You don't hide the current iframe and show the one you just hid. If you click back twice, flipping iframes doesn't work. I have to keep a log of your history. Further, I have to hack the back button, which I don't like. So, I'd like to use a built-in cache method if I can.
Possible solution: I have the main page that had two full-screen iframes in it. One iframe is the current page. The other is hidden and
is used to load the next page. After the animation, I flip which iframes is visible and which isn't.
Yes, this sounds like a good approach.
This is good except that the back button doesn't work properly. [...]
I have to hack the back button, which I don't like.
Single Page Applications (SPAs) can't use the back button as originally intended because the entire application exists within a single document.
Partly in response to this, we have
history.pushState()
which is a really good extension to the History API, enabling new "artificial" entries (describing new states) to be added to the browser's history, which, in turn, enables the back button to work exactly as the user might expect it to.
Further Reading:
http://html5doctor.com/history-api/
https://developer.mozilla.org/en-US/docs/Web/API/History/pushState
I have seen a method on Youtube and various other sites, which, upon changing to another page on the same server make the browser not directly redirect the user to the new page, but
stays on the same page until the new one is loaded
dynamically loads the new pages content seemingly without any idle time inbetween page changes
shows a progess bar on the top of the screen
leaves any html headers or other fixed content unchanged
In this gif you can see the animation on top of the page, upon changing the page, there is a progress bar and the new page is displayed seamless.
Here is where I am a little helpless, my attempts of finding something useful in this manner brought me practically nowhere, i do not know if there is a library/framework for this use that i simply cannot find or there is some messing around with dynamical page loading i do not know about.
How is such an effect achieved and what techonolgies are requiered?
You need a single page application framework. For example look at AngularJS
If you want, you can do it with pure JS code using AJAX.
I've created a framework in javascript for my personal single page applications, and I'm trying to add the stacking page feature to it.
if you have ever worked with android, you have seen that every new activity or page that is created in android, pushes the previous one back in a stack that can be accessed once the user presses the back button, just like the history in a browser.
the only difference is that in android the page that has been pushed to stack, stays there until the OS runs out of ram or something, but in browser the page gets destroyed no matter what.
now, in my framework, I have come up with a very simple soloution to keep pages in memory so that when user presses the back button, instead of loading data from server, I just find the page in the stack and if it exists, I just bring it to the front and push the current page back to the stack.
but there are some issues with this method:
if you keep the stacked page in a javascript variable or you keep it in the DOM and just make the display: none; style applied to it, bringing it to the front and make it visible requires painting the whole page again and if the page is too large, this means a lot of process and makes the transition between pages extremely slow, specially if you have some animation going on while page transition.
and if the stacked page is visible, and just pushed back using z-index, then the scrolling will cause problem, as the scroll for the previous pages are still available. and if you make the pages overflow: hidden; then still if you stack multiple pages, the scrolling becomes very slow and choppy because the browser has to scroll over multiple layers of painted pages.
and so my question is, is it even a good idea to stack pages with these problems?
or is it just not meant to happen in browsers yet?
Depending on your SPA I don't think it is a good idea to keep all pages in the DOM and hide them with display: none; or a different z-index. What you could try to do is to keep the state / the information of the page in a variable and remove all related elements from the DOM. When the user is going back one page you recreated the elements with the information you previously stored in a variable. This way you prevent to load information from the server and you don't have any problems with going back to any state.
You might be able to optimize it even further by keeping the last page in the DOM and simply hide it and keep the state of the pages before in a variable.
I am working on a project where the client wants a way of transitioning between content that basically works like page turning on an e-reader app. When you click on a link to go forward, the current content slides to the left and new content is revealed as it slides. If you click on a link to go backwards, the content slides in from the left and is superimposed. (If you're jumping to a page further off it's fine for the page to reload.)
There needs to be a distinct URL for each content block, and ideally this should work all the way down to IE7. Assume there are at least 50 pages, each with at least 2-300 words.
I know there are lots of jQuery page transition options, but most of the ones that I've looked at slide in the new content while the old content is sliding off or fades in the new content after the old content is gone (think slide.js). What I need is basically curtain.js that is vertical, triggered by a link instead of scrolling, and doesn't need to load in all of the content on the page at once.
Here's one way I've come up with possibly building this:
Current content is loaded in from the database (or whatever)
Content for the previous and next pages are also loaded in and stored in hidden divs
When a link is clicked, the current page slides off (or the previous page slides in)
The content that's no longer needed is deleted
New content is "preloaded" with AJAX and hidden
Local URL also changes with AJAX
Here's a crude diagram
That seems really inelegant, though. Is what I've outlined above possible to do? What would be a better way of doing this?
I am okay with JS/jQuery and PHP, learning AJAX.
I'd suggest jQuery UI tabs
No need to deal with AJAX. Just get the server to spit out the 50 pages of texts once and that's a wrap.
It's pretty straightforward to hook into the API, which gives you more control over the entire procedure. Enough control to hook up a function that updates the address bar when you tab through.
I'm trying to isolate the Facebook JS/Ajax event that fires when more content is loaded.
This is - when you've been on the page for a certain amount of time and new posts show up. Or when you scroll to the bottom of the page and more posts are loaded.
I've tried going through every aspect of Chrome's developer tools to pinpoint that event but I'm just not getting anywhere. Facebook is very tricky in the way it names everything so that it's not human friendly.
The closest I'm getting is finding an event listener on line 20 of a file called PBlKb9_thnY.js (which may be named dynamically so be different for others), but I can't get any further than this.
Basically I want to identify the function that loads more content and then attach my own Javascript code to the end of it through an extension.