I'm making a Django page that has a sidebar with some info that is loaded from external websites(e.g. bus arrival times).
I'm new to web development and I recognize this as a bottleneck. As it is, the page hangs for a fraction of a second as it loads the data from the other sites. It doesn't display anything until it gets this info because it runs python scripts to get the data before baking it into the html.
Ideally, it would display the majority of the page loaded directly off my web server and then have a little "loading" gif or something until it actually manages to grab the data before displaying that.
How can I achieve this? I presume javascript will be useful? How can I get it to integrate with my existing poller scripts?
You probably don't need up-to-the-second information, so have another process load the data into a cache, and have your website read it from the local cache.
The easiest but not most beautiful way to integrate something like this would be with iframes. Just make iframes for the secondary stuff, and they will load themselves in due time. No javascript required.
Related
2 years back I shelved a working Ruby web scraper that automatically download TV movie listings for a 1 week period at a time. Started to work on it again today and found that Ruby can neither access the controls or data of the web page being displayed.
Debugging shows that scripts are now generating the documents that load subsequent web pages. Also the initial scripts modify the current document when run (remove links). Any url used results in the same initial web page being loaded.
I am looking for suggestions on how to proceed to access the data in displayed web pages. I am not very knowledgeable about javascript but would pursue if I had a definite plan to follow. I believe I found the href to load the second web page but it only loads the initial page again so other mechanisms are in play (ie there are cookies mentioned in the script).
To download the information there are minimum of 28 web pages required and normally with downloading movie information there are several hundred web pages processed.
As you've discovered you can't scrape webpages with dynamic content with simple HTTP requests. You need to simulate the webpage actually being used in order for the Javascript to run and generate the content you need. This tutorial will probably help you do what you're trying to accomplish.
I am using python webkit.WebView and gtk to crawl a web page. However, the web page is kind of dynamically loaded by javascript.
The WebView "load-finished" event is not sufficient to handle this. Is there any indicator/event to let me know that the page is really fully loaded even the content produced by javascript?
Thanks,
There is no real way to determine if that page is fully loaded.
One method is to determine the amount of time since the last request. However, some pages will make repeated requests continually. This is common with tracking scripts and some ad scripts.
What I would do is use a set amount of time after the web view has said it finished loading... 5 seconds or so. It isn't perfect, but is the best you got, as there is no way to determine what "fully loaded" is for an arbitrary page.
Is its possible to have a javascript file that is aware of two different HTML files? And how would I do this?
I would like to be able to have two pages. index.html and pictures.html. I have an index.js that changes the display properties of index.html (it puts data based on people into tables and makes it look nice). I would like this current index.js file also to be able to edit the pictures.html file and change information there. index.html would link to pictures.html to display pictures of a person (based on the persons name I have them saved smith1.jpg, smith2.jpg, reagan2.jpg, ect). Is there anyway that this javascript file could get DOM elements based on their id or class of the second file (pictures.html) even though it "lives in" index.html? When i say lives in it is called at the top of the index.html page.
thanks
A script can access elements on another page if it was loaded in some way of connection.
For example, if you make a popup using var popup = window.open(), the return value will contain a reference to the opened popup and this allows access to elements within the popup. E.g. popup.document.getElementById('something'). Pages loaded within frames, iframes and such have similar ways of access.
So yes, if your page loads the second page its script can work there as well. I suggest avoiding this beyond opening and closing popups from a script though; a script should stay inside the box of its page and if it needs to do larger operations on another page, that usually means that you need to change your code architecture a bit.
You'll need to explore server-side programming to accomplish your goal.
http://en.wikipedia.org/wiki/Server-side_scripting
...Or you could write a client-side application in which "pages" are separate views of one actual page or are generated from backing data structures. If you want persistance of what is created/edited, you'll still need server-side programming.
You can use the html5 (group of technologies) postmessage api as well.. This allows you to send messages to another page, and in that page you define an event handler that knows how to handle the message.
This also works across domains.
Here is a blog with an example I just randomly found via google:
http://robertnyman.com/2010/03/18/postmessage-in-html5-to-send-messages-between-windows-and-iframes/
Not possible on the client side if editing the actual HTML file is your goal. If getting pictures to show up depending on stuff a user does on another page is all you care about then there are lots of options.
You can pass small sets of data like stuff the user entered into tables via cookies for accessing the right sets of image files in a pre-established scheme. This would actually persist until a user cleared out cookies.
You could wrap both pages in same-domain iframe elements with the parent element containing just the JS. This would allow you to persist data between pages and react to iframe load events but like everything in client-side JS, it's all gone when you reload the page.
Newer browsers have working file access objects that aren't total security nightmares. These are new and non-standard enough that it would take some doing to make it work for multiple browsers. This could be used to save files containing info that the user would probably have to be prompted to upload when they return to the site.
If the data's not sensitive you could get creative and use another service to stash collections of data. Use a twitter API to tweet data to some publicly visible page of a twitter account (check the Terms of Service if you're doing anything more than an isolated class project here). Then do an Ajax get request on whatever URL it's publicly visible at and parse the HTML for your twitter data.
Other stuff I'd look into: dataURIs, html5 local storage.
Note: None of these are approaches I would seriously consider for a professional site where the data was expected to be persistent or in any way secure regardless of where a user accesses it from.
This is a best practice type of question. I am developing a complete AJAX application. The user navigates to the main page of the application and everything from there on out is loaded via AJAX into the content section of the main page. Is it better to take all the javascript files I have and merge them into one file that is loaded on the main page or to split them up into just what is needed for each page that is loaded?
Putting it all in one file obviously has the benefit that only one HTTP request is made to load the javascript needed for the site and any request for a page there after will only need to fetch the HTML. But, this requires that every event that is wired up (using jQuery) be attached to the document using the live or on function. So everything will look like:
$(document).on('click', '#SomeButton', function () { });
Doing it this way will cause there to be many hundreds and possibly over a thousand events being tied to a single element, the document.
Putting them in separate files requires multiple HTTP requests to be made to load the various pages of the site but limits the number of events that are attached to the document.
Any thoughts on what is best here?
I would vote for separate js files for each page for bigger projects specially if your project is using any js library like jQuery and its plugins like grid plugin etc. In case you have a big single javascript file your first page will load slowly obviously giving your user a bad first impression. What we do is that we create separate js files for each page specially when there are ajax calls to load data for the pages. Plus there are separate files for each pluggable component like custome drop down or date counter etc. This way its easy to manage the code and customize it later.
Before deploying the app we can merge related files and create single file for a single page. For example if a page called editProfile.php uses a data picker, a jquery validation plugin and custom js to load user data, we can combine them in a single file so that only file will be loaded for a single page.
So I would vote for separate files for each page and applying optimizations before deploying.
Honnestly i'm not really an expert in this domain but this is my piece of advice on this subject on a production environment.
I would use CDNs for libraries (like jquery). They offer maximum cacheability, and there is a very big chance it is already cached in your client's browsers from visiting other websites. This saves some requests already.
Group and minify your common javascript code, like plugins, utilities, things used throughout your site. It will be requested once for all and will then be available.
Have a separate, minified, script file for each "page" you load dynamically that you will load along with your content.
Loading script for content pages:
Using the .load() method from jquery to load fragments of pages will unfortunately remove any <script> tag present in the fragment. As noted in the jquery load() method, this is to avoid "Permission denied" in IE.
What you can do is to have a <script id="contentScript"></script> tag in your base page and load the script along with the content by replacing the src.
I don't know if it is a good practice but it makes sense to me :-)
I'm developing a modal/popup system for my users to embed in their sites, along the lines of what KissInsights and Hello Bar (example here and here) do.
What is the best practice for architecting services like this? It looks like users embed a bit of JS but that code then inserts additional script tag.
I'm wondering how it communicates with the web service to get the user's content, etc.
TIA
You're right that usually it's simply a script that the customer embeds on their website. However, what comes after that is a bit more complicated matter.
1. Embed a script
The first step as said is to have a script on the target page.
Essentially this script is just a piece of JavaScript code. It's pretty similar to what you'd have on your own page.
This script should generate the content on the customer's page that you wish to display.
However, there are some things you need to take into account:
You can't use any libraries (or if you do, be very careful what you use): These may conflict with what is already on the page, and break the customer's site. You don't want to do that.
Never override anything, as overriding may break the customer's site: This includes event listeners, native object properties, whatever. For example, always use addEventListener or addEvent with events, because these allow you to have multiple listeners
You can't trust any styles: All styles of HTML elements you create must be inlined, because the customer's website may have its own CSS styling for them.
You can't add any CSS rules of your own: These may again break the customer's site.
These rules apply to any script or content you run directly on the customer site. If you create an iframe and display your content there, you can ignore these rules in any content that is inside the frame.
2. Process script on your server
Your embeddable script should usually be generated by a script on your server. This allows you to include logic such as choosing what to display based on parameters, or data from your application's database.
This can be written in any language you like.
Typically your script URL should include some kind of an identifier so that you know what to display. For example, you can use the ID to tell which customer's site it is or other things like that.
If your application requires users to log in, you can process this just like normal. The fact the server-side script is being called by the other website makes no difference.
Communication between the embedded script and your server or frames
There are a few tricks to this as well.
As you may know, XMLHttpRequest does not work across different domains, so you can't use that.
The simplest way to send data over from the other site would be to use an iframe and have the user submit a form inside the iframe (or run an XMLHttpRequest inside the frame, since the iframe's content resides on your own server so there is no cross domain communication)
If your embedded script displays content in an iframe dialog, you may need to be able to tell the script embedded on the customer site when to close the iframe. This can be achieved for example by using window.postMessage
For postMessage, see http://ejohn.org/blog/cross-window-messaging/
For cross-domain communication, see http://softwareas.com/cross-domain-communication-with-iframes
You could take a look here - it's an example of an API created using my JsApiToolkit, a framework for allowing service providers to easily create and distribute Facebook Connect-like tools to third-party sites.
The library is built on top of easyXDM for Cross Domain Messaging, and facilitates interaction via modal dialogs or via popups.
The code and the readme should be sufficient to explain how things fit together (it's really not too complicated once you abstract away things like the XDM).
About the embedding itself; you can do this directly, but most services use a 'bootstrapping' script that can easily be updated to point to the real files - this small file could be served with a cache pragma that would ensure that it was not cached for too long, while the injected files could be served as long living files.
This way you only incur the overhead of re-downloading the bootstrapper instead of the entire set of scripts.
Best practice is to put as little code as possible into your code snippet, so you don't ever have to ask the users to update their code. For instance:
<script type="text/javascript" src="http://your.site.com/somecode.js"></script>
Works fine if the author will embed it inside their page. Otherwise, if you need a bookmarklet, you can use this code to load your script on any page:
javascript:(function(){
var e=document.createElement('script');
e.setAttribute('language','javascript');
e.setAttribute('src','http://your.site.com/somecode.js');
document.head.appendChild(e);
})();
Now all your code will live at the above referenced URI, and whenever their page is loaded, a fresh copy of your code will be downloaded and executed. (not taking caching settings into account)
From that script, just make sure that you don't clobber namespaces, and check if a library exists before loading another. Use the safe jQuery object instead of $ if you are using that. And if you want to load more external content (like jQuery, UI stuff, etc.) use the onload handler to detect when they are fully loaded. For example:
function jsLoad(loc, callback){
var e=document.createElement('script');
e.setAttribute('language','javascript');
e.setAttribute('src',loc);
if (callback) e.onload = callback;
document.head.appendChild(e);
}
Then you can simply call this function to load any js file, and execute a callback function.
jsLoad('http://link.to/some.js', function(){
// do some stuff
});
Now, a tricky way to communicate with your domain to retrieve data is to use javascript as the transport. For instance:
jsLoad('http://link.to/someother.js?data=xy&callback=getSome', function(){
var yourData = getSome();
});
Your server will have to dynamically process that route, and return some javascript that has a "getSome" function that does what you want it to. For instance:
function getSome(){
return {'some':'data','more':'data'};
}
That will pretty effectively allow you to communicate with your server and process data from anywhere your server can get it.
You can serve a dynamically generated (use for example PHP or Ruby on Rails) to generate this file on each request) JS file from your server that is imported from the customers web site like this:
<script type="text/javascript" src="//www.yourserver.com/dynamic.js"></script>
Then you need to provide a way for your customer to decide what they want the modal/popup to contain (e.g. text, graphics, links etc.). Either you create a simple CMS or you do it manually for each customer.
Your server can see where each request for the JS file is coming from and provide different JS code based on that. The JS code can for example insert HTML code into your customers web site that creates a bar at the top with some text and a link.
If you want to access your customers visitors info you probably need to either read it from the HTML code, make your customers provide the information you want in a specific way or figure out a different way to access it from each customers web server.