We are working on a single-page-application (SPA) developed in ASP.NET MVC using knouckout and a wealth of other libraries. Routing will be done in front-end, maybe we will use crossroads.js. A lot of information is presented in virtual grids using slickgrid.js. All data is fetched from backend using AJAX.
Now, if you want to crawl and index such a site from SharePoint, how would you go about that? If you just load the main page with no javascript, it is almost empty.
Update
After more investigation into this issue, I have concluded that there are at least two possible solutions to this kind of problem.
Possible solution 1: Render HTML
This approach would involve detecting that a SP crawler is crawling your site, and then return static HTML pages for the crawler. PhantomJS could possibly be used for this. There are however several uncertain aspects to this solution, and I suspect that it would involve a lot of work.
Possible solution 2: Import data into SharePoint
As so clearly described by Josh below, you could import the data that you want crawlable into SharePoint. Then SharePoint can be configured to crawl the data, and the data is not Javascript dependant anymore as it is inside SharePoint. I think this is the best and easiest solution and will mark Josh answer as the accepted answer.
While I see that this question is getting a lot of close requests, I did run across this very question on a previous project. Sharepoint won't be able to index the page that data is manipulated on via your SPA, but inside SP, you can connect external data sources into the search service, thus exposing the data inside the SPA. You would then write custom search results tied to the content type of the data exposed in order to make the results a bit more friendly than just a data row. You might create an entry point in your SPA that can take in a URL with a parameter so you can send the user from the search results to the SPA in one shot.
There are a lot of interconnected concepts to this solution, so I'd suggest looking into connecting external data sources and adding them to the crawled index of SP. Then, create a content type out of the exposed objects. And finally, add a custom search result template for the content type. MSDN will be your friend on this and so will your SP administrator.
Related
When creating a technical site, I faced the need to transfer data and save it in a file. I am making a website in Django. In models I create input data. I receive these data in HTML and, if necessary, the user changes them and JS performs the necessary calculations, the results of which are sent to the same page. It is possible to do a lot of analytical work on outgoing data (build graphs, calculate a loan, calculate various project risks). I prefer to create several pages so as not to fill one. All pages have already been built. There was a problem of this nature: On the main page, JS did his job but I do not know how to access this data on other pages. Is there a solution to write a JSON string to a file or send them to Django model (update static data). I read a lot and looked for solutions, but I could not figure it out until the end. Maybe someone knows the solution.
Thank you in advance.
If need more information, please let me know.
This is my first request for help, so please don’t be strict if I cannot ask correctly.
Writing to a temporary or holding table in a database might be a solution. You would be able to fetch the data as needed when a new page loads.
There is probably a better solution though.
I am looking for either an existing framework or software that would allow us to summarize our large MySQL database and allow us to present the information in the form of charts or tables.
I have looked online and found a few that are standalone web services, however for our purposes we would need it to be embedded without our existing web app and able to run either on the front end or on our server.
we would also be able to make use of any existing framework(c#) for data aggregation as we could implement them in c# on our server and display the information using JavaScript. We are trying to avoid implementing our own version due to time constraints so any pre-made method would be preferable.
I know this is a fairly vague question but we are just looking for some suggestions as the topic of web based data analysis doesn't seem to return many useful options for us.
I'm generating an entire site using just an index.html with JS scripts.
The JS creates the HTML content based on JSON data received via the server-side API. This works great client-side and makes the site load speed and interaction very fast but there is a snag... when a crawler comes to index the page it will see a blank page.
The obvious solution is to provide an XML site map with static versions of all the pages. The problem is... how to generate static versions of each page when they are only generated client-side and all logic and templates are client-side?
This is not a new issue... I'm sure anyone generating pages dynamically client-side has hit this issue and solved it but I thought I'd ask the dev community before diving in and trying to solve this.
2019 Update
Tech has moved on significantly. I would encourage anyone looking to create SSR (server-side rendered) and client-side web apps in one isomorphic code base to take a look at the excellent Next.js.
Next.js wraps React with a server-side routing and rendering system built in Node.js, defines a standard interface to getting data for pages on server and client, and comes with some out of the box features that make it one of the best choices (IMHO) for both SSR and CSR web applications.
Oh... and they have a great tutorial too!
2013 Answer
I've managed to generate static pages from the client-side output by using PhantomJS and capturing the HTML output after the page and all JS has finished loading/executing. This method is slower than I would like and unlikely to scale well but it's the only option that I can think of so far.
The site already receives over 10,000 page views a day with over 8,000 unique visitors so pages get updated regularly as new comments / posts are created and then these changes are added to a queue which gets process in a separate server to generate static pages with Phantom.
The only other way I can think of doing this is to create a Node.js process that uses the same jsRender library and builds HTML output from the template files based on some data, but this would be time consuming to set up and would not generate the exact same output that the dynamic site creates. Google may frown on me serving it static pages that don't really represent the dynamic version that "normal" visitors can see.
This seems like an unsolvable issue. Either I generate the pages entirely server-side, or crawlers cannot index the pages. :(
I'm planning to write a web app using either Backbone or Angular. We want to push "widgets" from the server to the client (i.e. semi-complicated, dynamic but largely autonomous UI elements... something maybe like the popular TodoMVC app). So we'll need to send over a template, some javascript (controllers etc.), probably CSS, and JSON data (models).
We're debating how to send over all the information. How much can be, and what should be encapsulated in the JSON?
Is it possible to create files out of data passed over? i.e. could we pull out CSS and apply the rules to our document? I believe it would be easier to run javascript passed over this way.
I'm under the impression that being able to cache the template is important... does that require loading it as a (separate) resource rather than as part of some huge JSON object?
As for CSS, it needs to be loaded before inserting into the DOM (so we don't want just a promise). Would it make sense - or even be possible - to pass our CSS rules in the JSON and extract them somehow?
EDIT: to more fully describe what I'm working with, I'm focusing only on the front-end. The back-end can be customized to send over resources however we want - they'll optimize that depending on what the front-end needs. Our backend stack includes MongoDB, Tapestry, ActiveMQ.
The payload that needs to get sent over will be all the resources necessary to push something like a Mac Dashboard Widget or Windows Gadget into the browser. So HTML, CSS, Javascript, data will all get sent over. We want things to be snappy and minimize server requests as much as possible, as some of these payloads may get to be somewhat large.
Your questions are rather vague and the answers can change by coding on a different build platform. Your chosen application design can also impact the best practices to follow for implementing X type of application to X device(s). There's more than one way to skin a cat ya know.
What information are you sending over? Is it a lot of raw data arrays? Is it more document based where XML could be more beneficial? You'll need to work with your DBA if the data is stored in a repository and you'll be querying it. Will you need to write code to format your data or will the DBMS output it in the format you need?
What do you mean the CSS needs to be loaded before being inserted? Everything loads in the order you specify so making sure your CSS loads at the proper time shouldn't be an issue. If you're using ASP.NET, you can also use a SSM (style-sheet manager) to serve your CSS and only load whats required. SSM's are great if you have a lot of style-sheets to serve.
A lot of the questions you asked will be answered as you get further into the design phase. In order to get some solid answers and not just speculation on what is the best method, you will need to publish a lot more details than this. Any answers at this point shouldn't carry much weight in your decisions. They certainly wouldn't mine.
Intro
I know this has been asked before but the questions I found were either to specific or to general to provoke the kind of answer I was looking for. The best possible answer I can imagine would be an example using backbone and the least amount of server-side logic possible (no preferred language/framework there).
Problem
I am planning an javascript/ajax-heavy (backbone + mostly-json backend) application that implements a facetted search. Take for example a facetted search of a simple shoe shop application that lets you filter color, brand and type of shoes and sort by price and size or whatever else.
Assume I am using backbone or a similar framework on the client and a json service as a backend.
What would be a good (tradeoff between effort and result) strategy to achieve seo-friendliness as well as a snappy interface?
Resources
A solution that came to my attention is Hijax by reusing client-sided templates on the server-side, as described here: http://duganchen.ca/single-page-web-app-architecture-done-right
Resources that I digested without final conclusion
http://code.google.com/intl/de-DE/web/ajaxcrawling/
https://stackoverflow.com/a/6194427/818846
http://www.quora.com/Search-Engine-Optimization-SEO/If-I-have-data-that-loads-using-json-JavaScript-will-it-get-indexed-by-Google?q=seo+javascript
The general point in SEO friendliness: It should work without JavaScript.
It's also good for accessibility, so you should do it like this, if the user does not have JavaScript enabled (like the search engine does), it will work.
If he has JavaScript enabled, (like any sane human being does), it will work with all the nifty JavaScript features you've added.
As a general usability rule of thumb: If it works, it should also work without JavaScript!
The solution of your first link sounds right. The main issue of a single page app is that you have to render your templates on both sides, the backend and the frontend. Using the Mustache or google closures template will be good solution for this.
The same solution that was used for google+, where initially the side will be rendered on the server and you load a static html page, after that the page will be rendered on the client side but with the same templates as on the server.
Also remember that the search engines follow links much more often than they (ever?) complete forms.
This problem of enabling the crawlers to see your db contents is called the "dark web," "invisible web", "deep web" or "hidden web". Blog post
So re your problem statement:
a facetted search of a simple shoe shop application that lets you filter color, brand and type of shoes and sort by price and size or whatever else.
I'd suggest that you include searches via a hierarchy of links in addition to searching via forms with select fields.
Eg, on a secondary menu include all the different brands as individual links. Then each link should lead to a list of the products sold by that brand. The trick is arrange things so that the link to an individual shoe will take you back to the first page (the rich one page app) but showing the specific shoe. -- And the page should implement the Google Ajax-crawling recommendations that you reference in the OP.