This is bugging me, because I'm a complete newbie in web interaction:
How can I download a web page that is dynamically autoloading?
For instance, twitter's feed, facebook's wall, etc. The only restriction is that I'd like to do this on a client, with a background job, no browser involved. I'd prefer to use C#, but any language is completely welcome.
You can try to achieve your goal using a headless browser like phantomJS as discussed here.
Also check if the service you want to get data from offers you an API, which is always a better solution! Accessing websites in an automated way is no good practice.
Related
How can I tell what is the background of a website by looking at the source code of the page?
Like, what is the architecture, which web framework was used and similar?
For example, is it PHP or Node.js on the server-side? Was it developed in ASP.NET? Which JS framework is it using?
EDIT:
Found this for React.
And this for Angular.
You will not reliably get this information. You can, as Mirakurun said, make some guesses by looking at form actions, or URLs that appear in source-code, but it is not reliable, and you certainly won't know exactly what framework is on the back-end.
This is partially on purpose, as leaking specific framework information would allow attackers to target specific vulnerabilities.
I have been searching for an answer to this problem now for several weeks. I also previously tried to research this a few years ago to no avail.
Problem Summary:
My company has developed a web-based data analytics suite for a major beverage distributor. They have recently asked for a feature that allows the user to print or download a visually pleasing version of the rendered app as a PDF. I have had no luck in finding a solid, controllable, or reliable method to do this. I was hoping the stack community might be able to point me in the right direction.
Current Tech Stack:
Plack servers
Perl base on the Dancer framework
Standard web dev front-ends: HTML5, CSS3, Javascript, Jquery/UI
Client is using IE9/10 and Chrome.
Attempted Solutions Summary:
Obviously I started with the window.print() and tried to control what printed using classes and a specialized print.css but the output was still awful.
I looked in to pdfmachine and pdfbox and even contacted Adobe's acrobat development team directly to see if they had an out of the box solution our company could purchase. I was informed that such a product would be counter intuitive to their desired business model of putting an acrobat subscription on each client computer rather than a single server side application.
I have extensively searched the stack articles but did not feel that the articles I found covered what I was looking for.
At present, I am all out of ideas and am hoping somebody out there has had better luck at this than I have.
tl;dr = I need a pdf version of the rendered output of a complex reporting app.
Thanks for your time stack, I appreciate it.
A solution I have used in the past is to use PhantomJS running on a server to generate the PDF for download/email. Usually if the content is sensitive the server (that handles authentication) would provide a single use viewing token that is then passed to a PhantomJS process. It loads the URL with the viewing token then saves as a PDF.
Further info on Phantoms screen cap API can be found here on GitHub.
https://github.com/ariya/phantomjs/wiki/Screen-Capture
Is it something you can create in Perl using PDF::API2 or PDF::Create? You can load and modify and existing PDF (handy if you want standard headers and footers), and then insert the relevant content. The learning curve can be a bit steep, but simple reports should be easy enough.
See PDF::TextBlock and PDF::Table too - they are great little helpers.
Consider this service http://pdfmyurl.com/ . I try to use many perl modules, but they dont satisfy my problems.
I'm about to be designing a social website that must be able to handle a high volume of users.
Here's how I want to design it:
Sinatra on the backend with a full REST api to do all of the operations on the website
JQuery/HTML front end web app that communicates exclusively with the REST API
In this way, I only have to make one central API that other apps (iPhone, Android) app will communicate with.
Also, it seems like it will be less load on the server, since the server only has to serve the minimum amount of info and everything else is done client side.
How come more sites aren't done this way?
Why wouldn't I want to do this? It seems like a good idea to me...
Because it is fragile and search engines won't index your content.
37signals are working on a framework called Cinco for this kind of architecture:
http://thinkvitamin.com/code/javascript/37signals-cinco-framework-to-be-open-sourced/
But if you stick to just jQuery I think you'd miss a lot things you take for granted when using a framework, which you might have to re-implement. But it really depends on your app.
It's not too hard to get basic functionality without JS, and after add Ajax stuff to your application. For example - you can create your API to respond as plain html markup and to respond JSON.
I like idea to do all stuff around API with JS\ajax, it maybe easy and more elegant with new technologies, it giving you more capabilites, but even google can't indexing full ajax sites now.
Read:
http://www.google.com/support/webmasters/bin/answer.py?answer=81766
Does anyone know how disqus works?
It manages comments on a blog, but the comments are all held on third-party site. Seems like a neat use of cross-site communication.
The general pattern used is JSONP
Its actually implemented in a fairly sophisticated way (at least on the jQuery site) ... they defer the loading of the disqus.js and thread.js files until the user scrolls to the comment section.
The thread.js file contains json content for the comments, which are rendered into the page after its loaded.
You have three options when adding Disqus commenting to a site:
Use one of the many integrated solutions (WordPress, Blogger, Tumblr, etc. are supported)
Use the universal JavaScript code
Write your own code to communicate with the Disqus API
The main advantage of the integrated solutions is that they're easy to set up. In the case of WordPress, for example, it's as easy as activating a plug-in.
Having the ability to communicate with the API directly is very useful, and offers two advantages over the other options. First, it gives you as the developer complete control over the markup. Secondly, you're able to process comments server-side, which may be preferable.
Looks like that using easyXDM library, which uses the best available way for current browser to communicate with other site.
Quoting Anton Kovalyov's (former engineer at Disqus) answer to the same question on a different site that was really helpful to me:
Disqus is a third-party JavaScript application that runs in your browser and injects itself on publishers' websites. These publishers need to install a small snippet of JavaScript code that makes the first request to our servers and loads initial JavaScript loader. This loader then creates all necessary iframe elements, gets the data from our servers, renders templates and injects the result into some element on the page.
As you can probably guess there are quite a few different technologies supporting what seems like a simple operation. On the back-end you have to run and scale a gigantic web application that serves millions of requests (mostly read). We use Python, Django, PostgreSQL and Redis (for our realtime service).
On the front-end you have to minimize your payload, make sure your app is super fast and that it doesn't break in extremely hostile environments (you will be surprised how screwed up publisher websites can be). Cross-domain communication—ability to send messages from hosting website to your servers—can be tricky as well.
Unfortunately, it is impossible to explain how everything works in a comment on Quora, or even in an article. So if you're interested in the back-end side of Disqus just learn how to write, run and operate highly-scalable websites and you'll be golden. And if you're interested in the front-end side, Ben Vinegar and myself (both front-end engineers at Disqus) wrote a book on the topic called Third-party JavaScript (http://thirdpartyjs.com/).
I'm planning to read the book he mentioned, I guess it will be quite helpful.
Here's also a link to the official answer to this question on the Disqus site.
short answer? AJAX, you get your own url eg "site.com/?comments=ID" included via javascript... but with real time updates like that you would need a polling server.
I think they keep the content on their site and your site will only send & receive the data to/from disqus. Now I wonder what happens if you decide that you want to bring your commenting in house without losing all existing comments!. How easy would you get to your data I wonder? They claim that the data belongs to you, but they have the control over it, and there is not much explanation on their site about this.
I'm always leaving comment in disqus platform. Sometimes, comment seems to be removed once you refreshed it and sometimes it's not. I think the one that was removed are held for moderation without saying it.
As an hobby I wrote a game in JavaScript and would love to publish it to Facebook, can you write a Facebook application using pure HTML and JavaScript?
Yes. Absolutely. In fact, one of the options for developing a Facebook application is an embedded frame pointing to your website which can be whatever you want, including straight HTML web serving.
There are two officially sanctioned libraries for accessing Facebook information and one of them is Javascript based.
You can do quite a bit on Facebook with just Javascript/HTML. There's a lot of possibilities using services outside of your own site (think Flckr, Google Ajax API etc...)
Facebook applications are pleasantly simple to develop.
The documentation is indeed at:
http://developers.facebook.com/
I would also note that traffic with regards to Facebook, things can spin up really quickly. There are stories of students creating applications which quickly scaled to millions of users. I'm not saying this is likely to be the case for you, but it is can be an effective and a highly viral distribution platform.
Yes, though your app may not be able to do much on Facebook. Have a look at http://developers.facebook.com for documentation.