In the Firebug add-on for Firefox, how is Firebug able to get the connecting, waiting, and receiving time? Also, how is it that Firebug can know the file size before the file is even finished loading?
Is JavaScript used in these calculations, or does Firebug use another method altogether?
Firebug plugs into Firefox and can fetch data from there. The webserver maybe sends a Content-Length HTTP header, so this is the expected size of the downloaded file. Firefox puts all resources to load into a queue while parsing the website. The time from putting an item into the queue until it is fetched can be considered "waiting time". The HTTP fetch engine from Firefox can give information about how long a connect() to the server takes.
So to put a long story short, I believe Firefox has an API so that extensions can fetch information like that from the engine.
Firebug is most, if not all, JavaScript.
If you're truly interested, you can find the network monitor related bits and pieces here in the Firebug source tree.
Related
I'm trying to build a dyi adblock, that instead of just blocking ads, it first analyzes them based on their URL structure and their provided response from a fetch request
I'm using this as my starting point: https://github.com/AdrS/dyi-adblocker
I have a huge list of URLs and using webRequest.OnBeforeRequest to stop each request from that ad URL list and block them.
In the middle, I would like to first do a GET request to the stopped URL, get it's response, analyze everything and determine if it should be blocked or not.
I tried doing everything on a Google Chrome extension, based on thst github. But it seems to not be possible due to the listener not waiting for the fetch request (too async) + I'm getting "max stack size reached" errors. I searched for info on the internet and found out that it is somehow possible with a Mozilla Firefox extension, but couldn't find a solution.
Basically the steps should be:
Have a listener webRequest.OnBeforeRequest to stop requests from the urls in the list given.
fetch the URL's response
Analyze the URL's structure
Analyze the response
Calculate results
6.1. If results are positive, allow the request to go through
6.2. If results are negative, block the request
Speed is not an issue, it can even load for websites for minutes if needed, as this would be a small simple prototype.
Does anyone know a solution? Both Chrome and Firefox extension solutions work, as long as I can stall the webRequest.OnBeforeRequest (or anything else) for as long as I want until I do the needed analysis.
Any code examples and links to documentations are appreciated!
I am trying to do some high scale testing with my nodejs server so I want to find out how its possible to run the chrome browser on the terminal, similar to how nodejs runs the server.
I want to have a few hundred on a single machine which is why I want to avoid running the actually browser tab instances. It also needs to specifically be google chrome.
Right now Im trying to figure out how this is possible, even if it's just somehow running the messaging javascript from the terminal.
Can someone please point me to the right direction :]
Have you looked into Request, CasperJS, or PhantomJS? I think they may be just what you're looking for. Phantom is a headless browser that can browse webpages for you (useful for testing, taking screenshots, and taking actions on pages for you). CasperJS can help you out with that, as well, and Request allows you to make http requests from your server, rather than client. If you need to search/ traverse the page your server gets back, you can also use Cheerio, which is pretty much jQuery for your server.
Hope that helps!
This doesn't involve coding. I am just curious on how to make a page like the one described below.
I came across a website where we can attend quiz/tests.
I tried to debug the browser so that I can see if I could hack through the codes by getting the values that are getting passed in debugger.
But to my surprise the debugger is not coming up when I click F12 in that page.
Somehow I opened debugger for that page and I clicked on the Network tab to capture requests that are sent.
But as I was proceeding through the test, not even one request is getting captured in the debugger but the answers are getting validated and scores are getting updated !! I was not even able to do inspect element
I guess its a java applet as i saw the below line in the the launch button
flagPlayerCourse = true;launchApplet(secureSessionId,courseName, courseType,winParams, use508);disablePlayButton(1, 0);
the url had SinglePassUserCmd.cfm?sessionid=3xxxxx
So my question is how can we create such a webpage in which the requests are not captured in the debugger!? I would be happy if someone could tell me how do the same in asp.net. In which language can we develop such web pages!?
Applets are completely different world. Its almost as good as running a .net application on your client machine.
What you see in the debugger are ajax requests and resources loading. If a site doesn't make them, you won't see any network requests in the browser.
That doesn't mean that you can't capture the data being send. You can always use a debugging proxy like fiddler to see what traffic is going across. Ofcourse a secure site would secure their traffic over https.
Applets require a java plugin in your browser. There are similar plugins like Silverlight, flash/shockwave(swf) that too can make network requests.
I'm working on a headless browser based on WebKit (using C++/Qt4) with JavaScript support. The main purpose for this is being able to generate a HTML spanshot of websites heavily based on JavaScript (see Backbone.js or any other JavaScript MVC).
I'm aware that there isn't any way for knowing when the page is completely loaded (please see this question) and because of that, after I get the loadFinished signal (docs here) I create a timer and start polling the DOM content (as in checking every X ms the content of the DOM) to see if there were any changes. If there werent I assume that the page was loaded and print the result. Please keep in mind that I already know this is not-near-to-perfect solution, but it's the only one I could think of. If you have any better idea please answer this question
NOTE: The timer is non-blocking, meaning that everything running inside WebKit shouldn't be affected/blocked/paused in any way.
After testing the headless browser with some pages, everything seems to work fine (or at least as expected). But here is where the heisenbug appears. The headless browser should be called from a PHP script, which should wait (blocking call) for some output and then print it.
On my test machine (Apache 2.3.14, PHP 5.4.6) running the PHP script outputs the desired result, aka, the headless browser fetches the website, runs the JavaScript and prints what a user would see; but running the same script in the production server will fetch the website, run some of the JavaScript code and print the result.
The source code of the headless browser and the PHP script I'm using can be found here.
NOTE: The timer (as you can see in the source code of the headless browser) is set to 1s, but setting a bigger amount of time doesn't fix the problem
NOTE 2: Catching all JavaScript errors doesn't show anything, so it's not because of a missing function, wrong args, or any other type of incorrect code.
I'm testing the headless browser with 2 websites.
This one is working on both my test machine and in production server, while this one works only in my test machine.
I'm more propone to think that this is some weird bug in the JavaScript code in the second website rather than in the code of the headless browser, as it generates a perfect HTML snapshot of the first website, but then again, this is a heisenbug so I'm not really sure what is causing all this.
Any ideas/comments will be appreciated. Thank you
Rather than polling for DOM changes, why not watch network requests? This seems like a safer heuristic to use. If there has been no network activity for X ms (and there are no pending requests), then assume page is fully "loaded".
I'm looking for something (preferably a Firefox extension) which will allow me to see all AJAX subrequests. Basically, anytime an XMLHTTPRequest() is made, I want to know what URL was being requested and what (if any) GET and POST vars were passed along with it.
Unless I'm missing it, I don't see anything quite like this in Firebug or Web Developer Toolbar.
(In case you're curious, the main reason for wanting this is that I want to scrape a local copy of a site that is using JS to load all its contents, and I don't want to spend hours digging through their JS code when I could just see the subrequests being made.)
Firebug should show it in the Net panel.
EDIT: Also, if Firebug slows down your Firefox in a way that makes it unusable, like it does for me sometimes, WireShark (formerly Ethereal) will analyze all network traffic on your system, including HTTP and AJAX requests.
Fiddler 2 is a great tool for watching HTTP traffic.
* 2014 Update *
Since my original post, both Internet Explorer and Chrome have added built-in developer tools that are quite useful. While I still support and use Fiddler for non-web related http monitoring or when I need to really dig in deep and modify requests in transit, these days I find myself pretty much using IE or Chrome's built in tools as they are sufficient for 99% of my needs. They both have a networking tab, where you can monitor http requests, plus other nifty features for debugging webpages.
Internet Explorer F12 Tools Introduction
Documentation for Chrome's F12 Tools
FireBug, which is mentioned above is also a good tool if you're using FireFox.
I have used nearly all the big tools out there (fiddler, firebug, httpfox, live http headers). The best one in my opinion is HTTPFox if you are using Firefox because it has easy filtering capabilities, allows you to view your JSON the request/response as raw or formatted and the over all interface is very clean.
Also, I can't do without WFetch for testing web services and AJAX stuff. Usually I'll take the request from HTTPFox and paste it into WFetch. This way I don't have to go through the browser each time. I just hit F5 in WFetch to replay the request and I directly modify the request.
Side note: In WFetch, be careful about content-length if you have a POST payload. If you modify the payload the content-length needs to be changed as appropriate. For this reason I save my requests in a file and point WFetch to the file. In the file itself it is easy to know the length of the line and change content-length to that.
(source: koreus.com)
alt text http://www.port80software.com/assets/images/ld_wfetch3.gif
Firebug shows them in both the Net panel, and the Console view. You need to turn on "Show XMLHttpRequests" though.
Have a look at Tamper Data as well. It's a Firefox add-on that lets you have a look at each individual HTTP request (including any post data) and response, and the headers of both. I've found it very useful for debugging AJAX problems...although Firebug works just fine.