Develop Reference

Can sites detect auto-refreshing tools?

I don't want to get into details of my situation, but it's like this:
I'm part of a website and so are other people. On this site, we are all waiting for something to appear on the site, but one must refresh the site to see if anything new appeared. The site rules explicitly mention that auto-refreshing tools are prohibited.
If I for example use a browser extension that would refresh the site every minute, could the site detect that? Do these extensions make the same request to the site as if I clicked the refresh button? I'm sure they could detect the refreshing intervals, but that can be avoided by a random timer. So what is the most undetectable way of autorefreshing a site?

Different extensions work differently.some use the javascript reload function, some will just grab the url and replace window.location with it effectively making another GET request
The only difference in the actual request will be whether or not cache control is set etc..so they might use that for detection.So definitely you need to take a look in the documentation to see how the request is being issued in that specific extension
As far as circumstantial evidence they can cookie you,use local data storage,use your ip address to determine you are the same user which in that case you would need to find a workaround for each one

Offline HTML5 website with JavaScript search

I am building a simple website that needs to be able to run completely offline if needs be. With the intention of being a 50+ page searchable reference manual.
I need the whole site to be cached upon opening one page. I'm doing this with the appcache manifest and getting the site to cache and be viewed on an offline mobile seems to work ok.
The site has a basic JavaScript search facility (that was a freeware download) and while online this search works perfectly. As soon as the internet connection is stopped and the cached version is used the search no longer works, displaying one of two symptoms 1. Button is clicked and nothing happens or 2. A 'webpage cannot be found' kind of error is displayed.

Quote from https://developer.mozilla.org/en-US/docs/HTML/Using_the_application_cache#Gotchas
Never access cached files by using traditional GET parameters (like
other-cached-page.html?parameterName=value). This will make the
browser bypass the cache and attempt to get it from network. To link
to cached resources that have parameters parsed in JavaScript use
parameters in the hash part of the link, such as
other-cached-page.html#whatever?parameterName=value.
But that is exactly what your js-search does. It tries to load the subpages like this "http://www.filemanage.co.uk/offline/index.html?1350563635665" using XHR.
As a fix try this
// change in function sendRequest line 228 from
this.httpRequest.open("GET", uri+"?"+q, true);
// to
this.httpRequest.open("GET", uri, true);

How to check the authenticity of a Chrome extension?

The Context:
You have a web server which has to provide an exclusive content only if your client has your specific Chrome extension installed.
You have two possibilities to provide the Chrome extension package:
From the Chrome Web Store
From your own server
The problem:
There is a plethora of solutions allowing to know that a Chrome extension is installed:
Inserting an element when a web page is loaded by using Content Scripts.
Sending specific headers to the server by using Web Requests.
Etc.
But there seems to be no solution to check if the Chrome extension which is interacting with your web page is genuine.
Indeed, as the source code of the Chrome extension can be viewed and copied by anyone who want to, there seems to be no way to know if the current Chrome extension interacting with your web page is the one you have published or a cloned version (and maybe somewhat altered) by another person.
It seems that you are only able to know that some Chrome extension is interacting with your web page in an "expected way" but you cannot verify its authenticity.
The solution?
One solution may consist in using information contained in the Chrome extension package and which cannot be altered or copied by anyone else:
Sending the Chrome extension's ID to the server? But how?
The ID has to be sent by you and your JavaScript code and there seems to be no way to do it with an "internal" Chrome function.
So if someone else just send the same ID to your server (some kind of Chrome extension's ID spoofing) then your server will consider his Chrome extension as a genuine one!
Using the private key which served when you packaged the application? But how?
There seems to be no way to access or use in any way this key programmatically!
One other solution my consist in using NPAPI Plugins and embed authentication methods like GPG, etc. But this solution is not desirable mostly because of the big "Warning" section of its API's doc.
Is there any other solution?
Notes
This question attempts to raise a real security problem in the Chrome extension's API: How to check the authenticity of your Chrome extension when it comes to interact with your services.
If there are any missing possibilities, or any misunderstandings please feel free to ask me in comments.

I'm sorry to say but this problem as posed by you is in essence unsolvable because of one simple problem: You can't trust the client. And since the client can see the code then you can't solve the problem.
Any information coming from the client side can be replicated by other means. It is essentially the same problem as trying to prove that when a user logs into their account it is actually the user not somebody else who found out or was given their username and password.
The internet security models are built around 2 parties trying to communicate without a third party being able to imitate one, modify or listen the conversation. Without hiding the source code of the extension the client becomes indistinguishable from the third party (A file among copies - no way to determine which is which).
If the source code is hidden it becomes a whole other story. Now the user or malicious party doesn't have access to the secrets the real client knows and all the regular security models apply. However it is doubtful that Chrome will allow hidden source code in extensions, because it would produce other security issues.
Some source code can be hidden using NPAPI Plugins as you stated, but it comes with a price as you already know.
Coming back to the current state of things:
Now it becomes a question of what is meant by interaction.
If interaction means that while the user is on the page you want to know if it is your extension or some other then the closest you can get is to list your page in the extensions manifest under app section as documented here
This will allow you to ask on the page if the app is installed by using
chrome.app.isInstalled
This will return boolean showing wether your app is installed or not. The command is documented here
However this does not really solve the problem, since the extension may be installed, but not enabled and there is another extension mocking the communication with your site.
Furthermore the validation is on the client side so any function that uses that validation can be overwritten to ignore the result of this variable.
If however the interaction means making XMLHttpRequests then you are out of luck. Can't be done using current methods because of the visibility of source code as discussed above.
However if it is limiting your sites usability to authorized entities I suggest using regular means of authentication: having the user log in will allow you to create a session. This session will be propagated to all requests made by the extension so you are down to regular client log in trust issues like account sharing etc. These can of course be managed by making the user log in say via their Google account, which most are reluctant to share and further mitigated by blocking accounts that seem to be misused.

I would suggest to do something similar to what Git utilises(have a look at http://git-scm.com/book/en/Git-Internals-Git-Objects to understand how git implements it), i.e.
Creating SHA1 values of the content of every file in your
chrome-extension and then re-create another SHA1 value of the
concatenated SHA1 values obtained earlier.
In this way, you can share the SHA1 value with your server and authenticate your extension, as the SHA1 value will change just in case any person, changes any of your file.
Explaining it in more detail with some pseudo code:
function get_authentication_key(){
var files = get_all_files_in_extension,
concatenated_sha_values = '',
authentication_key;
for(file in files){
concatenated_sha_values += Digest::SHA1.hexdigest(get_file_content(file));
}
$.ajax({
url: 'http://example.com/getauthkey',
type: 'post'
async: false,
success:function(data){
authentication_key = data;
}
})
//You may return either SHA value of concatenated values or return the concatenated SHA values
return authentication_key;
}
// Server side code
get('/getauthkey') do
// One can apply several type of encryption algos on the string passed, to make it unbreakable
authentication_key = Digest::<encryption>.hexdigest($_GET['string']);
return authentication_key;
end
This method allows you to check if any kind of file has been changed maybe an image file or a video file or any other file. Would be glad to know if this thing can be broken as well.

How to implement #! based links?

I always wondered how to instantly navigate through pages using # or #! in URLs. Many websites like Google are using it on http://www.google.com/nexus/ , when user click any of the links, nothing changes and things open instantly, only URL changes, for ex: www.example.com/#contact or www.example.com/#home
How can I do this with 8 of my pages? (Home, Features, Rates, Contact, Support)

You may want to take a look at a basic AJAX tutorial (such as http://marc.info/?l=php-general&m=112198633625636&w=2). The real reason the URLS use #! is to have them get indexed by google. If you want you AJAX'ed URLs to be indexed by Google, you'll have to implement support for _escaped_fragment_ (see: http://code.google.com/web/ajaxcrawling/docs/specification.html).

The only reason this is used, is to show the state of an AJAX enhanced page in the url. This way, you can copy and bookmark the url to come back to the same state.
Older browsers don't allow you to change the url in the address bar without the page being reloaded. The latest browsers do (search for PushState). To work around this, you can change the hash of the url. This is the part that is normally used to jump to an anchor, but you can use it for other purposes using JavaScript.
The ! isn't strictly necessary for this process. The ! is implemented by Google. It allows these urls to be indexed. Normally hashes aren't indexed separately, because they mark only a different part of the same page (anchor). But by adding the !, you create a shebang or hashbang, which is indexed by Google.
Without explaining everything here, you should find a lot of information when you search for Ajax, HashBang and PushState.
Addition: Check History.js. It is a wrapper for the PushState api, that falls back to using hashes on older browsers.

Configure page based on address bar location

I'd like to build a site similar to http://www.20thingsilearned.com/ in the respect that an address typed into the address bar does not send the user to a new page, but configures the running application instead. Flip through their book and you'll notice the address bar location changing, but you're obviously not sent to new pages... you stay within the running application, which I would assume lives in only one location on the server. And you can use those addresses to jump to specific pages within the app. How is this happening? I assume it takes some server side magic. Any clues as to what is going on is appreciated!

The site you mention is built with spiffy new features of HTML5, specifically the History API. With HTML5 (in newer browsers), you can manipulate the address bar and browser path despite making asynchronous requests. More here: http://diveintohtml5.ep.io/history.html

They are using the HTML5 History API. See Manipulating the browser history (the section on "Adding and modifying history entries").