I have been trying to understand how exchange rates are updated real-time on this website. With a quick look at the 'network' tab on developer tools, it became clear that website is getting responses periodically from this url. The problem is that the response text from the requests consist of sequences of random letters and numbers. It seems that actual content is encrypted and since exchange rates are displayed on client side, response data should be somehow decrypted with JavaScript on front end (I think).
So, my question is, what are some hints to explore JavaScript decryption algorithm, since all 'js' files are minified and variable names are just letters? What kind of tools and practices could you use to solve this kind problems?
Any suggestion or help on this matter would be very much appreciated.
The source code (not minfied) can be seen here. You will notice that it uses a function rc4decrypt to decrypt the data. rc4decrypt is defined as:
function rc4decrypt (a){
return rc4(key,hexDecode(a))
};
where key is a global (window) variable. Further steps should be easy.
(Please be aware of any legal implications of your actions).
Related
I am using javascript/jquery to generate a sql query.
I have a sql query I'm generating and using inside a javascript/jquery script.
Something like this:
var storeName;
var query = "SELECT * FROM stores where storeName = '" + storeName + "';";
(storeName is generated through jquery when a user selects from html)
So when storeName is something like "Jackson Deli" the query runs just fine.
But then when storeName is "Jackson's Deli" it does not work and it seems to be because the apostrophe in Jackson's is treated like a closing quote. I know I can escape a quote by doubling it if I was hard-coding the query... so
SELECT * FROM stores where storeName = 'Jackson''s Deli';
should work. But I'm not hard-coding the query. Instead it's being generated by user input and may or may not have an apostrophe in the name. How would I go about escaping ' this character in this case? I would need it to work inside Javascript/jquery.
Would I need to write an if statement that looks for ' in storeName and replaces it with '' ??
Or is there another way to go about this?
EDIT:
Ouch! Normally, yes, I realize the perils of generating a query on the client side.
So here's some more context. I'm working with cartodb and following their documentation. Here's an example from their repo doing something similar to what I'm talking about (they have other examples too):
https://github.com/CartoDB/cartodb.js/blob/develop/examples/layer_selector.html
You can't run a query in cartodb that lets you modify data in any way -- you can only run queries that let you retrieve data. So I'm still thinking about what the best way to escape this quote character would be.
DO NOT GENERATE SQL ON THE CLIENT SIDE... EVER
That being said, if you are going to use a dynamic query, you are best off escaping the user input and binding it to a prepared statement on the server side.
If you post more details about which database (MySQL, Postgres, etc.) and what language you are using for server processing- you will get better answers.
Yes... I am fully aware this doesn't answer the question. Nobody should be creating code this way though.
Edit: Made the warning bigger for emphasis.
I see others have answered but I wanted to approach this question from a few angles.
The question you're asking is a good one. You recognize that the SQL doesn't work with single quotes. You realize that something needs to be escaped. These are a good starting point for a few considerations that will hopefully help you to architect software in a secure and maintainable way.
Never directly execute client code/content - Generating SQL or any kind of code/instructions (javascript, bytecode, compiled code) from a client is always a poor idea because it breaks a few critical concepts.
It's hard to maintain because you cannot control the input fully. Sure you could escape the SQL but that doesn't fix both strange case scenarios where you have other characters you didn't account for.
It isn't secure - Your relationship to variables, inputs, CGI params, file contents, database fields whose values came from the aforementioned list, or just about anything that came from a remote system, remote user cannot ever be trusted. Always check, sanitize and validate inputs. I can open the source to your page, see where you add a check for single quotes and change that and then execute the code to delete your records, have it email if certain stored procedures are available, run code on the SQL backend, drop databases (assuming the query runs under appropriate privileges.)
It blends/blurs the lines between client input/display and business logic. Research MVC, n-Tier development and other concepts for an introduction to the concepts of separating your business logic from display/inputs. This is critical not only for scalability and performance but also to reduce the change of issues such as this from causing critical security flaws.
Approach your software development from the bad-guys perspective - Instead of "How can I escape this string to make it work." try "How can I bypass the escape on this page to allow me to delete records, view things I should, etc.
Don't feel bad because the approach is wrong,learn from it. I see alot of comments about how you should never ever do this (and they're right) but many of us learned this lesson the hard way. We laugh at Little Bobby Tables because we've all written or had to support code that did this. The key is to understand the underpinning of why it's a bad idea and then use that in designing software. Welcome to the school of hard knocks. We're all graduates and thankfully you could learn from our comments rather than when somebody tinkers and corrupts, deletes or infiltrates your database and application.
To get you started on this journey may I suggest reading the following:
SQL Injections Explained
And as an added bonus XSS E.g. escaping OUTPUT that originated from an external system or person. for example a comment entry that contains Hi!!! <script>alert('Thanks to this site not escaping this output I get to run this code under your login. Thanks for the 4000 crates of free tshirts you just ordered for me');</script> how are you??? so that when you output it you get
Comments:Hi!!! <script>alert('Thanks to this site not escaping this output I get to run this code under your login. Thanks for the 4000 crates of free tshirts you just ordered for me');</script> how are you???
Which is "valid" HTML and the browser will execute it.
Final thoughts - Adopt the motto Trust but Verify and you'll be OK
FYI, CartoDB does not allow you to execute a query that changes something in the table, it's read-only.
Send data to your server first, then escape all chars that need to be escaped with addslashes() command (provided that you are using PHP).
addslashes() command on PHP
After you are done with eascaping characters, you can send your data to cartoDB using their API and your API key.
cartoDB does provide insert/update/delete tasks through its SQL API. See this link:
http://developers.cartodb.com/documentation/sql-api.html
I have come to the point of entering the melee on web-scraping webpages using Javascript, with Python3. I am well aware that my boot may be making contact with a dead horse, but I feel like drawing my six-shooter anyway. It's a spaghetti western; be my gray hat?
::Backstory::
I am using Python 3.2.3.
I am interested in gathering historical stock//etf//mutual_fund price data for YTD, 1-yr, 3-yr, 5-yr 10-yr... and/or similar timeframes for a user-defined stock, etf, or mutual fund. I set my sites on Morningstar.com, as they tend to provide as much data as possible without necessarily requiring a log-in; other folks such as finance.google.com &c tend to be inconsistent in what data they provide regarding stocks vs etfs vs mutual funds.
The trade-off in using Morningstar for this historical data, or "Trailing Total Returns" as they call it, is that for producing this data they use Javascript.
Here are some example links from Morningstar:
A Mutual Fund;
An ETF;
A Stock.
I am interested in the "Trailing Returns" portion, top row or so of numbers in the Javascript-produced chart.
::Attempted So Far::
I've confirmed that wget doesn't play with Javascript; even downloading all of the associated files [css, .js, &c] hasn't allowed me to locally render the javascript in browser or in script. Research here on StackOverflow confirmed this. Am willing to be corrected here.
My research informed me that Mechanize doesn't exist for Python3. I tried anyway, and turned into Policeman Javert crying out "I knew it!" at the error message "module does not exist".
::I've Heard Of...::
->Selenium. However, my understanding is that this requires Thy Favorite Browser to actually open up a webpage, navigate around, and then not close because there's no "close this tab//window" command//option for Selenium. What if I//my_user want to get historical data for many etfs, stocks, and/or mutual funds? That's a lot of tabs//windows opening up in a browser which was not necessarily desired to be opened.
->httplib2. I think this is nice, but I'm doubtful if it will play with Javascript. Does it, for example using the .cache and get options?
import httplib2
conn = httplib2.Http(".cache")
page = conn.request(u"http://the_url","GET")
->Windmill. See 'Selenium'. I am, however, off-key enough to sing 'Man of La Mancha'.
->Google's webscraping code. Would an attempt at downloading a Javascript-laden page result in ... positive results?
I've read chatter about having to "emulating a browser without a browser". Sounds like Mechanize, but not for Python3 as I currently understand.
::My Question::
Any suggestions, pointers, solutions, or "look over here" directions?
Many thanks,
Miles, Dusty Desert Villager.
When a page loads data via javascript, it has to make requests to the server to get that data via the XMLHttpRequest function (XHR). You can see what requests they are making, and then make them yourself, using wget!
To find out which requests they are making, use the Web Inspector (Chrome and Safari) or Firebug (Firefox). Here's how to do it in Chrome:
wrench/tools/developer tools/Network (tab at the top of the tools)/XHR filter at the bottom.
Here's an example request they make in javascript
If you look closely at the XHR request url, you notice that all trailing returns have the same format:
http://performance.morningstar.com/Performance/cef/trailing-total-returns.action?t=
You just need to specify t. For example:
http://performance.morningstar.com/Performance/cef/trailing-total-returns.action?t=VAW
http://performance.morningstar.com/Performance/cef/trailing-total-returns.action?t=INTC
http://performance.morningstar.com/Performance/cef/trailing-total-returns.action?t=VHCOX
Now you can wget those URIs and parse out the data directly.
im building a web app in html5.. basically a form with a time counter and questions and answers.
im looking for a way that the user cannot change the score (that is calculated from the time took to answer the question) via browser debugger or etc.
encrypting the raw data sounds like an options.. but when the data is at dom, the user can change it.
i added some "time checking" in server side.. but still i would prefer some client side protection as well.
any suggestions? thanks
I'm no web pro, but I'd say just stick all the validation on the server side. From what I know about people exploiting MMORPGs, there is always a way to access/change client side data.
What you're asking for is impossible. No matter how you implement it, the user can use debugging tools to alter how the code runs in their browser - or, ultimately, just generate the HTTP POST request themselves, independent of your code.
Well, since you're saying you're using html5, why don't you just use the storage support?
e.g:
var store = sessionStorage.question= new Array();
store[0]="10s";
store[1]="5s";
Now just set that programmatically! It will last for the whole session
Put that in a file and import it and the better-than-average user wont know where to look!
You can also check This Link for a more robust solution
As Nick says, a determined user will be able to get round any encryption scheme you use on the client machine. At most you can make it difficult for them to break. You need to do two things, 1) encrypt so as to make tampering difficult and 2) try to detect any tampering that does occur.
I don't know what is available off the shelf for Javascript, if available then use AES for encryption and HMAC to detect tampering. If you have to write your own then use RC4 for encryption (not as strong as AES but much simpler to code) and a checksum to detect tampering.
One thing you can do to make it more difficult for an attacker to find your encryption key and HMAC key is not to store them in one place. Have two arrays such that the real key is array1 XOR array2. That way the actual key is not explicitly in code anywhere.
The Quest
I'm trying to talk to a SRCDS Server from node.js via the RCON Protocol.
The RCON Protocol seems to be explained enough, implementations can be found on the bottom of the site in every major programming language. Using those is simple enough, but understanding the protocol and develop a JS library is what I set out to do.
Background
Being a self taught programmer, I skipped a lot of Computer Science Basics - learned only what I needed, to accomplish what I wanted. I started coding with PHP, eventually wrapped my head around OO, talked to databases etc. I'm currently programming with JavaScript, more specifically doing web stuff with node.js ..
Binary Data?!?!
I've read and understood the absolute binary basics. But when it comes to the packet data I'm totally lost. I'd like to read and understand the wireshark output, but I can't make any sense if it. My biggest problem is probably that I don't understand what the binary representation of the various INT and STRING (char ..) from JS look like and how I convert from data I got from the server to something usable in the program.
Help
So I'd be more than grateful if someone can point me to a tutorial on these topics. Tutorial as in "explanation that mere mortals can understand, preferably not written by a C.S. professor". :)
When I'm looking at the PHP reference implementation I see (too much) magic happening there which I can't translate to JS. Sending and reading data from a socket is no problem, but I need to know how PHPs unpack function works respectively how I can do that in JS with node.js.
So I hope you can see what I'm trying to accomplish here. First and foremost is understanding the whole theory needed to make implementing the protocol a breeze. But because I'm only good with scripting languages it would be incredibly helpful if someone could guide me a bit in the HOWTO part in PHP/JS..
Thank you so much for your time!
I applaud the low level protocol pursuit.
I'll tell you the path I took. My approach was to use the client and server that already spoke the protocol and use libpcap to do analysis. I created a library that was able to unpack the custom protocol I was analyzing during this phase.
Its super helpful to start with diagrams like this one:
From the wiki on TCP. Its an incredibly useful way to visualize the structure of the binary data. Its tightly packed, so slicing it apart requires attention to detail.
Buffers and Binary
I read up on Buffer. Its the way you deal with Binary in node. http://nodejs.org/docs/v0.4.8/api/buffers.html -- the first thing to realize here is that buffers can be accessed bit by bit via array syntax, ie buffer[0] and such.
Visualization
Its helpful to be able to dump your binary data into a hex representation. I used https://github.com/a2800276/hexy.js to achieve this.
node_pcap
I grabbed https://github.com/mranney/node_pcap -- this is the equivalent to wireshark, but you can programmatically poke at all outgoing and incoming traffic. I added udp payload support: https://github.com/jmoyers/node_pcap/commit/2852a8123486339aa495ede524427f6e5302326d
I read through all mranney's "unpack" code https://github.com/mranney/node_pcap/blob/master/pcap.js#L116-171
I found https://github.com/rmustacc/node-ctype
I read through all their "unpack" code https://github.com/rmustacc/node-ctype/blob/master/ctio.js
Now, things to remember when you're looking through this stuff. Most of the time they're taking a binary Buffer representation and converting to a native javascript type, like say Number or String. They'll use advanced techniques to do so -- bitwise operations like shifts and such. You don't necessarily need to understand all that.
The key things are:
1) endianness -- the ordering of bits (network and host byte order can be reverse from each other) as this pertains to how things are unpacked
2) Javascript Number representation is quirky -- node-ctype goes into detail in the comments about how they convert the various number types in javascript's Number. Integer, float, double etc are all Number in javascript land.
In the end, its likely fine if you just USE these unpackers for your adventures. I ended up having to unpack things that weren't covered in these libraries, like GUIDs and such, and it was tremendously helpful to study the source.
Isolate the traffic you're looking at
Filter, filter, filter. Target one host. Target one direction. Target one message type. Focus on stripping off data that has a known fixed length first -- often times the header in a protocol is a good place to start. Once you get the header unpacking into a nice json structure from binary, you are well on your way.
After that, its one field at a time, top to bottom, one message at a time. You can use Buffer#slice and the unpack functions from node-ctype to grab each piece of data at a time.
I use DropBox and I've had some trouble reaching to my files from other computers:
I not always want to login to anything when I'm in a public computer, but I like being able to reach my stuff from wherever I am.
So I've made a simple application that when put in the public folder, ran and given the right UID, creates (still in your public folder) an HTML of all the content in the folder (including subfolders) as a tree of links.
But I didn't risk loading it anywhere, since there are slightly private things in there (yes, I know that the folder's name is "PUBLIC").
So I've came up with the idea to make it a simple login page, given the right password, the rest of the page should load. brilliant!, but how?
If I did this by redirecting to other HTML on the same folder, I'd still put the html link in the web history and the "url's accessed" history of the administrator. So I should generate itin the same page.
I've done it:
alt text http://dl.dropbox.com/u/3045472/validate.png
And currently the page is a textbox and a button, and only if you type in the right password (defined in the generator) the rest of the page (with the link-tree) loads. The fault is that everything (password, URL's) is easily reachable through the source code.
Now, assuming I only want to avoid silly people to get it all too easily, not make a bulletproof all-content-holding NSA certified website, I though about some ways to make these information a bit harder to get.
As you may have already figured, I use a streamwritter to write an html file (head, loop through links, bottom), then it's extremely configurable, and I can come up with a pretty messy-but-working c# code, though my javascript knowledge is not that good.
Public links in DropBox look like this:
Summarizing: How do I hide the URL's ande the password to show them (MAINLY the password, of course) in my source-code so that no that it should require some effort on reading ?
P.S.: It's not that personal, if someone REALLY wants it, it could never be 100% protected, and if it was that important, I wouldnt put it in the public folder, also, if the dude really wants to get it that hard, he should deserve it.
P.S. 2.: "Use the ultra-3000'tron obfuscator!!11" is not a real answer, since my javascript is GENERATED by my c# program.
P.S. 3.: I don't want other solutions as "use a serverside application and host it somewhere to redirect and bla bla" or "compress the links in a .RAR file and put a password in it" since I'm doing this ALSO to learn, and I want the thrill of it =)
Update 1:
The one answer so far gives a perfect way (according to this question) to hide my password.
Now I want a good way to hide the URL's, maby a code snippet of the example URL I gave being composed, and if it's too tricky, maby how to generate it in C#, or anything ?
Update 2:
I thought about maybe making three "obfuscating methods" and choosing them randomly in the runtime. So anyone who figures out how to read one XML, could only read about one third of them, and maybe having a hard time finding the other rest of this third..
Update 3:
Just thought about REGEX, the URL could be neatly crowded by dummy not-url-allowed characters added randomly that would be removed by something like:
regex.replace(url, ^[^\w\d/:-\.%]+$,"")
So the nosy dude should have to be pretty advanced into programming somehow, eh? could anyone tell me if it would work or not ?
Well, as it seems you already know, this is a rather poor choice of security mechanism, but if you insist...
Don't store the actual string in the source. Store, for example, its MD5 hash. Then, when the user types in a password, compute its MD5 hash and compare it with the expected one.
Check out:
MD5 in JavaScript
MD5 in C#
To elaborate on miorel's idea, you can also encrypt the whole page, using password as a key. Basically, encode all content into one big string, ask for the password and decrypt that string. If the password is wrong, it will show loads of rubbish, that is it. Like
content = "encrypted string"
function decrypt(str, key) { your algorithm of choice here }
document.write(decrypt(content, prompt('Password?')))
The only thing you need is a decrypt implementation in javascript - but that's easy to google out, for example here or here.
This also renders the separate 'login' page useless.
Granted, this is akin to asking how you can strip in public without people seeing you, but given that, I'm assuming that the password you are trying to store is the one to DropBox. I suppose you could obfuscate the password and store it in a cookie. That would at least prevent someone from simply viewing the source to see the password, but obviously wouldn't stop someone running something like Fiddler and seeing it.
[snipped server side suggestion]
EDIT: To munge the Urls, why don't you simply build the urls on the fly and have the links call a javascript function to get the url? Your server-side code would populate an array in this function with obfuscated urls and the calling code would simply pass an index into the array. Thus, on viewing the source, there would be no instances of "http" anywhere other than static unsecure links.
ADDITION Ok. now that I have a better bead on the problem, it is easier to devise solution. There are libraries for doing encryption on the net in javascript (e.g. http://point-at-infinity.org/jsaes/) but the problem comes down to key management. Since its javascript, it is going to be public but there are hoops you can devise to make it harder to determine the key. In general, those tricks involve indirection. For example, store a lengthy stream of random characters (e.g. 40-50 or more) that is generated by your C# code and stored in the HTM file. In addition, the C# code would would store into your javascript function an array numeric values that represent pointers into the long stream of text that were used by the C# code to encrypt the passwords (or just the whole url).