Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
Aloha, Stackoverflow.
I frequently come across web applications, and wonder to myself, "How could I write a script/application which would interface with that?" (purely academic, not for spamming purposes!).
For example, the website Omegle; people have written Python scripts to interface with the website and run a chat without opening the browser... how? I will admit that WEB programming is not my strongest area, but I would really like to know how one could extract the protocol being used from such applications, and use this knowledge to create custom apps and tinker with the service.
So basically, how can I figure out the inner workings of a web app (ie. imeetzu.com such that I can write code to interface with it from my desktop?
Thank you in advance!
You'll need a set of tools to start with:
A browser with a debugging window (Chrome is particularly good for this). This will allow you in particular to access the network calls that your browser directly makes (there's a caveat coming), and to see:
their content
their parameters
their target
A network packet sniffer to trace down anything that goes through Flash (or WebSockets). I'm quite fond of Ethereal (now called Wireshark), though if you're in the US, you could be breaking the law by using it (depends on the use you make of it). This will allow you to see every TCP frame that enters and leaves your network interface.
The knowledge you will need:
Ability to identify and isolate a network stream. This comes through practice
Knowledge of the language the app you are trying to reverse-engineer is written in. If JavaScript isn't your cup of tea, avoid JS-based stuff
Maths and cryptography. Data may very well be encrypted/obfuscated/stegg-ed from time to time. Be aware and look out for it.
In this particular case, looks like you might have to deal with Flash. There are additional resources to help on this, although all of them are non-free. There is one particularly good Flash decompiler called SoThink SWF decompiler, which allows you to turn a SWF into a FLA or a collection of AS sources.
That's all for the tools. The method is easy - look what data comes in/out and figure out by elimination what is what. If it's encrypted, you'll need IVs and samples to hope to break it (or just decompile the code and find how the key/handshake is done). This is a very, very extensive field and I haven't even touched the tip of the iceberg with this - feel free to ask for more info.
(How do I know all this? I was a contributor to the eAthena project, which reverse-engineered a game protocol)
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I'm a C# developer, and use XAML for user interfaces. Lately I've been wondering something regarding HTML5+JavaScript development (used in Universal Windows App development, as well as for websites). Javascript is compiled and executed at run-time on the client device. So any user can go into the folder where they're stored on computer, and see all the code in it, right?
There is no unreadable alternative for js and html5. That's why so many websites are so slow in adopting html5 video and replacing adobe flash.
But if your entire application is client side and you worry about your code being stolen you're doing something wrong. Almost any application requires serverside code that isn't accessible.
And it doesn't matter anyway, who cares about some js that makes a div draggable or moves some html around.
I dont think readable javascript code is of any value...what matters is the server side code like php or ASP which really matters in the security of the websites
And even if the developer didnt want the user to read the javascript framework.. what option does he/she have to prevent it..?none!
The client side code is indeed visible by the client. If something is available client side, then you won't need a round trip to the server to get it.
For example you could imagine a simple calculator application. You could write it client-side, in Javascript, the app can ouptut the calculations immediately. Or you could write it server-side (in wathever language you want), which means you need to ask the server for the calculation (with an ajax request probably), and wait for it to respond.
Also some things doesn't make sense on the server-side. Pretty much any action that changes the DOM, which only exists client-side, in the browser.
I wrote about this on my blog a while back, see Protecting Your Code,
as an addendum to my free ebook, Programming Windows Store Apps with HTML, CSS, and JavaScript, 2nd Edition.
The short of it is that JS code it not protected, though you can make things a little more difficult with minification/uglification like many website authors do. You can also take steps by putting some of the code you care about into Windows Runtime Components written in C++ (C# can be decompiled). The only really secure solution is to have code on a server, and draw from that in an app which of course doesn't work for all cases, but is an option.
Note that some of my comments in that blog from 3 years ago might be a little dated. I believe that current Windows Store policy now allows you to load code from a remote server at run time.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
How can I prevent the user running my web page after downlading the page source?
We can't prevent downloading the source code, but can encrypt it.
But it doesn't good enough for me, because the encrypted code can work after downloading.
Thank in advance.
As a general rule, if the code runs on a remote machine it can be manipulated so they can execute it anyway.
You can make this more difficult through code obfuscation or by implementing some sort of DRM, but I would suggest that this will largely be more trouble than it's worth (since it just takes one person to break it and your code is back out).
1) You could, for example, require that some key be downloaded from a site you control before it'll execute, but the recipient might simply sniff their traffic and pass that value to the game themselves.
2) Or you could possibly set up your game to stream each of the levels or some important aspect of it to your game client, but again, there's not a whole lot stopping someone from just reading these aspects and implementing this mechanism themselves.
3) Perhaps you could encrypt these level packages dynamically on the server with a time-based key, but it just takes that one bored programmer with the technical know-how to reverse-engineer what your method is.
4) Another option that comes to mind is requiring some regular polling to a server you control and requiring some sort of response, but again, if your client can predict what this response is supposed to look like, it's easy for someone to rewrite the game to talk to their own program instead of your server.
5) You could also daisy chain a ridiculous number of dependencies of your javascript logic (breaking your own code into a number of dependencies) so it's slightly more difficult for another user to rebuild the required paths on their system. This might be useful to put off a casual user, but I doubt it'd put off a more knowledgeable user.
All in all, I'd suggest that you simply make the game available as is. Various game companies larger than you have attempted to implement DRM measures of their own with disastrous results (when they don't work as advertised) or just plain annoying for the end user.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
we are currently evaluating the use of AngularJS in a Enterprise Application (e-Banking) as a Single Page Application.
Many of the devs#work are already convinced that there no other way to go. It is the trend, it is future proof (html, css and js) it is easy to do, less burden on the server etc etc.
Despite the interesting part of this framework, i am not convinced that there are factors that must be considered before going this road. These can be:
Maintainability of code
Testability of code, not only the ui part
Continuous integration (like TeamCity or TFS)
Developer friendliness like debugging, navigating through code
Security (if there is a risk)
Has anyone any experience on enterprise apps build like that? I would rather go with Asp.Net MVC4. Please no hypothetical answers, i don't want to start a war. Real life experience is really appreciated.
Regards
I had thought about this when incorporating Angular into our enterprise application. First, some optional background information. I really needed to keep all my page state on the client side since recreating it on the server side is a very resource intensive task. I started off with building it with just JQuery, since I only needed a few things, but soon I found myself having trouble keeping track of the callbacks I wrote just last week. Hence, I wanted to refactor. While I was at it, Angular looked like the best fit for my programming style.
To address your concerns:
I think that it is more maintainable, since it encourages you to split things up. This, however, depends totally on your team and your discipline to keep things maintainable.
The Angular team definitely thought about testing when they were making it. Things are very easy to test, from unit testing to e2e testing. Others have even found a middle ground in between.
We use Jenkins at our Microsoft shop for continuous integration since it was easier for us to plug in our own programs to make it work with our complex environment setups. However, I don't think this relates much to which framework you decide to go with.
Coming from Visual Studio, I find JavaScript in general leaves something to be desired. I love edit and continue, unwinding the stack, and dragging that yellow arrow around. That stuff is just pure magical bliss. Debugging on the browser just doesn't give you all of that.
Security is pretty good. Looking through the docs, you can see that the team is concerned with security issues such as XSS and CSRF attacks. However, anything done over the browser is hard to secure. There are new attacks and bugs discovered every other day. If you are really concerned about security, you wouldn't have an online app. You would make sure that your customers only banked in the branch with multiple forms of ID. With that said, I don't think Angular is any less secure than any other JavaScript solution.
I would like to raise another issue with all of these JavaScript frameworks/libraries. They're all fairly new and constantly evolving. Look at how many functions JQuery has deprecated recently, and you'll get a sense of how hard it is to keep on top of your libraries. Many of these are also forgotten and left to rot. Angular being one of Google's products also makes me a little scared, since they have been cancelling waves of their projects. Hopefully, Angular does not become one of those.
The documentation is also lacking. Although the community is good, the documentation is not. Many pages are half done. Again, compared to the latest stuff from Microsoft, the documentation is pretty sparse.
This being one of the new things, it is harder to hire someone to help/take over this stuff.
Despite all these drawbacks, I really like it, and I'm definitely going to push my company towards adopting it.
Yes I have used Angular with Asp.net WebAPI along with MongoDB .Pls find my findings
1)Testability is absolutely fantatic .You can easily test you unit test using karma and jasmine
2) CI. Integration of Jenkins is available with Karma
3) Easy to debug similar to Javascript debugging can use chrome extension Bartrang to debug angular apps easily
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any way for me to make values generated by a separate program available to JavaScript?
I'm fairly new to web design and JavaScript, so I'm not really sure if it is possible to do what I want at all. I know some of this is intentionally made difficult to do (prevented?) due to security issues. I don't need an actual written out solution so much as a confirmation that is or isn't possible, and maybe the appropriate API/function sets that would be used for it.
I want to have a web page which uses third party JavaScript API functions based on values being generated in a separate program on the device accessing the website, which is talking to external devices. I want this to be an ongoing process though; once the webpage is open it should continue to monitor the values and update the calls as appropriate.
Some sort of direct interaction would be preferable (is there a way to pass in a callback function or something?), but I haven't seen anything so far which looks like it could do that.
Is it possible to do something like write the values to an XML file on the program side and have a loop which repeatedly re-reads the file out on the webpage side? I don't know that this would be fast enough for what I want.
I'm not worried about compatibility with a large range of browsers or anything, so even a solution which uses something like a rarely supported HTML5 syntax would be fine as long as I can get one browser that it works on.
You can have your external application expose some sort of web interface on a server and then have the client side perform AJAX requests via Javascript to pull new data from the web interface.
This is actually a very, very good question. Some people may have skimmed it and to them it would look like lots of other newbie-misunderstands-the-web type questions. But this is actually different:
I want to have a web page which uses third party javascript API functions based on values
being generated in a seperate program on the device accessing the website, which is talking
to external devices
The bolded part is what makes this different.
Yes it's possible. It's how Flash and Java plugins can have access to your webcam and microphone. But it's not easy.
Firstly, this is not your usual server-based interaction (if it was, it would be much simpler and lots of web devs would be rightfully annoyed at yet another newbie asking such questions). You basically want your web app to have access to hardware on the user's machine.
I'm not worried about compatibility with a large range of browsers or anything
Good, that makes it a bit simpler. Because the web APIs don't handle this sort of thing at all. You need low level access to the browser.
You basically have 2 options:
If Flash or Java supports accessing the hardware you're trying to access then you can try writing an applet in either Flash or Java. Of the two, Java is slightly more friendly to programmers wanting to do low level, non-standard stuff. But flash is often more friendly to users (for some reason, Java applets to this day tend to be huge and take a long time to load).
If you can't do what you want in Flash or Java then you really only have one other option: write a browser plugin. For this there are two options (note: I've never done this before so I'm only going by what I've read). First is you can develop a plugin based on the old Netscape Plugin API. Google around to learn more about it. For IE on windows you have a second option of developing an ActiveX plugin. you can try to google around for that as well.
A bit of googling led me to Firebreath which is a cross-browser plugin framework that can compile to both NPAPI and ActiveX.
Oh, there is a third option. For Chrome and Firefox you can write javascript based plugins. Though like ActiveX on IE, the plugins won't be cross-browser compatible. I'm not even sure if the javascript plugin API can access really low level hardware.
What you're trying to do is interesting. Not many people have done it. But it's doable. Good luck.
I like Ivan's idea. Here's how I would do it:
In your external program, start up a web server that can accept WebSocket connections. Choose a very high port that isn't used standardly. For doing WebSockets, I like node.js and Socket.IO, but WebSocket libraries exist for Java, Python, Ruby, etc.
Then have your webpage open a WebSocket to your locally running web server which can very quickly stream data to your javascript app. See the examples on Socket.IO to see how very easy it could be. Every time your external program got a new piece of data, it could push it to your webpage, and a javascript callback would be fired with the new data.
UPDATE:
Since your other program is written in C++, this question and answers might be interesting: https://stackoverflow.com/questions/3916217/standalone-c-websocket-server-library.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I was wondering what would be the most ethical way to consume some bytes (386 precisely) of content from a given Site A, with an application (e.g. Google App Engine) in some Site B, but doing it right, no scraping intended, I really just need to check the status of a public service and they're currently not providing any API. So the markup in Site A has a JavaScript array with the info I need and being able to access that let's say once every five minutes would suffice.
Any advice will be much appreciated.
UPDATE:
First all thanks much for the feedback. Site A is basically the website of the company that currently runs our public subway network, so I'm planning to develop a tiny free Android app for anyone to have not only a map with the whole network and its stations but also updated information about the availability of the service (and those are the bytes I will eventually be consuming), etcétera.
There will be some very differents points of view, but hopefully here is some food for thought:
Ask the site owner first, if they know ahead of time they are less likely to be annoyed.
Is the content on Site A accessible on a public part of the site, e.g. without the need to log in?
If the answer to #2 is that it is public content, then I wouldn't see an issue, as scraping the site for that information is really no different then pointing your browser at the site and reading it for yourself.
Of course, the answer to #3 is dependent on how the site is monetised. If Site A provides advertistment for generating revenue for the site, then it might not be an idea to start scraping content, as you would be bypassing how the site makes money.
I think the most important thing to do, is talk to the site owner first, and determine straight from them if:
Is it ok for me to be scraping content from their site.
Do they have an API in the pipeline (simply highlighting the desire may prompt them to consider it).
Just my point of view...
Update (4 years later): The question specifically embraces the ethical side of the problem. That's why this old answer is written in this way.
Typically in such situation you contact them.
If they don't like it, then ethically you can't do it (legally is another story, depending on providing license on the site or not. what login/anonymousity or other restrictions they have for access, do you have to use test/fake data, etc...).
If they allow it, they may provide an API (might involve costs - will be up to you to determine how much the fature is worth to your app), or promise some sort of expected behavior for you, which might itself be scrapping, or whatever other option they decide.
If they allow it but not ready to help make it easier, then scraping (with its other downsides still applicable) will be right, at least "ethically".
I would not touch it save for emailing the site admin, then getting their written permission.
That being said -- if you're consuming the content yet not extracting value beyond the value
a single user gets when observing the data you need from them, it's arguable that any
TOU they have wouldn't find you in violation. If however you get noteworthy value beyond
what a single user would get from the data you need from their site -- ie., let's say you use
the data then your results end up providing value to 100x of your own site's users -- I'd say
you need express permission to do that, to sleep well at night.
All that's off however if the info is already in the public domain (and you can prove it),
or the data you need from them is under some type of 'open license' such as from GNU.
Then again, the web is nothing without links to others' content. We all capture then re-post
stuff on various forums, say -- we read an article on cnn then comment on it in an online forum,
maybe quote the article, and provide a link back to it. Just depends I guess on how flexible
and open-minded the site's admin and owner are. But really, to avoid being sued (if push
comes to shove) I'd get permission.
Use a user-agent header which identifies your service.
Check their robots.txt (and re-check it at regular intervals, e.g. daily).
Respect any Disallow in a record that matches your user agent (be liberal in interpreting the name). If there is no record for your user-agent, use the record for User-agent: *.
Respect the (non-standard) Crawl-delay, which tells you how many seconds you should wait before requesting a resource from that host again.
"no scraping intended" - You are intending to scrape. =)
The only reasonable ethics-based reasons one should not take it from their website is:
They may wish to display advertisements or important security notices to users
This may make their statistics inaccurate
In terms of hammering their site, it is probably not an issue. But if it is:
You probably wish to scrape the minimal amount necessary (e.g. make the minimal number of HTTP requests), and not hammer the server too often.
You probably do not wish to have all your apps query the website; you could have your own website query them via a cronjob. This will allow you better control in case they change their formatting, or let you throw "service currently unavailable" errors to your users, just by changing your website; it introduces another point of failure, but it's probably worth it. This way if there's a bug, people don't need to update their apps.
But the best thing you can do is to talk to the website, asking them what is best. They may have a hidden API they would allow you to use, and perhaps have allowed others to use as well.