My workplace keeps client records in a database which is accessible via an aspx application hosted on our local network. The page allows logging in, simple searching, and retrieving of information, which is displayed on this webpage. A task I am performing requires I access each record, copy a block of text into a new database which we are creating, and move to the next record (thousands of times).
My question is, is there a way to automate this? I've looked at browser macros, but the task of searching for each record does not allow for a simple macro interface. I was hoping to write a program which directly interacts with the aspx application without the need for the browser. In the code I'd be able to tell the application which records i'm looking for (by date, or unique ID, for example), and have the aspx application return the appropriate fields of text I need. I don't have any back-end access to the aspx application (it is a third party, proprietary application and their developers are not interested in helping me with this task)
Can this be done?
If yes, in broad terms, what sort of solution should I be looking at? Is there a particular language that is more suitable to this task?
Any help is appreciated. I realize the question is a bit vague and broad, but the third-party application is only hosted locally so I can't give an example. I know a little ruby, and came across the gem Mechanize, which looks like it might work... but I can't find much about interacting with asp.net
You're on the right track with ruby mechanize. Mechanize will keep track of the cookies for you and make it easy to submit forms (logging in, asp doPostBack actions). Give it a try, and if you get stuck, post specifics in a new question.
Related
I'm attempting to create my first ever full-blown website. The site will host a form that takes a user's link and scrapes the inputted website. Besides scraping, I'm also looking for click and input functionality (for future projects).
I managed to create a PhantomJS function that scrapes a website depending on the parameters. However, I have absolutely no idea how to implement this into my Xampp server which locally hosts my website...PhantomJS only runs off of a command line as far as I'm aware.
Is it even possible to host a PhantomJS program on a website? Should I be using something else, such as NodeJS or another language? Ultimately, I'll go learn anything in order to make this happen.
Any ideas/suggestions are much appreciated. I apologize if this question seems dumb.
I'm just now tapping into real GUI-heavy stuff and I'm looking to integrate a GUI for a Python 3 application/("game"?). I need this to be cross-platform, responsive, and able to communicate with other users and a server.
I've looked at desktop application creation but I think that the best thing would be a web application. Now, here are the obstacles:
I need the web app to be able to use all my Python 3 code as its black box; it needs to be able to send a function call to the running python code.
More often than not, everything on the page is changed if a single value is changed.
If another user changes a value, it would affect multiple other users instantaneously.
I can't have the page constantly refresh.
The HTML "GUI" needs to have some degree of separation from the Python code. (I can't have Python sending snippets of HTML, only data is allowed.)
I realize that this is a lot to address, so my question then is:
How would one go about making a web app that has specific (most) data values update consistently without having to refresh the entire page?
My proposed solution would have some sort of simplistic javascript constantly checking to see if anything has been posted for the users closely interacting with each other (players in the same server, if you will). Each value in the HTML would have a linked javascript function which is linked to a value or function in the backend python code.
"Player" changes value on webpage -> JavaScript function call -> Python black box on "Server" -> JavaScript checking loop sees that there has been a change via simple Python variable on "Server" -> JavaScript gets all values via calls to functions in python code -> Update HTML for all "Players"
Finally, please note, I have no idea how to carry out my proposed solution, and I very much welcome a different, better approach. Thanks in advance.
So, for anyone who is trying to do the same thing as me, the answer is something called "websockets".
I initially tried Kivy, but found that it was too cumbersome and just hard to get good styling in. Using HTML, CSS, and JavaScript, you can make pretty fancy looking stuff, and Kivy just isn't robust enough to make something attractive without having to write tons of excess code. Since what I'm trying to make is basically a styled data readout/hud, the HTML combination takes the cake.
What I'm currently using for the websockets is a module called "Tornado" which allows you to update any data on the client side without them reloading the page. I'm sending this data as a single JSON to update the whole page.
In conclusion, I decided on HTML/CSS/JavaScript for the GUI and Python/Tornado for fast server-side computation.
I am just getting started couchdb and have been looking into writing couch apps. the idea is pretty cool. From what I can tell so far, its great for writing front facing apps, but what I have not been able to tell yet is securing data and your code.
For example, if I have an app on something like iriscouch, it seems like the whole database is replicatable to the world. which is fine for those things. But if I were to host something on something like cloudant (or self hosted) where I dont want joe schmoe to replicate my db (assuming I have it at my own domain), is there a way to secure and not let all of your source code out?
Maybe its just my understanding so far. But when you have javascript you can always see the source. Is that any different when building couch apps? Such as if I wanted to keep the back end js code proprietary.
When building front ends, the client code obviously doesnt matter. But Im wondering if is possible to use the html/js to build apps where I can keep the back end proprietary. Or is this just out of scope of couchapps?
CouchDB has very primitive access control. It is not possible to build a security minded web application, like a shopping cart, using this access control system. In order to enforce more complex access control rules you need to build a middle-ware between JavaScript and CouchDB. In this case the user logs into the middle-ware, and this code is responsible for keeping track of who the user is and what resources they have access to.
When I say "Vanilla Coding", I am referring to websites that don't utilize server side coding (such as PHP, ASP, etc.), only HTML, JavaScript, and CSS.
I know that there are a plethora of sites that already exist that don't utilize (to my knowledge) any of the common, server side languages used by many others (PHP, ASP, etc.), but still function just fine!
I am confused! How do these sites continue to save login information, keep records, etc. etc. without using a server side scripting language? Is there something that I am missing? Can JavaScript access more (such as databases and local files) than what I thought it could?
EDIT
Turns out I've made a serious and shameful mistake in assuming that just because it ended with a .html extension that it was client-side only. That is okay though because I'm learning. Thanks so much for the help everybody!
Essentially, unless you have some sort of server-side programming, you don't stand a chance at making a site with any amount of functionality. To break it down for you:
What you can do without server-side scripting:
Serve static pages
What you need server-side scripting for:
Absolutely everything else
Even something so simple as keeping a site consistent and up to date is a nightmare on wheels without, at the very least, some some sort of management system that pre-generates the static pages to be served. (Technically, one could argue that Copy+Paste in Notepad counts as this.)
As has been mentioned elsewhere; obfuscating the true nature of precisely what system is being used is trivial; and having URLs ending in, say, .html while using PHP is no issue.
Edit: In the most perverse case I can think of off the top of my head, you could have a lighttpd server masquerading as an IIS server, serving pages generated by an offline renderer fed to it by a Perl FastCGI script, sent together with PHP signature heading and using a mix of .asp and .jsp file extensions.
Of course, noone would do something as silly as that. I think…
No client side script can access server side information (like a database) without some sort of server side communication (through something like ajax or the like)
If you really ( i mean really as in don't do it ) want to do logins and the like on clients side, you would have to make some sort of cookie that you store on the user's computer, also you would need a list of users (which anyone can read) to use against
This answer is very late but I leave this reply for anyone who may stumble upon it.
Using javascript/jQuery, and various APIs a simple site can be created only using client-side coding.
For instance, a simple shopping cart type of site can be created. I've done it before.
There are few (not many) strictly 100% jQuery based shopping cart solutions that are open-source.
How does the PG (pay gateway) get taken care of? You are limited to accepting payment through paypal, google checkout, and direct deposit.
What about allowing customers to leave comment? You can use API's like Disqus. What about chat support? Zopim is pretty handy.
How do you get notified when purchase is made? Paypal & google checkout notifies you.
What about sending mass email? Mail Chimp.
Personally, I almost always use WordPress or some other types of CMS but using only vanilla coding to build a simple site is not only feasible but very sensible in certain circumstances.
You're not going to see whether a site is using a server side language unless they let you see the file extensions. With URL rewriting, MVC patterns, etc., it's easy to hide, or even fake that information. Therefore, chances are very good that the sites that you think aren't using a server side language are actually using one.
Now, a site can save certain information in cookies, such as some basic preferences, but any authentication they appear to be doing wouldn't actually be doing anything without a server-side script accessing a database somewhere.
As a side note - I have worked on a site where the content was actually static, but made to look like a blog or CMS. It was an absolute nightmare and hugely error-prone.
What are these sites that you think aren't using server-side scripting?
Nowadays a lot of sites are using Javascript as a server side solution, Node.js being the most popular. Check out this list: https://github.com/joyent/node/wiki/Projects,-Applications,-and-Companies-Using-Node
Preamble
So, this question has already been answered, but as it was my first question for this project, I'm going to continue to reference it in other questions I ask for this project.
For anyone who came from another question, here is the basic idea: Create a web app that can make it much easier to create other web applications or websites. To do this, you would basically create a modular site with "widgets" and then combine them into the final display pages. Each widget would likely have its own set of functions combined in a Class if you use Prototype or .prototype.fn otherwise.
Currently
I am working on getting the basics down: editing CSS, creating user JavaScript functions and dynamically finding their names/inputs, and other critical technical aspects of the project. Soon I will create a rough timeline of the features I wish to create. Soon after I do this, I intent to create a Blog of sorts to keep everyone informed of the project's status.
Original Question
Hello all, I am currently trying to formalize an idea I have for a personal project (which may turn into a professional one later on). The concept is a reflective web application. In other words, a web application that can build other web applications and is actively used to build and improve itself. Think of it as sort of a webapp IDE for creating webapps.
So before I start explaining it further, my question to all of you is this: What do you think would be some of the hardest challenges along the way and where would be the best place to start?
Now let me try to explain some of the aspects of this concept briefly here. I want this application to be as close to a WYSIWYG as possible, in that you have a display area which shows all or part of the website as it would appear. You should be free to browse it to get to the areas you want to work on and use a JavaScript debugger/console to ask "what would happen if...?" questions.
I intend for the webapps to be built up via components. In other words, the result would be a very modular webapp so that you can tweak things on a small or large scale with a fair amount of ease (generally it should be better than hand coding everything in <insert editor of choice>).
Once the website/webapp is done, this webapp should be able to produce all the code necessary to install and run the created website/webapp (so CSS, JavaScript, PHP, and PHP installer for the database).
Here are the few major challenges I've come up with so far:
Changing CSS on the fly
Implementing reflection in JavaScript
Accurate and brief DOM tree viewer
Allowing users to choose JavaScript libraries (i.e. Prototype, jQuery, Dojo, extJS, etc.)
Any other comments and suggestions are also welcome.
Edit 1: I really like the idea of AppJet and I will check it out in detail when I get the time this weekend. However, my only concern is that this is supposed to create code that can go onto others webservers, so while AppJet might be a great way for me to develop this app more rapidly, I still think I will have to generate PHP code for my users to put on their servers.
Also, when I feel this is ready for beta testers, I will certainly release it for free for everyone on this site. But I was thinking that out of beta I should follow a scheme similar to that of git: Free for open source apps, costs money for private/proprietary apps.
Conceptually, you would be building widgets, a widget factory, and a factory making factory.
So, you would have to find all the different types of interactions that could be possible in making a widget, between widgets, within a factory, and between multiple widget making factories to get an idea.
Something to keep on top of how far would be too far to abstract?
**I think you would need to be able to abstract a few layers completely for the application space itself. Then you'd have to build some management tool for it all. **
- Presentation, Workflow and the Data tier.
Presentation: You are either receiving feedback, or putting in input. Usually as a result of clicking, or entering something. A simple example is making dynamic web forms in a database. What would you have to store in a database about where it comes/goes from? This would probably make up the presentation layer. This would probably be the best exercise to start with to get a feel for what you may need to go with.
Workflow: it would be wise to build a simple workflow engine. I built one modeled on Windows Workflow that I had up and running in 2 days. It could set the initial event that should be run, etc. From a designer perspective, I would imagine a visio type program to link these events. The events in the workflow would then drive the presentation tier.
Data: You would have to store the data about the application as much as the data in the application. So, form, event, data structures could possibly be done by storing xml docs depending on whether you need to work with any of the data in the forms or not. The data of the application could also be stored in empty xml templates that you fill in, or in actual tables. At that point you'd have to create a table creation routine that would maintain a table for an app to the spec. Google has something like this with their google DB online.
Hope that helps. Share what you end up coming up with.
Why use PHP?
Appjet does something really similar using 100% Javascript on the client and server side with rhino.
This makes it easier for programmers to use your service, and easier for you to deploy. In fact even their data storage technique uses Javascript (simple native objects), which is a really powerful idea.