All my research so far suggests this can't be done, but I'm hoping someone here has some cunning ideas.
I have a form on a website which allows users to bulk upload lots of URLs to add to a list on the server. There's quite a lot of server-side processing to do on each URL, so to avoid timeouts and to display progress, I've implemented the upload using jQuery to submit the URLs one at a time using ajax.
This is all working nicely. However, part of the processing on each URL is deduplicating it against the complete list. The ajax call returns a status indicating either a successful upload or a rejection due to duplication. As the upload progresses, I tell the user how many URLs have been rejected as duplicates (along with overall progress and ETA).
The problem now is how to give the user a complete list of the failed duplicate URLs. I've kept them in an array in my jQuery, and would like the user to be able to click on a link on the form to download a text file containing those URLs. Is this possible just using client-side processing?
The server-side processing basically handles a single keyword at a time. I'd rather not have to store the duplicates in a database table with some kind of session key which gets sent with every ajax call, and is then used at the end to generate the text file server-side (and then gets cleaned up some time later). I can see how to do this, but it seems very clunky and a bit 20th century.
I haven't used it myself yet, but Downloadify was built for exactly this purpose I think.
Downloadify is a tiny JavaScript + Flash library that enables the generation and saving of files on the fly, in the browser, without server interaction.
It was created by Doug Neiner who is also pretty active on Stack Overflow.
It needs Flash 10 to work.
Related
I'm a total stranger to JavaScript and very new to Python. Please pardon my ignorance if any!
I have an internal website which allows me to download release reports. I have an idea to create a userscript to download the release report in a CSV.
This is how the flow is
User lands on a release report page
Makes a selection and clicks on a button
This shows release data in raw format (which is extremely crude and not user friendly). A download JSON button is present on the page which can let you download the data in JSON format
I observed on step 2 an API call is made with a release ID which is of course dynamic.
My plan is to
show a button download CSV when Download JSON button is available - I have completed this
Intercept the GET request whose request URL is dynamic and fetch the JSON
Flatten this JSON and create a CSV with whatever I need - I have this right now with python.
My questions are -
Is it possible to fetch network request headers - since the URL will be dynamic I have to rely on the last call the browser made? I know the format but a regex request URL will not work (I realised later how obviously silly that was)
My JSON normalising code is in python. Since the JSON itself is very dynamic and complex I used Pandas but I have no idea if I can even invoke it through a JavaScript.
Is there a better way to do this using just Python maybe? Maybe I shouldn't rely on a userscript, is there an alternative to work with a browser for this specific task?
I'm looking for any direction or even a possible different approach.
Many thanks.
I can't seem to get 'Requests' made by the browser. I'm exploring chrome.webrequests for this but not sure if I can integrate all these things together. I'm also thinking if selenium could ease any of these tasks.
New to k6, working with a web application that presents a spinner briefly on the home page while css and js files load.
Once the files are loaded and scripts are available, a login form is added (replacing the spinner).
With k6, is there a way to wait until a specific body element (the login form) is available in the body before continuing with the next step (ie. populating the username and pwd and submitting the form to login)?
Currently, when I review the response body, I see the spinner element only. Adding a delay does not appear to affect the body returned, even though the login form should, in theory, have been added to the page.
If the element is added to the body after the initial page load, will it be detected by k6 and made available in the response?
Thanks for your help.
Bill
k6 doesn't work like a browser - the load tests are written in JavaScript, but when you request an HTML file, the JavaScript in that file isn't executed. It usually can't be executed even with eval() or something like that, since k6 doesn't have a DOM or any of the usual browser APIs. So you have to explicitly specify any HTTP requests you want your k6 scripts to make, and in your case I assume that the spinner and login form are generated by a JavaScript somewhere in the home page.
To simplify working with such highly dynamic websites when you use k6, you can use the site normally in your browser, record the browser session as a .har file and export it, and then convert that .har file to a k6 script with the k6 convert command like this: k6 convert session.har -O k6_script.js. You can find more information about the whole process here.
k6 doesn't execute client side code, nor does it render anything. It makes requests against the target system and loads them. This makes it efficient to make a large number of reqeusts, but creates new things that must be solved in certain cases.
Capturing all the requests necessary - typically using the k6 convert to convert a HAR file works well to give a foundation of a script. I suggest using the other options in converting to limit any third party requests. e.g. --only or --skip. More info here: https://support.loadimpact.com/4.0/how-to-tutorials/how-to-convert-har-to-k6-test/
Since you recorded your browser session, if your application/site uses anything to prevent CSRF attacks, you must handle those values/correlate them. e.g. .NET sites use VIEWSTATE, if you were testing a .NET app, you would need to instruct the VUs to extract the viewstate from the response body and reuse it in your requests that require it
In a similar vein to point 2, if you are submitting a form, you probably don't want to utilize the same details over and over again. That typically just tests how well your system can cache or results in failing requests (if you are logging in and your system doesn't support concurrent logins for the same user as one example). k6 is able to utilize CSV or JSON data as a source for data parameterization. You can also generate some of this inline if it's not too complex. Some examples are here: https://docs.k6.io/docs/open-filepath-mode
i have a bunch of websites of mine that need to call an external server asking for some data. This server contains sensible information such as discount codes that need to be applied on the website, based on what the user is doing.
Since i can't do a synchronous ajax call on users' actions (deprecated and bad practice), i'm doing it async everytime a user loads the page on these websites.
The call retrieves all the possible discount codes but i'd like people not being able to decode it and get them for free.
The server creates a json array and encodes it as base64, then the JS in the user browser will decode it (atob() function) giving me the json i needed to work with.
I'd like to apply an encryption to this like an algorithm for letters swapping or something. This means that even if the user reads my json result is not going to be able to read it BUT he could read it when my JS code tries to decode it (pausing with the debugger).
So my question is: could javascript obfuscation solve my issue? My JS would contain the algorithm to take my string back to its original form but the whole code is obfuscated therefore debugger won't work. Is that a safe approach to the matter? If not, how could i do it safely?
Summing it up, the system works like this:
1.User loads page in website
2.JSONP AJAX call asks all the available discount codes from my server
3.Server gets the list, makes a JSON of it and returns a base64 string of this json
4.Client gets this and when he clicks on something particular i decode the json and apply the correct discount based on calculations.
I hope it's clear enough for you to give some suggestions :)
thanks!
If you make your decryption on the Client machine, then there is no way of making it secure. You could apply obfuscation, or other means of hiding your process, but ultimately there is a point in the code which makes the decryption, and is available to the client. You should not do that on the client side.
You could maybe do this:
Website loads
It sends request for promo codes, along with the user details.
You send only data that is available for that user.
In this way you do not need to encrypt anything, but it will involve some rethinking and change to server side logic.
There is maybe another solution, which is hacky, and also requires server change.
Send the codes as you already send them, but send them incomplete. Maybe remove last 2 or more digits. Then the user requests to activate promotion, and sends the incomplete code. Then you look in your table code that starts (like '%'), and create the working code on the server side.
I've coded some php which retrieves facebook user ids, stored in my database, and then goes on to request their user names. It all works fine, however this process adds an additonal 1.8 seconds to my pageload, which I would like to avoid if possible.
So I rebuilt the code, and added some javascript which looks for the user ids and then requests the user names. This method runs a lot faster and my page load is 1.8 sec shorter. However in order to make it work I'm adding my app access token to the javascript, which ofcourse is a security black hole
The javascript fb.api call looks like that at the moment
FB.api('/'+userid, {access_token : 'appAccessToken'}, function(response) {
userNameSpan.text(response.name);
});
Is there any other way I can make this work, without falling back to the pure server-side solution which is awfully slow?
I'm getting no answers at all to my questions lately. Anyway, the only other solutions I can think of are ajax calling the php file, which will be requesting the user names itself, or storing the user names in the database along with the user ids.
The ajax solution is problematic for a lot of reasons, mainly because the platform is joomla and the php is a custom module, which means that I will have to either call the file outside of the joomla framework (security issues arise) or build my own helper file for the custom module in order for it to work with the com_ajax component of joomla, which ofcourse is a timesink.
The database solution has the downside that the usernames will not be dynamic, meaning that if a user chnages his/her user name after he/she has been added in the database, the change will not be available to me. I guess I could also store the time the user was inserted in the database and check how much time has passed, and perform a facebook request for his/her username only after a predefined time has passed, and then updating the database if the username has changed. Not quite dynamic, but close enough
If any1 has any other ideas, please don't be shy
Although I'm gonna go for the database solution, I found another solution that can decrease the pageload significantly.
Instead of making an api call for each user, you can do a batch call, requesting info about many users in one call. It is explained here https://developers.facebook.com/docs/graph-api/making-multiple-requests
That would cut down my calls from 6 to 1. However since reading up a bit about facebook names, I realized they do not change that often any more, so the database solution is better for my case
I need to upload a file with a form in a single-page (AJAX) web application, but the file is useless without the rest of the data on the form in that panel. There are only three INPUTs, but one can be a quite lengthy text area. How can I capture this?
If I upload the file using an isolated AJAX file-upload technique (like the Fine Uploader http://fineuploader.com/ widget), then I have to handle two counter-intuitive elements:
CLIENT: The user transmits the file, the main part of the transaction, before actually approving the transaction. They then wait for this to complete, even if they decline to continue. The UI must disable screen elements to prevent the scenario where a client might submit the associated data before or during the file upload. It's extra effort to send the file at the wrong point in the process.
SERVER: It requires a ticketed-cache. The back-end must cache the uploaded file and provide a ticket to the client for this file. The client must send this ticket with the upcoming request. Ideally, the cache also should clean up old tickets under various circumstances, such as if the form is cancelled, another file is uploaded, or the user session times out. More extra work (although this ticketed-cache functionality would be nice to have in my server).
Is it a sensible solution to instead place the whole form in an IFRAME? Will I have problems manipulating that and making it appear to be a well-integrated part of the single-page application? I've always stayed away from them in the past.
My platform is jQuery, ASP.NET MVC, the usual browsers (plus probably mobile).
This was pretty easily resolved in my case by simply setting the target of the form post, including file and text inputs, to a non-visible iframe, and watching for the iframe onload event.