Here's what my file picker looks like
filepicker.pick({
mimetypes: ['image/*'],
services: ['COMPUTER', 'URL'],
maxSize: 5 * 1000 * 1024
}, function(FPFile) {
// do stuff to file
});
The problem is that when a url select is chosen, instead of uploading the file to file picker the url is served directly. This makes storage unreliable because the external host can take the file down, etc.
Is there a simple way to ensure that when using the URL upload the file is directly hosted?
There are two options for this.
The first of which is the Store API. When you receive the fpurl back, call the store api and this will save the file to your storage directly. This might be best if, for example, people can select files at will, but you only want to persist them when they choose to save something. e.g. when uploading for a new post. Why persist items if they don't decide to create a new post in the end?
https://developers.filepicker.io/docs/web/#store
The second option is to replace your pick call, with the pickAndStore call, which does both at once and saves you having to do the store command in your pick callback.
https://developers.filepicker.io/docs/web/#pickAndStore
While pickAndStore may feel like it saves work, if there's a chance that you don't actualy need to persist data after a user picks it, then I'd go the extra distance with your own custom callbacks (this is where you'd create difference conversions too).
The best way is to use the filepicker.store api - see https://developers.filepicker.io/docs/web/#store for more information
Related
I've been searching in StackOverflow, but it seems that this question has not been asked yet. It's an architecture question about files being uploaded to GCP Storage.
TL;DR : Is there any issue using bucket.getFiles() directly (from a server), rather than storing each filename in my db, and then asking for them one by one and returning the array to the client ?
The situation:
I’m working on a feature that will allow the user to upload image attachements linked to a delivery note. This delivery note can have multiple attachements.
I use a simple upload button on my client (mobile device), and upload the content in GCP in a path/to/id-deliveryNote folder such as: path/to/id-deliveryNote/filename.jpg path/to/id-deliveryNote/filename2.jpg etc…
Somewhere else in the app the user should be able to click and download on each of those attachements.
The solution
After the upload being done in GCP, I asked myself how to read those files and give the user a download link to the file. That’s when I found the: bucket.getFiles() function.
Since my path to files are all below the same id-deliveryNote/ prefix, I leverage the usage of bucket.getFiles(prefix) and after the promise resolve can safely return to my user the list of links available.
The issue
I do not store the filenames in my deliveryNote table in my DB. Which can sound a bit problematic, relying on GCP to know the attachements of one deliveryNote. The way I see it is that, in my way I do not need to replicate the information in our DB (and possibly handling failure at two spots), and if I need those files I will at the ask GCP to give me their links. The opposed way of thinking is that, storing the names you will be able to list the attachements for the clients, and then generating the download link, when the user click a specific attachement.
My question is: Is there any issue using bucket.getFiles() directly (from a server), rather than storing each filename in my db, and then asking for them one by one and returning the array to the client ?
Some point that could influence the chosen method:
GCP costs per call difference ?
Invalid application data structure ?
Other things ?
There is no issue with using this method to return the link for the files to download. In the API documentation for this method - accessible here - they even show an example of returning files using prefixes as well. You just need to look out that Cloud Storage actually doesn't use real folders and only names that look like they are in folders - more details in this case here - so you don't mix up concepts when working with names and prefixes.
For the pricing point, you can get the whole pricing for Google Cloud Storage in this documentation, including how much each operation will cost - for example, it will cost you $ 0.02 per 50000 operations for object gets, retrieving bucket and object metadata - storing data, etc. After you check that, you can compare with your database costs as well, to check it if this point will impact you.
To summarize, there is no problem for you to follow this. The advantage of storing the names on Database, it's actually that even though you could have failure in two spots, it's more probable for you to face issues in only one place and this way the replication would be a great thing to have. So, you just need to decide which one fits you best.
My requirement - Single page application having a File upload and many other sliders/dropdowns that fire events to server whenever there is a change.
Ideal workflow for my application is
1) User uploads file using File Upload widget-> server should retain this file and use this for further requests from the same client
2) User changes sliders or dropdown values -> Client fires requests to the server. Server makes use of the file that was uploaded initially and applies some logic and returns a response that is rendered somewhere on the page.
User can simultaneously open the app in multiple tabs on web browser and each tab should be isolated, i.e., user can upload different files in different tabs and server should retain files and respond respectively.
Please let me know what technologies/frameworks are best fit for my use case.
Any links/example pages having the similar use case would also be very helpful.
I'd suggest this short of scheme:
When you upload the file, the server stores the file somewhere, coins a unique ID for that file and then stores that unique ID such that it can associate that unique ID with the recently uploaded file - probably using a simple database to associate the two in a persistent manner.
The unique ID is then returned to the client after the upload. The client window then keeps track of that unique ID in a javascript variable for that window.
Whenever that window wishes to apply some operation to that file, it sends an Ajax request to the server with the file's unique ID and parameters for the operation. For example, an ajax request to crop a file like this: http://x.com/api/crop?fileID=19482302&x=0&y=0&h=1024&w=2048.
When the server receives the Ajax request, it uses the unique ID to find which file it is associated with, applies the operation and returns the results.
The client receives the results and renders whatever it needs to render to show the results.
To keep unused files from building up on the server, the server probably needs to "age" away files that are no longer in use. For this, you can create some sort of expiration. When a file is accessed, it's recent access time is recorded. Then, you can have some periodic timer (say every hour) that cycles through the files and remove any who haven't been accessed recently (say in the last day or 6 hours or whatever time is appropriate for your app).
I am building a way for users to upload tracks with information about that track but I would like to do this asynchronously much like YouTube does.
At the moment there is an API endpoint of tracks that accepts a POST request with the uploaded file and all the meta data. It processes the track, validates everything and will then save the path to the track and all of its meta data in the database. This works perfectly but I am having trouble thinking of ways to do this asynchronously.
The user flow will be:
1) User selects a track and it starts uploading
2) A form to fill in meta data shows and user fills it in
3) Track is uploaded with its metadata to the endpoint
The problem is that the metadata form and the file upload are now two separate entities and now the file can finish uploading before the metadata is saved and vice-versa. Ideally to overcome this both the track and metadata would be saved in the browser as a cookie or something until they are both completed. At that point both would be sent to the endpoint and no changes would be required at the back end. As far as I am aware there is no way of saving files client side like this. Oh apart from that filesystem API which is pretty much deprecated.
If anyone has any good suggestions about how to do this it would be much appreciated. In a perfect world I would like there to be no changes to the back end at all but little changes are probably going to be required. Preferably no database alterations though.
Oh by the way I'm using laravel and ember.js just in case anyone knows of any packages already doing this.
I have thought about this a lot few months ago.
The closest solution that I managed to put together is to upload file and store it's filename, size, upload time (this is crucial) and other attributes in DB (as usual). Additionally, I've added the column temporary (more like a flag) which would initially be set to TRUE and only after you would sent meta data it would be negated.
Separately, I've set the cron job (I used Symfony2, but in Laravel is all the same) that would run on every 15-30 minutes and delete those files (and corresponding database records) which had temporary = TRUE and exceeded time window. In my case it was 15 minutes but you could set it to be coarse (every hour or so).
Hope this helps a bit :)
I'm using a JSON file to autopopulate a drop down list. It's by no means massive (3000 lines and growing) but the time taken to refresh the page is becoming very noticeable.
The first time the page is loaded the JSON is read, depending on what option the user has selected dictates which part of the JSON is used to populate the drop down.
It's then loaded on every refresh or menu selection after. Is it possible to somehow cache the values to prevent the need for it to be reloaded time and time again?
Thanks.
EDIT: More Info:
It's essentially a unit converter. The JSON holds all the details. When a users selects 'Temp' for example a call is made and the lists are populated. Once a conversion is complete you can spend all day running temp conversions and they'll be fine but everytime a user changes conversion type so now length, the page refreshes and takes a noticeable amount of time.
Unfortunately, I don't know of a standardized global caching mechanism in PHP. This article says that Optimizer Plus, a third party accelerator, is being included in core PHP starting in version 5.5. Not sure what version you are using but you could try that.
On a different note, have you considered file storage as andrew pointed out? I think it combined with $_SESSION could really help you in this case. Let me give you an example that would work with your existing JSON data:
Server Side
Store your JSON data in a .json file on your PHP server:
{
"data": "some data",
"data2": "more data",
"data3": [
...
],
etc.
}
Note: Make sure to properly format your JSON data. Remember all strings must be enclosed in double quotes ".
In PHP, use an if statement to decide the appropriate action:
error_reporting(E_ALL);
ini_set("display_errors", "On");
session_start();
if(isset($_SESSION['dataCache'])) {
echo json_encode($_SESSION['dataCache']);
} else {
$file = 'data.json';
if (!is_file($file) || !is_readable($file)) {
die("File not accessible.");
}
$contents = file_get_contents($file);
$_SESSION['dataCache'] = json_decode($contents, true);
echo $contents;
}
So lets dig into the above coding a little more. So here's what we are doing in a nutshell:
Turn on error reporting and start session support.
Check to see if we've already read the file for this user.
If so, pull the value from storage and echo it out and exit. If not continue below.
Save off the file name and do a little error checking to ensure PHP can find, open and read the contents of the file.
Read the file contents.
Save the decoded json, which is not an array because of the `true` parameter passed to `json_decode`, into your `$_SESSION` variable.
Echo the contents to the screen.
This will save you the time and hazzle of parsing out JSON data and/or building it manually on the server. It will be cached for the users session so that they can use it through out.
Client Side
I assume you are using ajax to fetch the information? If not correct me, but I was assuming that's where some of your JavaScript comes into play. If so you may consider this:
Store the returned data in sessionStorage on the user's browser when it's returned from the server:
$.ajax({
...
success: function (res) {
localStorage.setItem("dataCache", JSON.stringify(res));
},
...
});
Or if you use promise objects:
$.ajax({
...
}).done(function (res) {
localStorage.setItem("dataCache", JSON.stringify(res));
});
When you need to read it you can do a simple test:
var data;
// This returns null if the item is not in local storage.
// Since JavaScript is truthy falsy, it will be evaluated as false.
if(localStorage.getItem("dataCache")) {
data = JSON.parse(localStorage.getItem("dataCache"));
} else {
// Make ajax call, fetch object and store in localStorage in the success or done callbacks as described above
}
Notes:
localStorage is a new feature in HTML5, so it's not fully supported on all browsers yet. Most of the major ones do however, even as far back as IE8 (I think). However, there is no standardized size limit on how much these browsers are required to hold per site.
It's important to take that into consideration. I can guarantee you probably will not be able to store the entire 30,000 line string in localStorage. However, you could use this as a start. Combined with the server side solution, you should see a performance increase.
Hope this helps.
I use the browser's cache to ensure that my large chunk of JSON is only downloaded once per session. I program in ASP.NET, but I'm sure PHP has the same mechanisms:
On session start, I generate a random string as session key for my dynamic JavaScripts. This key get stored in the ASP.NET session state under the key JsonSessionID. That way I can refer to it in my page markup.
I have a "generic http handler" (an ashx file) that when called by the browser, returns a .js file containing my JSON.
In my HTML I include the dynamic script:
<script type="text/javascript" src="/dynamicJSON.ashx?v=<%= JsonSessionID %>"></script>
The browser will automatically cache any URLs included as scripts. The next time the browser is asked to load a cached script from a URL, it will just load up the file from the local disk. This includes dynamic pages like this.
By adding the ?v= in there, I ensure that the JSON is updated once per session.
Edit
I just realized that your JSON is probably static. If that's the case, you can just put your JSON into a static .js file that you include in your HTML, and the browser will cache it.
// conversionData.js
var conversionData = { "a":1,"b":2,"c":3 };
When you include the conversionData.js, the conversionData variable will be in scope with the rest of your page's JavaScript that dynamically updates the drop-downs.
Edit 2
If you are serving static files, this blog post has a good pattern for cache-busting based on the file's date modified property. i.e. the file is only downloaded when it is changed on the server.
I have yet to find a good method for cache-busting JSON created via database lookup tables, other than per-session. Which isn't ideal because the database could change mid-session.
Once you've got your JSON data decoded into an object you can just keep the object around, it should persist until a page reload at least.
If you want to persist between reloads you might want to look at HTML5's localStorage etc.
You would need to come up with an age strategy, maybe just dump the current date in there with it as well so you can compare that and expire as needed.
I would suggest storing your json data to a session. On first page load you can write a script to get your json data, then store them into a session.
on each page load/refresh afterwards you can check our session to decide what to do - use the session data or fetch again your json data.
This approach suites me for small scale data (for example: an array of products - colors - sizes - prices).
Based on your data you should test you loading times.
Here is a simple hack:
Create a call to a php file as GET request with parameter "bla-bla.html"
or "bla-bla.css"... well you know, it makes browser think it is not a php, but rather "html" or "css". And browser will cache it.
To verify that the trick is working - go to the "network" tab of the browser dev panel and you will see column "type" there along with "transferred" - instead of having php there and actual size, you will find "html" and "(cached)"
This is also good to know when you passing parameters like "blah-blak.html" to the php file and expect it will not be cached. Well, it will be cached.
Tested on FireFox Quantum 57.0.1 (Mac 64bit)
P.S.
Chrome 63 on Mac is capable of recognising real file type in this situation. So it cannot be fooled.
Thinking out of the box here:
but if your list has 3000 lines and growing (as you said)
is it possible for you to establish its maximum size ?
let's say the answer is 10,000 (max) items; then do you really need an ajax call ?
you could transfer the data straight away with the page
(depending on your architecture of course, you could come out with different solution)
Here is what I am trying to do: I am making a custom text file containing a test. This test is unique to the user and well I don't want my server to stack all those text files.
Is there a way to use Ajax/JavaScript/jQuery to find if the user has already finished the download and if they have get a return value (1 if finished) so the response can be sent back to the PHP file and it can delete that file off the server (real-time like)?
I know there are plenty of ways to do this using PHP. Sort of like run clean up upon user log out and so on but I wanted to try using the method above since it can have many other applications that might be cool to use. I tried most search engines but they have nothing close to what I need.
Why do you need to store them in a file? Just use a PHP script or such that creates the test and outputs it directly to the user. That's how it will get deleted when the download is complete.
If it's important you may want the user to return to your server with the hash of the downloaded file. If the hash matches you know two things:
1. The user downloaded the file successfully
2. It's now ok to delete the file
Well it is very simple. I don't know how to make a PHP webpage send itself to the user other than to make the PHP make a text file and force send that to the user. This creates the problem of having so many text files in a temporary folder.
Now if the test required say 15 chapters each having a text or HTML format file then the script neatly zips all those files and sends them to the user. Again falling on the same problem if the user is finished downloading I am trying to get any type of script to delete the temporary zip or text file out of the temporary directory in somewhat real time.
If I could MD5 a downloaded file using JavaScript I welcome it and it would be a hack solution to the problem but how will the JavaScript gain access to the root access of the download folder of the user? There are security issues there if I am not mistaken. Hope this helps round the question a bit more.
I have a good solution for you here using the jQuery File Download plugin I created. It allows for you to get the behavior of performing an Ajax file download (not actually possible possible) complete with Success and Failure callbacks. In a nutshell you can just use the Success callback (that indicates the file download was successful) to perform an Ajax post back to the server to delete the file. Take a look at the blog post for an example on how to use the Success callback option or a demo which uses those callbacks in the context of showing modals to inform the user of what is going on.