I am trying to retrive JSON files from google cloud storage. The JSON files are stored in the same folder as their corresponding audio files, as follows:
bucket
folder1
audio.wav
audio.json
folder2
audio.wav
audio.json
I'm using nodejs, and calling cloud storage as follows:
bucket.getFiles({
delimeter: 'audio' //should be preventing loading of audio files but it isnt
}).then((data) => {
console.log(data)
}).catch((error) => {
console.log(error)
})
I have a feeling it might be related to how Google handles folders, but I'm not really sure. Here is how the files are being stored in the first place:
const storage = require('#google-cloud/storage')();
const filename = `recordings/${path.parse(req.body.RecordingUrl).name}/audio.wav`;
const file = storage
.bucket(config.RESULTS_BUCKET)
.file(filename)
Which results in some sort of string that assigns the audio file a unique ID as the folder name, and then appends the /audio.wav, and somehow cloud storage recognizes that as a folder. Just a guess, but not sure how to solve that if it's the case. Anyone know what I might be doing wrong?
If you're trying to list all objects that end in .json, then you don't want to set a delimiter at all. The delimiter is used for hierarchical listing. You almost never want to set it to any values other than unset and /. See list documentation.
Related
I am trying to create a firebase function that downloads a photo off the web via URL and uploads it to firebase storage.
Using Axios to stream the file to the bucket. it appears the file gets uploaded but i cant download or view it.
This is my code:
let fileURL = 'https://www.example.file.path.png'
let myFile = await axios.get(fileURL, { responseType: 'stream' })
let destFile = bucket.file(photoId).createWriteStream({ contentType: myFile.headers['content-type']})
myFile.data.pipe(destFile)
And here is the storage console from firebase:
I have messed around with the storage api and attempted using the upload and save functions. using axios get and write streams is the closest that I'v got to getting this to work.
Reading the example in the docs only aids in my confusion because the file is never reference in the upload function.. just the file name??
Feel like i'm almost there considering the file or rather the name of the file is there and the size and the type.. just not the content?
This problem is related with Firebase console only. There is no problem with downloading with command: gsutil cp gs://<your bucket>/yourfile . (doc) and as well it is working in Google Cloud Console in Storage browser (direct link).
However indeed file uploaded like this is not possible to be downloaded or previewed with Firebase console.
After many tries I have realized that this is related with custom metadata Access token, which underneath is calledfirebaseStorageDownloadTokens. It appears automatically if ex. you download the file directly in Firebase console.
Anyway I noticed that value of the metadata is irrelevant. According to my test if you change your createWriteStream method parameters to:
let destFile = bucket.file(photoId)
.createWriteStream({
metadata: {
contentType: myFile.headers['content-type'],
metadata: {
firebaseStorageDownloadTokens: 'canBeAnyThing'
}
}
});
The problem disappears...
A file already downloaded to Firebase Storage and affected by the issue can be fixed by adding the same metadata. In the screenshot you have provided you can see "File Location" if you open you will see link "Create new access token" or as well you can add it in GCP Storage Browser adding it manually in "Edit metadata" for such object (remember to refresh FB console in browser).
At this point we can think of why it's looks like this. I have found interesting information in github here.
Is it possible to write a firebase cloud function that triggers when a new fila was created in firebase storage - onFinalize (but we don't know the exact bucket in advance) ?
Inside my firebase storage a have a folder 'loads' and inside I have folders named with load id like:
/loads/-Lw1UySdunYyFMrrV6tl
/loads/-LwisA5hkNl_uxw3k36f
/loads/-LwkPUm-q7wNv-wZ49Un
https://ibb.co/NnQkTyC here's a screenshot of storage
I want to trigger cloud function when new file has been created inside one of these folders. I don't know in andvance where the file will be created. I don't know if this is even possible. That's why I need an advice.
My main goal is to merge 2 pdf files in one within cloud functions. In my app (TMS written with vuejs) on frontend I create confirmationOrder.pdf using html2canvas/jsPDF and then save it to storage/loads/${loadID}. And later on user can manually upload POD.pdf on the same bucket. When it happens I want my cloud function to merge these two pdfs in one new file (in same storage bucket). But again I don't know the bucket in advance.
Here's how I upload PDFs in frontend:
async uploadConfPDF({ commit }, payload) {
const PDF = payload.PDF;
const loadID = payload.loadID;
const fileName = payload.fileName;
const fileData = await firebase
.storage()
.ref(`loads/${loadID}/${fileName}.pdf`)
.put(PDF);
const confOrderURL = await fileData.ref.getDownloadURL();
return confOrderURL;
},
Any help is highly appreciated. Sorry if my explanation could seem not clear enough. English is not my native language.
EDIT FOLLOWING YOUR QUESTION RE-WORKING
Based on your code and on the print screen of your Cloud Storage console, you are working in the default bucket of your project, which location's URL is gs://alex-logistics.appspot.com.
As we can see on the print screen of your Cloud Storage console, the files in your bucket are presented in a hierarchical structure, just like the file system on your local hard disk. However, this is just a way of presenting the files: there aren't genuine folders/directories in a bucket, the Cloud Storage console just uses the different part of the files paths to "simulate" a folder structure.
So, based on the above paragraphs, I think that your question can be re-phrased to "In a Cloud Function, how can I extract the different parts of the path of a file that is uploaded to the default bucket of my Firebase Project?".
Answer:
In a Cloud Function that is triggered when a file is added to the default bucket, you can get the file path as follows:
exports.yourCloudFunction = functions.storage.object().onFinalize(async (object) => {
const filePath = object.name; // File path in the bucket.
//...
});
Since we use an onFinalize event handler, you get the path of this new file by using the name property of the object Object, which is of type ObjectMetadata.
You can then use some String methods to, for example, extract from this path the ${loadID} you refer to in your question.
I am using react-native-fetch-blob to download excel file and these files are stored inside app/documents folder which I am not able to open in iphone and I don't know the absolute path of the iphone's document folder to save the file there.
here is my code:
downloadFile(){
const dirs = RNFetchBlob.fs.dirs;
let context = this,
userBarRelationId = context.state.userBarId;
RNFetchBlob
.config({
fileCache : true,
path : dirs.DocumentDir + '/excel.xlsx'
})
.fetch('GET', 'http://localhost:5000/getExcel/'+userBarRelationId, {
})
.then((res) => {
console.log('The file saved to ', res.path(), res.data)
})
}
Wait, you're asking to save the data in a Documents folder other than the one your sandboxed application has access to?
This is not possible by iOS design. See also here. Not even a minute of google searching reveals you that, not to mention that it is basically a core principle.
If I didn't understand you I apologize, but then I have no clue what you want to know. If you're asking for how to save the file to the user's iCloud Drive (which kind of has a Documents folder, too), you'll have to invest some more work. I've never done it myself, but a good starting point is this.
I have an ng-repeat that, among other thing, outputs on image:
<div class="installation" get-products install-index="{{$index}}" ng-repeat="installation in installations track by $index">
...
<img ng-src="{{installation.logo}}" />
...
</div>
When my app starts it downloads needed images and stores their location in a local database. When the page is viewed the installations are populated:
<div class="installation ng-scope" ng-repeat="installation in installations track by $index" install-index="43" get-products="">
...
<img src="C:/Users/.../AppData/Local/Packages/.../LocalState/installations/.../...png" ng-src="C:/Users/.../AppData/Local/Packages/.../LocalState/installations/.../...png">
...
</div>
(dots used to hide person and client data)
If I paste the src location into my browser I see the image so I know it's saved at that location. However, in my app it's not showing. This is a constant issue through the app with the downloaded files. I know the image are in the correct area and the src location is correct but none of them show.
--- EDIT ---
I do have white listing applied as I was getting an unsafe for file:///. Also, when I was using a relative path it was working fine. I had a preloaded database that pointed to file inside the app files.
I don't think it's an access issue since I have a .db file at the same location that all my data is being pulled from.
--- EDIT ---
I set it as file:///C:/... and I'm having the same issue.
I also tried file:///C:/... , http://localhost/..., http://localhost:/..., http://localhost:C/..., C:/..., and file:///.... None of witch give me anything. The first two localhost items do give me a broken image icon, that's about it. I'm not running a local server, just thought I'd try it.
You can do this in two different ways:
1) Use the file protocol
2) use a local host server to store the picture and access it from the local host
for security reasons you cannot use your file system path for images. you shouldn't even use it at all, because when your app gets hosted, you wouldn't be accessing the image via such paths.
method 1:
just add file:/// in place of the c:/. file is the protocol for your file system, just as http or HTTPS is a web protocol.
NB: I haven't tested or used this before so I'm not really certain. I'm posting this from a small mobile device. but I believe it should work.
method 2:
start your wampserver or python server or any local server you have. put the image in a folder where your server can access (if wampserver, this would be a folder or directory in your WWW). say the name of the folder is "my_images" and your wampserver is running on localhost.. you can access the image like so:
http://localhost/my_images/image_name
use this path for your ng-src.
Because I Cordova File and Windows weren't playing nice using the call for cordova.file.dataDirectory didn't work. Instead I used the fs object returned by window.requestFileSystem(...,function(fs){...});
When generating my save to path as well as the path to create directories and location data I used fs.winpath which returned C:/.... The web (which Cordova basically is) won't allow you to have access to local files not associated with the site/apps structure, which is now obvious.
I dug in to the fs object and found fs.root.nativeURL points to ms-appdata:///local/. Switching everything over to this still downloaded all files and directories to the same location but stored that to the database as the file location. When the app loaded the ms-appdata path instead of the C:/ path the images displayed.
oh, a Cordova app.. why don't you place the file in an images folder In your project. since all files will be loaded using index.html (I assume). you can easily refer to the file relative to the location of index.html. how I would normally organize my project is that, my index.html and folders containing resources like js, CSS etc would be on thesame level, so I can easily get the image files using ng-src="img/image_name". so I could have a structure like this
index.HTML
img
..image_name.ext
..image2.ext
css
..style.css test it in a browser location if it works, it will work on the device. Cordova would know how to translate d into something it can recognise.
This is some sample code, i quickly put together. I tested it and it worked. Firstly i create a directory using file plugin and then download to this directory using file transfer. Replace the url parameter of file transfer with the url you wish to download from.
$ionicPlatform.ready(function() {
$cordovaFile.createDir(cordova.file.externalDataDirectory,
file_location,false).then(
function(success){
return success;
},function(error){
return error;
}).then(function(value){
var url = material.file_uri;
var targetPath = cordova.file.externalDataDirectory
+ "/" +file_location + "/" + file_name;
var trustHosts = true
var options = {};
$cordovaFileTransfer.download(url, targetPath, options, trustHosts)
.then(function(result) {
console.log(result)
}, function(err) {
console.log(err)
}, function (progress) {
$timeout(function () {
console.log(Math.floor((progress.loaded / progress.total) * 100));
})
});
})
})
I am having this weird problem. I have an app that can create playlists with this code
var playlist = new Windows.Media.Playlists.Playlist();
...
WinJS.Utilities.id("appbar-save-button").listen("click", function ()
{
var savePicker = new Windows.Storage.Pickers.FolderPicker();
savePicker.fileTypeFilter.append("*");
savePicker.pickSingleFolderAsync().then(function (folder)
{
playlist.saveAsAsync(folder, "My Playlist", Windows.Storage.NameCollisionOption.replaceExisting, Windows.Media.Playlists.PlaylistFormat.windowsMedia);
});
})
The problem comes when I try to acces this file with this code
WinJS.Utilities.id("appbar-open-button").listen("click", function ()
{
var openPicker = Windows.Storage.Pickers.FileOpenPicker();
openPicker.fileTypeFilter.append(".wpl");
openPicker.pickSingleFileAsync().then(function (file)
{
Windows.Media.Playlists.Playlist.loadAsync(file).then(function (playlist)
{
// Print the name of the playlist.
});
});
})
On the commented line I get an exception : Cannot access the specified file or folder (⑰ᑲÕ). The item is not in a location that the application has access to (including application data folders, folders that are accessible via capabilities, and persisted items in the StorageApplicationPermissions lists). Verify that the file is not marked with system or hidden file attributes.
I have given the application Document library capabilities with File Type associations with type .wpl but still I get this exception. How can I fix it
EDIT: Adding the videos to the future acces list seems to solve the problem for app created playlists, but for random playlists the problem persists.
As far as i can see, the issue is not with the permission to load the 'playlist' file; using the file picker if a file is selected at any location on the computer, user will get access to the file; same applies to the folder picker where access to all files in the folder will be there. After that, if the selected file/folder is added to the FutureAccessList, the folder/file will be accessible later also.
The playlist may be containing files that are in the folders not accessible to user. To confirm this, try to open a playlist with no files or files in the music library location only - after giving the application 'music library' capability. If this works - application need to have settings to add folders with the music. playlist that will contain files in the selected folders will only load.