Develop Reference

How can I store a JSON file in Javascript to use continuously?

I'm making a website and on one of my pages I have this button that when clicked, should take a haiku from a JSON file that contains over 5000 and display it to the user. Currently I do this the following way.
<script>
const haikuUrl = 'http://mywebsite/haikus.json';
async function getHaiku() {
const num = Math.floor(Math.random() * 5145);
const response = await fetch(haikuUrl);
const data = await response.json();
const {line1, line2, line3} = data[num];
document.getElementById('line1').textContent = line1;
document.getElementById('line2').textContent = line2;
document.getElementById('line3').textContent = line3;
}
document.getElementById('haikuButton').addEventListener('click', () => {
getHaiku();
});
</script>
I'm pretty much new to JS, and after looking around and watching some videos this was the only way I could get this to work but I'm pretty much sure there has to be a better way to do this. Right now, I'm having to load the whole file every time just to get 1 random object from the JSON. What I'd like to have is a constant or a variable that just sits there waiting for the user to ask for a random one. Something like this:
<script>
const data= [{},{},{}...{}]; //an array of all the haikus loaded once at the beginning
function getHaiku() {
const num = Math.floor(Math.random() * 5145);
const {line1, line2, line3} = data[num];
document.getElementById('line1').textContent = line1;
document.getElementById('line2').textContent = line2;
document.getElementById('line3').textContent = line3;
}
document.getElementById('haikuButton').addEventListener('click', () => {
getHaiku();
});
</script>
So pretty much the same just without the having to fetch the whole data every time.
I guess one option could be to hardcode the whole dataset into the js file into a variable, but something tells me there's got to be a better way.

I would first fetch the data, then choose a random haiku by using the randomly generated number as an index of the data array. Something like below:
I have not tested the code below so I am not sure if it works, but hopefully this nudges you in the right direction.
let data;
let button = document.getElementById('haikuButton');
let randomHaiku = '';
// Fetch data
async function getHaikus(){
const response = await fetch('http://mywebsite/haikus.json')
data = await response.json();
}
// Generate random number
function generateRandomNumber(array){
return Math.floor(Math.random() * array.length)
}
// get random haiku
button.addEventListener('click', ()=>{
let index = generateRandomNumber(data)
randomHaiku = data[index]
console.log(randomHaiku);
}, false)
getHaikus();

If I understand correctly your question, one optimization you can do is as follows:
const haikuUrl = 'http://mywebsite/haikus.json';
let haikus = null;
async function getHaiku() {
const num = Math.floor(Math.random() * 5145);
if (!haikus) {
const response = await fetch(haikuUrl);
haikus = await response.json();
}
const {line1, line2, line3} = haikus[num];
[...]
}
So it would download the file the first time the user clicks on the button, but not download it again if the user clicks on the button again (unless they close the webpage in between).
You should also definitely use the length of the array instead of hardcoding 5145, as mentioned in other answers.
It is also certainly possible to embed the data directly in the JS file, but it won't make much difference, it will just make the JS file (which must be downloaded as well) much bigger. So it wouldn't solve the problem of needing to download all the haikus when you need just one.
Making it so that it really only downloads one haiku is a bit more complicated. You could have a backend that the frontend requests a single haiku to (but that probably increases complexity significantly if you currently don't have a backend). You could also store all haikus in separate files with predictable names (for instance haikus/42.txt) and fetch only a single such file.

Trying to retrieve localUri from assets in Expo MediaLibrary, why does this return null?

I have two questions here, an Expo MediaLibrary question, and a linked Javascript/scoping question.
I have a ReactNative Expo app that loads recent images from a users media library. For this I need to use the MediaLibrary.getAssetsAsync() method.
Annoyingly, .getAssetsAsync() does not return localUri needed to display the image. It returns uri property which looks like this: "uri": "ph://8A1262C3-23F7-4BD3-8943-C01128DCEEB1"
Unless I'm mistaken, I can't use an asset file uri like this to display images in react, we need localUri. So for each photo I am calling the MediaLibrary.getAssetInfoAsync() method - which returns localUri. Here's how I am accomplishing that:
const selectAlbum = async (albumName) => {
const foundAlbum = await MediaLibrary.getAssetsAsync({album: albumName, mediaType: 'photo', first: 20})
const assets = foundAlbum['assets'] //array of photos or 'assets'
const assetsWithInfo = []
//for each selected photo
for(let i=0; i<assets.length; i++){
const assetWithInfo = getAssetInfo(assets[i].id)
//console.log(assetWithInfo) //doesnt work, null etc - WHY?!
assetsWithInfo.push(assetWithInfo)
}
}
const getAssetInfo = async (id) =>{
let res = await MediaLibrary.getAssetInfoAsync(id)
let asset={
creationTime: res['creationTime'],
isFavorite: res['isFavorite'],
localUri: res['localUri'],
}
//console.log(asset) //works, object correctly displays
return asset
}
My questions are:
Is there a more efficient way to do this i.e. display images from MediaLibrary without having to call so many functions. This feels messy and more complicated than it should be.
In the getAssetInfo async, the asset object is correctly displayed (using the console.log) with all properties. But when I return this object to the selectAlbum function and console.log the result, I get null etc. Why?

For #2, the issue is you're not awaiting the result of the call.
const assetWithInfo = await getAssetInfo(assets[i].id); should fix it.

CSS styles won't update before fetching file

Intended Goal - User selects different colors from various color inputs and creates their own theme. Once the colors are chosen, the user clicks the download button and gets the generated CSS file with the colors he/she chose.
Issue - I'm able to download the CSS file, but I'm getting the original values despite changing the inputs to different colors.
What I've Done
The CSS file that's being downloaded already exists and all of the colors that correspond to different elements are done via CSS variables.
I'm updating the changes live by doing the following.
import { colors } from './colorHelper'
const inputs = [].slice.call(document.querySelectorAll('input[type="color"]'));
const handleThemeUpdate = (colors) => {
const root = document.querySelector(':root');
const keys = Object.keys(colors);
keys.forEach(key => {
root.style.setProperty(key, colors[key]);
});
}
inputs.forEach((input) => {
input.addEventListener('change', (e) => {
e.preventDefault();
const cssPropName = `--${e.target.id}`;
document.styleSheets[2].cssRules[3].style.setProperty(cssPropName, e.target.value);
handleThemeUpdate({
[cssPropName]: e.target.value
});
console.log(`${cssPropName} is now ${e.target.value}`)
});
});
Then, I fetched the stylesheet from the server, grabbed all the CSS Variables and replaced them with their actual value (hex color value).
After that, I got the return value (new stylesheet without variables) and set it for the data URI.
async function updatedStylesheet() {
const res = await fetch("./prism.css");
const orig_css = await res.text();
let updated_css = orig_css;
const regexp = /(?:var\(--)[a-zA-z\-]*(?:\))/g;
let cssVars = orig_css.matchAll(regexp);
cssVars = Array.from(cssVars).flat();
for (const v of cssVars) {
updated_css = updated_css.replace(v, colors[v.slice(6, -1)]);
};
return updated_css;
}
const newStylesheet = updatedStylesheet().then(css => {
downloadBtn.setAttribute('href', 'data:text/plain;charset=utf-8,' + encodeURIComponent(css));
downloadBtn.setAttribute('download', 'prism-theme.css');
})
I already have a download button setup in my HTML and I grabbed it earlier in my script so that it was available anywhere for me to use. downloadBtn
I set up the button to fire and grabbed the new sheet.
downloadBtn.addEventListener('click', () => {
newStylesheet();
});
The Result
I get the initial color values of the stylesheet despite changing the colors within the color inputs on the page. So the CSS file isn't being updated with the new colors before I download the file.

You could use PHP to pass the values to a new page. Let's say you chose the colors you want then click a button that says "Generate" that takes you to the "Generate Page".
The values would be passed directly into the HTML and you would download from the Generate Page instead.
This is if you know PHP of course, just a suggestion on how you might solve it.
(would comment, but can't due to reputation)

How to automate downloading a file from a site?

I want to download several data files from this URL: https://pselookup.vrymel.com/
The site contains a date field and a download button. I want to download data for multiple years (which would mean a lot of requests) and I want to make it automatically.
I've created a Javascript snippet, however, it keeps downloading just the same file over and over again.
$dateField = document.getElementsByClassName('csv_download_input__Input-encwx-1 dDiqPH')[2]
$dlButton = document.getElementsByClassName('csv_download_input__Button-encwx-0 KLfyv')[2]
var now = new Date();
var daysOfYear = [];
for (var d = new Date(2016, 0, 1); d <= now; d.setDate(d.getDate() + 1)) {
daysOfYear.push(new Date(d).toISOString().substring(0,10));
}
(function theLoop (i) {
setTimeout(function () {
$dlButton.click()
$dateField.value = daysOfYear[i]
if (--i) { // If i > 0, keep going
theLoop(i); // Call the loop again, and pass it the current value of i
}
}, 3000);
})(daysOfYear.length-1);
How could I download all of the files automatically?

First off, javascript in the client is probably not the best language to do this nor the best approach to make this happen. It might work, but it's better to know what is best when choosing an approach to a problem. Also, it will avoid for you clicking ~800 times in the popup accepting the download.
You can get the files in a programatically way by just learning what you browser is doing to get the file and trying to reproduce it in bunch.
After inspecting the calls you can see that it's calling an endpoint and that endpoint is returning a link which contains the file that you can download.
Well, that is going to be easy, so now you just need to make the script in any language to be able to retrieve them.
I've chosen javascript but not client side, but nodejs which means that this has to run from your computer.
You could do the same with bash, python or any other language.
To run this do the following:
Go to a new empty directory
Run npm install axios
Create a file with the code I pasted let's call it crawler.js
Run node crawler.js
This has been tested using node v8.15.0
// NOTE: Require this to make a request and save the link as file 20190813:Alevale
const axios = require('axios');
const fs = require('fs');
let now = new Date();
let daysOfYear = [];
const baseUrl = 'https://a4dzytphl9.execute-api.ap-southeast-1.amazonaws.com/prod/eod/'
for (var d = new Date(2016, 0, 1); d <= now; d.setDate(d.getDate() + 1)) {
daysOfYear.push(new Date(d).toISOString().substring(0,10));
}
const waitFor = (time) => {
return new Promise((resolve => setTimeout(resolve, time)))
}
const getUrls = async () =>{
let day
for (day of daysOfYear) {
console.log('getting day', baseUrl + day)
// NOTE: Throttle the calls to not overload the server 20190813:Alevale
await waitFor(4000)
await axios.get(baseUrl + day)
.then(response => {
console.log(response.data);
console.log(response);
if (response.data && response.data.download_url) {
return response.data.download_url
}
return Promise.reject('Could not retrieve response.data.download_url')
})
.then((url) =>{
axios({
method: 'get',
url,
responseType: 'stream'
})
.then(function (response) {
// NOTE: Save the file as 2019-08-13 20190813:Alevale
response.data.pipe(fs.createWriteStream(`${day}.csv`))
})
.catch(console.error)
})
.catch(error => {
console.log(error);
});
}
}
getUrls()

You can instead of simulating the user, get the link to download from:
https://a4dzytphl9.execute-api.ap-southeast-1.amazonaws.com/prod/eod/2019-08-07
just change the date at the end to the date of the file you want to download. And use axios to get this URL.
This will save you sometime (in case you don't really need to simulate the click of the user etc)
Then you will get a response like this:
{
download_url":"https://d3u9ukmkxau9he.cloudfront.net/eod/2019-08-07.csv?Expires=1566226156&Signature=QRUk3tstuNX5KYVPKJSWrXsSXatkWS-eFBIGUufaTEMJ~rgpVi0iPCe1AXl5pbQVdBQxOctpixCbyNz6b9ycDgYNxEdZqPr2o2pDe8cRL655d3zXdICnEGt~dU6p35iMAJkMpPSH~jbewhRSCPUwWXQBfOiEzlHwxru9lPnDfsdSnk3iI3GyR8Oc0ZP50EdUMHF7MjWSBRbCIwnu6wW4Jh0bPmZkQDQ63ms5QxehsmtuGLOgcrC6Ky1OffVQj~ihhmBt4LGhZTajjK4WO18hCP3urKt03qpC4bOvYvJ3pxvRkae0PH1f-vbTWMDkaWHHVCrzqZhkAh3FlvMTWj8D4g__&Key-Pair-Id=APKAIAXOVAEOGN2AYWNQ"
}
and then you can use axios to GET this url and download your file.

Javascript: Compare two selected files by file path

I'm looking for a way to check if two files/documents (PDF, JPG, PNG) are the same.
If a user selects one or more files, I convert the File Object to a javascript object. I'm keeping the size, type, filename and I create a blob so I can store the object in my redux store.
When a user selects another file I want to compare this file with the files that already has been added (so I can set the same blobURL).
I can check if two files has the same name, type and size but there is a change that all these properties match and the files aren't the same so I would like to check the file path. Unfortunately, that property isn't provided in the File Object. Is there a way to get this or another solution to make sure both files are (not) the same?

No there is no way to get the real path, but that doesn't matter.
All you have access to is a FakePath, in the form C:\fakepath\yourfilename.ext (from input.value), and sometimes a bit more if you gained access to a directory.
But anyway you probably don't want to check that two files came from the same place on the hard-disk, this has no importance whatsoever, since they could very well have been modified since first access.
What you can and probably want to do however, is to check if their content
are the same.
For this, you can compare their byte content:
inp1.onchange = inp2.onchange = e => {
const file1 = inp1.files[0];
const file2 = inp2.files[0];
if(!file1 || !file2) return;
compare(file1, file2)
.then(res => console.log('are same ? ', res));
};
function compare(file1, file2) {
// they don't have the same size, they are different
if(file1.size !== file2.size)
return Promise.resolve(false);
// load both as ArrayBuffers
return Promise.all([
readAsArrayBuffer(file1),
readAsArrayBuffer(file2)
]).then(([buf1, buf2]) => {
// create views over our ArrayBuffers
const arr1 = new Uint8Array(buf1);
const arr2 = new Uint8Array(buf2);
return !arr1.some((val, i) =>
arr2[i] !== val // search for diffs
);
});
}
function readAsArrayBuffer(file) {
// we could also have used a FileReader,
// but Response is conveniently already Promise based
return new Response(file).arrayBuffer();
}
<input type="file" id="inp1">
<input type="file" id="inp2">
Now, you say that you don't have access to the original Files anymore, and that you can only store serializable data. In this case, one less performant solution is to generate a hash from your Files.
This can be done on front-end, thanks to the SubtleCrypto API,
but this operation being quite slow for big files, you may want to do it systematically from server instead of doing it on front, and to only do it on front when the sizes are the same:
// a fake storage object like OP has
const store = [
{ /* an utf-8 text file whose content is `hello world`*/
name: "helloworld.txt",
size: 11,
hash: "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9" // generated from server
}
];
// the smae file as the one we fakely stored
const sameFile = new File(['hello world'], 'same-file.txt');
// a file the same size as the one we stored (needs deep check)
const sameSizeButDifferentContent = new File(['random text'], 'differentcontent.txt');
inp.onchange = e => tryToStore(inp.files[0]);
tryToStore(sameFile); // false
tryToStore(sameSizeButDifferentContent);
// hash: "a4e082f56a58e0855a6abbf2f4ebd08895ff85ea80e634e02b210def84b557dd"
function tryToStore(file) {
checkShouldStore(file)
.then(result => {
console.log('should store', file.name, result)
if(result)
store.push(result);
// this is just for demo, in your case you would do it on the server
if(!result.hash)
generateHash(file).then(h => result.hash = h);
});
}
async function checkShouldStore(file) {
const {name, size} = file;
const toStore = {name, size, file}; // create a wrapper object
// first check against the sizes (fast checking)
const sameSizes = store.filter(obj => obj.size === file.size);
// only if some files have the same size
if(sameSizes.length) {
// then we generate a hash directly
const hash = await generateHash(file);
if(sameSizes.some(obj => obj.hash === hash)) {
return false; // is already in our store
}
toStore.hash = hash; // save the hash so we don't have to generate it on server
}
return toStore;
}
async function generateHash(file) {
// read as ArrayBuffer
const buf = await new Response(file).arrayBuffer();
// generate SHA-256 hash using crypto API
const hash_buf = await crypto.subtle.digest("SHA-256", buf);
// convert to Hex
const hash_arr = [...new Uint8Array(hash_buf)]
.map(v => v.toString(16).padStart(2, "0"));
return hash_arr.join('');
}
<input type="file" id="inp">