Simple CSV parsing in Javascript - javascript

Here is my problem:
I am trying to parse a local CSV file in JavaScript. The file looks like this:
Year,Promo,Surname,Name,Mail
2005/2006,0,XXXX,XXXXX,xxxxx.xxxxx#gmail.com
(...)
2006/2007,1,XXXX,XXXXX,xxxxx.xxxxx#gmail.com
(...)
2007/2008,2,XXXX,XXXXX,xxxxx.xxxxx#gmail.com
etc.
I tried to parse it using several librairies (PapaParse.js, jquery-csv, d3.js...), but:
either the parsing fails (my array is empty)
or I get a XMLHttpRequest cannot load - Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource error, since my file is stored locally.
Is there a simple solution to parse a CSV file in JavaScript, in order to access the data? I looked up hundreds of posts on forums but I could not get it to work.
Thank you very much (excuse me, I am quite new in JS).
Tom.

this answer is canonical in that it addresses anyone's problem that might be described by the question. Only the first of these answers is meant for the OP, although, regarding the last comment, the edit section I added at the bottom is also specifically for the OP.
if you are doing this for a small, local app, you probably can do one of these two things:
launch the browser with CORS disabled:
Chrome.exe --disable-web-security
in the source there is also instructions for firefox
src
run a micro server for your files:
If you’ve got Python installed (most Mac and Linux users do), you can start a quick local web server for testing. Using the command prompt, navigate to the directory that has your HTML files and run the following command:
python -m SimpleHTTPServer
Your files should now be accessible from http://localhost:8000/ and may have a good chance of working when file:/// does not.
src
A better solution, if you run into CORS issues with the python server, might be local-web-server from node: https://www.npmjs.com/package/local-web-server
the typical user looking for an answer to this question is probably using node:
'use strict';
var fs = require('fs');
var readFilePromise = function(file) {
return new Promise(function(ok, notOk) {
fs.readFile(file, function(err, data) {
if (err) {
notOk(err)
} else {
ok(data)
}
})
})
}
readFilePromise('/etc/passwd').then(function(data) {
// do something with the data...
})
src
edit: setting it up for a simple application:
Make the server a serivce in rc.d or wherever. Follow a guide like this: https://blog.terminal.com/using-daemon-to-daemonize-your-programs/
Don't make the server a local service that is active! Instead, make a script to launch your app, and only from that script start the daemon. In your init script for the service, write a check to look for your app's PID or something every few minutes and autoshutdown when the app is no longer running.

Here is a code sample code for basic parsing of CSV you could try.
First step: Read the file.
We can read the file content using the FileReader class method readAsText, because the content in a CSV file is just some text .
Read more about FileReader here: https://developer.mozilla.org/en-US/docs/Web/API/FileReader
This code should be in an 'async' function. Because we have used 'await' to wait for the promise to resolve or reject.
Here the file variable is the File Object you have from the file input HTML element.
const fileContent = await(() => {
const promise = new Promise((resolve,reject) => {
const fileReader = new FileReader();
fileReader.onloadend = ()=>{
try {
const content = fileReader.result;
resolve(content);
} catch (error) {
reject(error);
}
};
fileReader.readAsText(file);
});
return promise;
})();
Second step: Transforming the file.
Here I transformed the file content into an array. A 2D array containing the CSV data.
/** extract the lines by splitting the text content by CRLF */
const linesArray = fileContent.split('\r\n');
const outcomeArray = [];
for (let rowIndex = 0; rowIndex < linesArray.length; rowIndex++) {
/** Checking whether the line is empty or not.
It's possible that there is a blank line in the CSV file.
We shall process only if not blank */
if (linesArray[rowIndex].trim()) {
/** Extract the cell out of the current line */
const currentline = linesArray[rowIndex].split(',').map((cellData, columnIndex) => {
/** Forming the data as an object. This can be customised as needed */
return {
rowIndex,
columnIndex,
value: cellData?.trim()
};
});
outcomeArray.push(currentline);
}
}
Example
If we parse a CSV having this content:
10,11
20,21
Output is a 2D array as below:
[
[
{
"rowIndex": 0,
"columnIndex": 0,
"value": "10"
},
{
"rowIndex": 0,
"columnIndex": 1,
"value": "11"
}
],
[
{
"rowIndex": 1,
"columnIndex": 0,
"value": "20"
},
{
"rowIndex": 1,
"columnIndex": 1,
"value": "21"
}
],
]

Related

How to add app names and paths to list in JSON file through electron-store

I'm trying to make a simple game launcher. Launches one specific game and tool apps from the list that is user customizable. Using electron-store and showOpenDialog, I select a tool executable, showOpenDialog takes a full path from Dialog and then it's made into var_toolName (simplified file name, path.basename+regex+replace to cut out part before first non-letter character) and var_toolPath (just full path stringified with join()). Those both variables are stored in config file through electron-store. One problem is I don't know how to append next tool executable to the list, instead replacing, and I've been trying for few days.
What I expect JSON list of apps inside config.json to be looking (just an idea, not specific coding):
tools: (so I know it's a part of config.json related to list of apps, overall config.json has few simple stuff stored such as window position, game executable path)
toolName1
toolPath1
toolName2
toolPath2
toolName3
toolPath3
and etc.,
Two roles of this JSON list is to make a HTMl list inside electron app (so user knows what tools are going to be launched together with game and, optionally, could toggle on/off specific tools and delete, I haven't reached that part yet, I'm guessing that would require some work on electrons-store and ipcMain/Renderer). Second role would be including full paths of tools to be launched together with game executable
Script for selecting tool executables and saving stuff to config.json:
const AppConfig = require('electron-store')
const appConfig = new AppConfig()
const path = require('path') //for executable/game folder path manipulations
//*****************************
// add tools
//*****************************
// 1. detect button click
document.getElementById('add').addEventListener('click', addTool);
//2. select tool exe + save path
function addTool() {
dialog.showOpenDialog({
title: 'Select tool executable.',
filters: [{
name: 'Tool start file',
extensions: ['exe', 'jar']
}],
properties: ['openFile']
},
(exeFromDialog) => {
var var_exeToolPath = exeFromDialog.join(); //removes square brackets
var var_toolName = path.basename(var_exeToolPath).split(/[/._-]/g)[0];
//path.basename removes path until file, split+regex takes only first part until first character (one of ._/)
appConfig.set(
"tools", {
"toolName": var_toolName,
"toolPath": var_exeToolPath
}
)
})
}
How current config.json look (the 'tools' part just gets replaced, instead being appended)
{
"winPosition": {
"x": 1130,
"y": 480,
"width": 202,
"height": 602
},
"exePOEPath": [
"C:\\Program Files (x86)\\Grinding Gear Games\\Path of Exile\\PathOfExile_x64.exe"
],
"tools": {
"toolName": "tool3",
"toolPath": "D:\\tool3.jar"
}
}
Current look of app (list is just a fake filler):
You need to get available tools first the append your new tool to it and save
See bellow,
let _tools = appConfig.get("tools");
if(!_tools){
_tools = {};
}
_tools[var_toolName] = var_exeToolPath;
appConfig.set(
"tools", _tools
)
So expected output will be,
{
.
.
.,
"tools": {
"tool1": "D:\\tool1.jar",
"tool2": "D:\\tool2.jar",
"tool3": "D:\\tool3.jar"
}
}

How to get producer of a PDF in google app script

I am trying to write a gmail add-on where I iterate over all emails and create a report based on their producers. Iterating over emails is the easiest part and I have done that, however I can't find any way to get producer line of each PDFs.
So far I tried
analyzing the blob, however this is something like writing a PDF library to parse all syntax. producer tag is not clearly present
adding pdf.js, which is a third party open source tool to extract such information. However, I couldn't add it due to ES3 - ES6 support issue.
What's the best way to get the producer line of a PDF in google app script?
Thank you
You want to retrieve the value of Producer from PDF file.
I could understand like above. If my understanding is correct, how about this sample script? In this sample script, from your shared PDF files, the value of Producer is retrieved by 2 regular expressions from the file content. Please think of this as one of several answers.
Sample script:
When you use this script, please set the folder ID of folder that PDF files are put. This script retrieves the value from all PDF files in a folder.
var folderId = "### folderId ###";
var files = DriveApp.getFolderById(folderId).getFilesByType(MimeType.PDF);
var regex = [/Producer\((\w.+)\)/i, /<pdf:Producer>(\w.+)<\/pdf:Producer>/i];
var result = [];
while (files.hasNext()) {
var file = files.next();
var content = file.getBlob().getDataAsString();
var r = regex.reduce(function(s, e) {
var m = content.match(e);
if (Array.isArray(m)) s = m[1];
return s;
}, "");
result.push({
fileName: file.getName(),
fileId: file.getId(),
vaueOfProducer: r,
});
}
Logger.log(result); // Result
Result:
This sample result was retrieved from a folder (my Google Drive) that the shared 3 PDF files were put.
[
{
"fileName": "2348706469653861032.pdf",
"fileId": "###",
"vaueOfProducer": "iText� 7.1.5 �2000-2019 iText Group NV \(iText; licensed version\)"
},
{
"fileName": "Getting started with OneDrive.pdf",
"fileId": "###",
"vaueOfProducer": "Adobe PDF library 15.00"
},
{
"fileName": "DITO-Salesflow-040419-1359-46.pdf",
"fileId": "###",
"vaueOfProducer": "iText 2.1.7 by 1T3XT"
}
]
Note:
About the file of 2348706469653861032.pdf, the characters which cannot be displayed are included in the value of Producer.
This is a sample script. So please modify this for your situation.

updating a file that is being used in a variable

I'm using a variable to store a json file that is used as a reference in my code. i have a code that checks if the variable is outdated or not. if it is outdated, it will be updated from its source.
const someFile = require('./something.json')
Everytime the file was outdated, the program tries to update it. and it was successful. however, nodejs kept using the old .json file (that has been replaced) as a reference. making my code output an outdated response.
so how can I tell node.js to use the updated file? thank you in advance!
//myjson.json
{
"data": {
"name": "uday",
"age": 25
}
}
controller.js
var myJSON = require('./myjson.json');
var fs = require('fs');
console.log("before change-->", myJSON);
myJSON.data.age = 26;
console.log("after change-->", myJSON);
fs.writeFileSync('myjson.json', JSON.stringify(myJSON));
console result
before change--> { data: { name: 'uday', age: 25 } }
after change--> { data: { name: 'uday', age: 26 } }
Your issue is with the const keyword, when a value gets assigned to a constant it cannot be changed (Info), just use let keyword and your code should work.
Update
If both files exist :
let someFile= require('./ab.json')
console.log(someFile); // Will Print content of ab.json
someFile= require('./bc')
console.log(someFile); // Will Print content of bc.json
But this would only work for already existing files on app start. So for a run time created file you cannot use require for that you need another option like the fs module (Docs Node 8.x)
P.S : While it's valid solution, if the file is a json and gets changed regularly you can just use redis and it's Keyspace Notifications if this is a large scale application if not fs is the way to go.
As described in Require Documentation, require will cache the file, so even if you update the file and use require again to get the file you wont see the changes.
Caching
Modules are cached after the first time they are loaded. This means
(among other things) that every call to require('foo') will get
exactly the same object returned, if it would resolve to the same
file.
Instead can try reading the file each time.
var fs = require('fs');
var data = fs.readFileSync('something.json', 'utf8');
var someFile = JSON.parse(data);

Electron will-download keeps getting interrupted

I am trying to download a file, but it keeps getting interrupted, and I have no idea why. I can not find any information on how to debug the reason it got interrupted either.
Here is where I am saving the file:
C:\Users\rnaddy\AppData\Roaming\Tachyon\games\murware\super-chain-reaction\web.zip
window.webContents.session.on('will-download', (event, item, webContents) => {
let path = url.parse(item.getURL()).pathname;
let dev = path.split('/')[3] || null;
let game = path.split('/')[4] || null;
if (!dev && !game) {
item.cancel();
} else {
item.setSavePath(Settings.fileDownloadLocation(dev, game, 'web'));
item.on('updated', (event, state) => {
let progress = 0;
if (state == 'interrupted') {
console.log('Download is interrupted but can be resumed');
} else if (state == 'progressing') {
progress = item.getReceivedBytes() / item.getTotalBytes();
if (item.isPaused()) {
console.log('Download is paused');
} else {
console.log(`Received bytes: ${item.getReceivedBytes()}; Progress: ${progress.toFixed(2)}%`);
}
}
});
}
});
Here is my listener that will trigger the above:
ipcMain.on(name, (evt) => {
window.webContents.downloadURL('http://api.gamesmart.com/v2/download/murware/super-chain-reaction');
});
Here is the output that I am getting in my console:
Received bytes: 0; Progress: 0.00%
Received bytes: 233183; Progress: 0.02%
Download is interrupted but can be resumed
I have a host file setup:
127.0.0.1 api.gamesmart.com
When I try to access the path http://api.gamesmart.com/v2/download/murware/super-chain-reaction in chrome, the file downloads just fine into my Downloads folder. So, what is causing this?
If you set the specific directory for downloading, you should use full file path with the file name in item.setSavePath() method. The best way to do it, fetching the file name from downloaditem object (item in your case) itself. You can use item.getFilename() to get the name of the current download item easily. here is the doc
And also there is a good way to get frequently used public system directory paths in electron. That is, using app.getPath(name) method. name would be the pre-defined String by electron for several directories. here is the doc
So, your complete setSavePath function would be,app.getPath("downloads") + "/" + item.getFilename()
In your case, if you are OK with your file path extraction method, only thing you are missing is filename at the end of the download path.
Of course you can use any other string as the file name if you wish. But remember to put correct extension though. :)
My solution was to use the correct Windows path separator (\), .e.g. 'directory\\file.zip'. Generally, Node.js uses / for any platform, but this seems to be sensitive about the path separator.

Unity WebGL External Assets

I'm developing some webGL project in Unity that has to load some external images from a directory, it runs all fine in the editor, however when I build it, it throws a Directory Not Found exception in web console. I am putting the images in Assets/StreamingAssets folder, that will become StreamingAssets folder in the built project (at root, same as index.html). Images are located there, yet browser still complains about not being able to find that directory. (I'm opening it on my own computer, no running web server)
I guess I'm missing something very obvious, but it seems like I could use some help, I've just started learning unity a week ago, and I'm not that great with C# or JavaScript (I'm trying to get better...) Is this somehow related to some javascript security issues?
Could someone please point me in the right direction, how I should be reading images(no writing need to be done) in Unity WebGL?
string appPath = Application.dataPath;
string[] filePaths = Directory.GetFiles(appPath, "*.jpg");
According to unity3d.com in webGL builds everything except threading and reflection is supported, so IO should be working - or so I thought:S
I was working around a bit and now I'm trying to load a text file containing the paths of the images (separated by ';'):
TextAsset ta = Resources.Load<TextAsset>("texManifest");
string[] lines = ta.text.Split(';');
Then I convert all lines to proper path, and add them to a list:
string temp = Application.streamingAssetsPath + "/textures/" + s;
filePaths.Add(temp);
Debug.Log tells me it looks like this:
file://////Downloads/FurnitureDresser/build/StreamingAssets/textures/79.jpg
So that seems to be allright except for all those slashes (That looks a bit odd to me)
And finally create the texture:
WWW www = new WWW("file://" + filePaths[i]);
yield return www;
Texture2D new_texture = new Texture2D(120, 80);
www.LoadImageIntoTexture(new_texture);
And around this last part (unsure: webgl projects does not seem easily debuggable) it tells me: NS_ERROR_DOM_BAD_URI: Access to restricted URI denied
Can someone please enlighten me what is happening? And most of all, what would be proper to solution to create a directory from where I can load images during runtime?
I realise this question is now a couple of years old, but, since this still appears to be commonly asked question, here is one solution (sorry, the code is C# but I am guessing the javascript implementation is similar). Basically you need to use UnityWebRequest and Coroutines to access a file from the StreamingAssets folder.
1) Create a new Loading scene (which does nothing but query the files; you could have it display some status text or a progress bar to let the user knows what is happening).
2) Add a script called Loader to the Main Camera in the Loading scene.
3) In the Loader script, add a variable to indicate whether the asset has been read successfully:
private bool isAssetRead;
4) In the Start() method of the Loading script:
void Start ()
{
// if webGL, this will be something like "http://..."
string assetPath = Application.streamingAssetsPath;
bool isWebGl = assetPath.Contains("://") ||
assetPath.Contains(":///");
try
{
if (isWebGl)
{
StartCoroutine(
SendRequest(
Path.Combine(
assetPath, "myAsset")));
}
else // desktop app
{
// do whatever you need is app is not WebGL
}
}
catch
{
// handle failure
}
}
5) In the Update() method of the Loading script:
void Update ()
{
// check to see if asset has been successfully read yet
if (isAssetRead)
{
// once asset is successfully read,
// load the next screen (e.g. main menu or gameplay)
SceneManager.LoadScene("NextScene");
}
// need to consider what happens if
// asset fails to be read for some reason
}
6) In the SendRequest() method of the Loading script:
private IEnumerator SendRequest(string url)
{
using (UnityWebRequest request = UnityWebRequest.Get(url))
{
yield return request.SendWebRequest();
if (request.isNetworkError || request.isHttpError)
{
// handle failure
}
else
{
try
{
// entire file is returned via downloadHandler
//string fileContents = request.downloadHandler.text;
// or
//byte[] fileContents = request.downloadHandler.data;
// do whatever you need to do with the file contents
if (loadAsset(fileContents))
isAssetRead = true;
}
catch (Exception x)
{
// handle failure
}
}
}
}
Put your image in the Resources folder and use Resources.Load to open the file and use it.
For example:
Texture2D texture = Resources.Load("images/Texture") as Texture2D;
if (texture != null)
{
GetComponent<Renderer>().material.mainTexture = texture;
}
The directory listing and file APIs are not available in webgl builds.
Basically no low level IO operations are supported.

Categories

Resources