I try to download, unzip and process a GTFS file in zip format. Downloading and unzipping are working, but I get error message when try to use txt files with gtfs-utils module in gtfsFunc(). Output is undefined. Delays are hardcoded just for testing purpose.
const dl = new DownloaderHelper('http://www.bkk.hu/gtfs/budapest_gtfs.zip', __dirname);
dl.on('end', () => console.log('Download Completed'))
dl.start();
myVar = setTimeout(zipFunc, 30000);
function zipFunc() {
console.log('Unzipping started...');
var zip = new AdmZip("./budapest_gtfs.zip");
var zipEntries = zip.getEntries();
zip.extractAllTo("./gtfsdata/", true);
}
myVar = setTimeout(gtfsFunc, 40000);
function gtfsFunc() {
console.log('Processing started...');
const readFile = name => readCsv('./gtfsdata/' + name + '.txt')
const filter = t => t.route_id === 'M4'
readStops(readFile, filter)
.then((stops) => {
const someStopId = Object.keys(stops)[0]
const someStop = stops[someStopId]
console.log(someStop)
})
}
As #ChickenSoups said, you are trying to filter stops files with route_id field and this txt doesnt have this field.
The fields that stops has are:
stop_id, stop_name, stop_lat, stop_lon, stop_code, location_type, parent_station, wheelchair_boarding, stop_direction
Perhaps what you need is read the Trips.txt file instead Stops.txt, as this file has route_id field.
And you can accomplish this using readTrips function:
const readTrips = require("gtfs-utils/read-trips");
And your gtfsFunc would be:
function gtfsFunc() {
console.log("Processing started...");
const readFile = (name) => {
return readCsv("./gtfsdata/" + name + ".txt").on("error", console.error);
};
//I used 5200 because your Trips.txt contains routes id with this value
const filterTrips = (t) => t.route_id === "5200";
readTrips(readFile, filterTrips).then((stops) => {
console.log("filtered stops", stops);
const someStopId = Object.keys(stops)[0];
const someStop = stops[someStopId];
console.log("someStop", someStop);
});
}
Or if what you really want is to read Stops.txt, you just need to change your filter
const filter = t => t.route_id === 'M4'
to use some valid field, for example:
const filter = t => t.stop_name=== 'M4'
Stop data don't have route_id field.
You should try other data, such as Trip or Route
You can look at the first row in your data file to see which field do they have.
GTFS data structure here
Related
I am trying to count number of pages pdf file by create function by not using inbuilt methods.
const fs = require("fs");
const pdf = require("pdf-parse");
let data = fs.readFileSync("pdf/book.pdf");
const countpage = (str) => {
return str.reduce((el) => {
return el + 1;
}, 0);
};
pdf(data).then(function (data) {
console.log(countpage(data.numpages));
});
i am try to output as like number of pages =272 like this when i use data.numpages i got result
but when same think by create function and count then i didn't got result
output like this {numberpage:273}
I'm trying to replicate the code in this article:
https://depth-first.com/articles/2020/08/24/smiles-validation-in-the-browser/
What I'm trying to do different is that I'm using a textarea instead of input to take multi-line input. In addition to displaying an error message, I also want to display the entry which doesn't pass the validation.
The original validation script is this:
const path = '/target/wasm32-unknown-unknown/release/smival.wasm';
const read_smiles = instance => {
return smiles => {
const encoder = new TextEncoder();
const encoded = encoder.encode(`${smiles}\0`);
const length = encoded.length;
const pString = instance.exports.alloc(length);
const view = new Uint8Array(
instance.exports.memory.buffer, pString, length
);
view.set(encoded);
return instance.exports.read_smiles(pString);
};
};
const watch = instance => {
const read = read_smiles(instance);
document.querySelector('input').addEventListener('input', e => {
const { target } = e;
if (read(target.value) === 0) {
target.classList.remove('invalid');
} else {
target.classList.add('invalid');
}
});
}
(async () => {
const response = await fetch(path);
const bytes = await response.arrayBuffer();
const wasm = await WebAssembly.instantiate(bytes, { });
watch(wasm.instance);
})();
For working with a textarea, I've changed the watch function to this and added a <p id="indicator"> element to the html to display an error:
const watch = instance => {
const read = read_smiles(instance);
document.querySelector("textarea").addEventListener('input', e => {
const { target } = e;
var lines_array = target.value.split('/n');
var p = document.getElementById("indicator");
p.style.display = "block";
p.innerHTML = "The size of the input is : " + lines_array.length;
if (read(target.value) === 0) {
target.classList.remove('invalid');
} else {
target.classList.add('invalid');
}
});
}
I'm not even able to get a count of entries that fail the validation. I believe this is async js and I'm just a beginner in JavaScript so it's hard to follow what is happening here, especially the part where the function e is referencing itself.
document.querySelector("textarea").addEventListener('input', e => {
const { target } = e;
Can someone please help me in understanding this complicated code and figuring out how to get a count of entries that fail the validation and also printing the string/index of the same for helping the user?
There is a mistake in you code to count entries in the textarea:
var lines_array = target.value.split('\n'); // replace /n with \n
You are asking about the function e is referencing itself:
The destructuring assignment syntax is a JavaScript expression that makes it possible to unpack values from arrays, or properties from objects, into distinct variables. You can find more informations Mdn web docs - Destructuring object
How to combine these two codes, so it doesn't just covert csv to Json (first code), but also save this as an json array in an extra file?(second code)
this (first) code converts csv file to json array:
const fs = require("fs");
let fileReadStream = fs.createReadStream("myCsvFile.csv");
let invalidLineCount = 0;
const csvtojson = require("csvtojson");
csvtojson({ "delimiter": ";", "fork": true })
.preFileLine((fileLineString, lineIdx)=> {
let invalidLinePattern = /^['"].*[^"'];/;
if (invalidLinePattern.test(fileLineString)) {
console.log(`Line #${lineIdx + 1} is invalid, skipping:`, fileLineString);
fileLineString = "";
invalidLineCount++;
}
return fileLineString
})
.fromStream(fileReadStream)
.subscribe((dataObj) => {
console.log(dataObj);
// I added the second code hier, but it wirtes the last object of the array (because of the loop?)
}
});
and this (second) code saves the json array to an external file:
fs.writeFile('example.json', JSON.stringify(dataObj, null, 4);
The quistion is how to put the second codes into the first code (combine them)?
You can use .on('done',(error)=>{ ... }) method. (csvtojson). Push the data into a variable in subscribe method and write the data as JSON in .on('done'). (test was successful).
Check it out:
let fileReadStream = fs.createReadStream("username-password.csv");
let invalidLineCount = 0;
let data = []
csvtojson({ "delimiter": ";", "fork": true })
.preFileLine((fileLineString, lineIdx)=> {
let invalidLinePattern = /^['"].*[^"'];/;
if (invalidLinePattern.test(fileLineString)) {
console.log(`Line #${lineIdx + 1} is invalid, skipping:`, fileLineString);
fileLineString = "";
invalidLineCount++;
}
return fileLineString
})
.fromStream(fileReadStream)
.subscribe((dataObj) => {
// console.log(dataObj)
data.push(dataObj)
})
.on('done',(error)=>{
fs.writeFileSync('example.json', JSON.stringify(data, null, 4))
})
Not sure if you are able to change the library but I would definitely recommend Papaparse for this - https://www.npmjs.com/package/papaparse
Your code would then look something like this:
const fs = require('fs'), papa = require('papaparse');
var readFile = fs.createReadStream(file);
papa.parse(readFile, {
complete: function (results, file) {
fs.writeFile('example.json', JSON.stringifiy(results.data), function (err) {
if(err) console.log(err);
// callback etc
})
}
});
This code currently takes the unique ID from the form response and puts it as the folder name where all the uploaded file goes to. I want to change the code so that it takes a text field from the Google Form response for organized view, such as First Name_Last Name.
const PARENT_FOLDER_ID = "2ih2384723847234878h23jkbi2j3b4ijb23i4bhwed34f";
const initialize = () => {
const form = FormApp.getActiveForm();
ScriptApp.newTrigger("onFormSubmit").forForm(form).onFormSubmit().create();
};
const onFormSubmit = ({ response } = {}) => {
try {
// Get a list of all files uploaded with the response
const files = response
.getItemResponses()
// We are only interested in File Upload type of questions
.filter(
(itemResponse) =>
itemResponse.getItem().getType().toString() === "FILE_UPLOAD"
)
.map((itemResponse) => itemResponse.getResponse())
// The response includes the file ids in an array that we can flatten
.reduce((a, b) => [...a, ...b], []);
if (files.length > 0) {
// Each form response has a unique Id
const subfolderName = response.getId();
const parentFolder = DriveApp.getFolderById(PARENT_FOLDER_ID);
const subfolder = parentFolder.createFolder(subfolderName);
files.forEach((fileId) => {
// Move each file into the custom folder
DriveApp.getFileById(fileId).moveTo(subfolder);
});
}
} catch (f) {
Logger.log(f);
}
};
Let's say the names are the first and second questions.
/* ... */
const n = response.getItemResponses().slice(0, 2)
const fn = n[0].getResponse();
const ln = n[1].getResponse();
/* ... */
const subfolderName = `${fn}_${ln}`;
/* ... */
I have a CSV file can contain around million records, how can I remove columns starting with _ and generate a resulting csv
For the sake of simplicity, consider i have the below csv
Sr.No Col1 Col2 _Col3 Col4 _Col5
1 txt png 676766 win 8787
2 jpg pdf 565657 lin 8787
3 pdf jpg 786786 lin 9898
I would want the output to be
Sr.No Col1 Col2 Col4
1 txt png win
2 jpg pdf lin
3 pdf jpg lin
Do i need to read the entire file to achive this or is there a better approach to do this.
const csv = require('csv-parser');
const fs = require('fs');
fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (row) => {
// generate a new csv with removing specific column
})
.on('end', () => {
console.log('CSV file successfully processed');
});
Any help on how can i achieve this would be helpful.
Thanks.
To anyone who stumbles on the post
I was able to transform the csv's using below code using fs and csv modules.
await fs.createReadStream(m.path)
.pipe(csv.parse({delimiter: '\t', columns: true}))
.pipe(csv.transform((input) => {
delete input['_Col3'];
console.log(input);
return input;
}))
.pipe(csv.stringify({header: true}))
.pipe(fs.createWriteStream(transformedPath))
.on('finish', () => {
console.log('finish....');
}).on('error', () => {
console.log('error.....');
});
Source: https://gist.github.com/donmccurdy/6cbcd8cee74301f92b4400b376efda1d
Actually you can handle that by using two npm packages.
https://www.npmjs.com/package/csvtojson
to convert your library to JSON format
then use this
https://www.npmjs.com/package/json2csv
with the second library. If you know what are the exact fields you want. you can pass parameters to specifically select the fields you want.
const { Parser } = require('json2csv');
const fields = ['field1', 'field2', 'field3'];
const opts = { fields };
try {
const parser = new Parser(opts);
const csv = parser.parse(myData);
console.log(csv);
} catch (err) {
console.error(err);
}
Or you can modify the JSON object manually to drop those columns
Try this with csv lib
const csv = require('csv');
const fs = require('fs');
const csvString=`col1,col2
value1,value2`
csv.parse(csvString, {columns: true})
.pipe(csv.transform(({col1,col2}) => ({col1}))) // remove col2
.pipe(csv.stringify({header:true}))
.pipe(fs.createWriteStream('./file.csv'))
With this function I accomplished the column removal from a CSV
removeCol(csv, col) {
let lines = csv.split("\n");
let headers = lines[0].split(",");
let colNameToRemove = headers.find(h=> h.trim() === col);
let index = headers.indexOf(colNameToRemove);
let newLines = [];
lines.map((line)=>{
let fields = line.split(",");
fields.splice(index, 1)
newLines.push(fields)
})
let arrData = '';
for (let index = 0; index < newLines.length; index++) {
const element = newLines[index];
arrData += element.join(',') + '\n'
}
return arrData;
}