Code
const Xray = require('x-ray');
const xray = Xray();
// How do I read a file, rather than a URL?
const url = 'https://www.tel-o-fun.ga/';
xray(url, '.marker')((err, value) => {
console.log(value);
});
My goal
I am using x-ray to scrape some date from a website. For testing and development purposes, I would like to parse data from a local file rather than a remote resource.
How do I load a local file into x-ray, instead of pointing it to a remote URL?
This example from the x-ray repo solved my problem. Simply pass an HTML string instead of a URL:
const path = require('path');
const Xray = require('x-ray');
const read = require('fs').readFileSync;
const html = read(path.resolve(__dirname, 'index.html'));
const xray = Xray();
xray(html, '.marker')((err, value) => {
console.log(value);
});
Related
I'm receiving events in JSON format via a POST route, I would like to save these events in a file like 'example.json' and be able to query it.
I tried using writeFileSync, but it rewrites the entire file. With the flag {flag: 'a+'} I was able to save more than one record, but when I try to require 'example.json', I get an error 'Unexpected token { in JSON'.
Works fine when the file has only one record, but gives the error after the second one.
Code:
const filePath = './example.json';
const fs = require('fs');
const file = require('./example.json');
app.post('/events', (request, response) => {
response.send(request.body);
const contentString = JSON.stringify(request.body);
return fs.writeFileSync(filepath, contentString, {flag: 'a+'});
});
example.json that works:
{"type":"call.new","call_id":"71252742562.40019","code":"h9e8j7c0tl0j5eexi07sy6znfd1ponj4","direction":"inbound","our_number":"1130900336","their_number":"11999990000","their_number_type":"mobile","timestamp":"2020-04-01T00:00:00Z"}
example.json (with two records) that stop working:
{"type":"call.new","call_id":"71252742562.40019","code":"h9e8j7c0tl0j5eexi07sy6znfd1ponj4","direction":"inbound","our_number":"1130900336","their_number":"11999990000","their_number_type":"mobile","timestamp":"2020-04-01T00:00:00Z"}{"type":"call.ongoing","call_id":"71252731962.40019","code":"h9e8j7c0tl0j5eexi07sy6znfd1ponj4","direction":"inbound","our_number":"1130900336","their_number":"11999990000","their_number_type":"mobile","timestamp":"2020-04-01T00:00:00Z"}
How can I write this JSON in a readable form? That does not present the error above and it is possible to perform the require.
Could someone help me, please?
Try to read the JSON file, parse it, add new elements to the array and then overwrite the file.
const fs = require("fs");
const path = require("path");
const FILE_PATH = path.join(__dirname, "./elements.json");
const file = fs.readFileSync(FILE_PATH);
const elements = JSON.parse(file);
const newElement = { id: Date.now() };
const updatedElements = [...elements, newElement];
fs.writeFileSync(FILE_PATH, JSON.stringify(updatedElements));
See more here: https://nodejs.org/api/fs.html#fsappendfilesyncpath-data-options
I am working on reactjs/typescript applications. I am trying to download some files from azure storage v2. Below is the sample path I am supposed to download files. In this path, enrichment is the container name, and the rest all are folders. I am trying to download the last modified file from reportdocument folder.
enrichment/report/SAR-1234-56/reportdocument/file1.docs
I tried something below.
#action
public async reportDownload(sarNumber: string) {
let storage = globals.getGlobals('StorageAccount03');
console.log(storage);
let containerName = globals.getGlobals('StorageAccount03ContainerName');
let marker = undefined;
let allUploadPromise: Array<Promise<unknown>> = [];
const config = {
path: `/Storage/getsastoken/?storageName=${storage}&containerName=${containerName}`,
method: "GET",
success: (url: any) => {
const containerURL: ContainerURL = new ContainerURL(
url,
StorageURL.newPipeline(new AnonymousCredential()));
const listBlobsResponse = containerURL.listBlobFlatSegment(
Aborter.none,
marker,
);
}
};
await handleRequest(config);
}
From here I am struggling to download the latest modified file from the above path.
can someone help me to fix this? Any help would be greatly appreciated. Thank you
It's better to use #azure/storage-blob library and then the code would be something like below instead of directly trying to call blob REST API like you were trying in your code which seems unnecessary reinventing the wheel. The library already does it for you. Refer this for details.
const { BlobServiceClient } = require("#azure/storage-blob");
const account = "<account name>";
const sas = "<service Shared Access Signature Token>";
const containerName = "<container name>";
const blobName = "<blob name>";
const blobServiceClient = new BlobServiceClient(`https://${account}.blob.core.windows.net${sas}`);
async function download() {
const containerClient = blobServiceClient.getContainerClient(containerName);
const blobClient = containerClient.getBlobClient(blobName);
// Get blob content from position 0 to the end
// In browsers, get downloaded data by accessing downloadBlockBlobResponse.blobBody
const downloadBlockBlobResponse = await blobClient.download();
const downloaded = await blobToString(await downloadBlockBlobResponse.blobBody);
console.log("Downloaded blob content", downloaded);
// [Browsers only] A helper method used to convert a browser Blob into string.
async function blobToString(blob) {
const fileReader = new FileReader();
return new Promise((resolve, reject) => {
fileReader.onloadend = (ev) => {
resolve(ev.target.result);
};
fileReader.onerror = reject;
fileReader.readAsText(blob);
});
}
}
The SAS token expiry bothers me.You cannot have a static SAS token that expires sooner unless we can set long expiry (user-delegation SAS token is short lived). Do we really have the capability to create the SAS token dynamically in javascript runtime? I think it's only possible in NodeJS runtime.
Using the ldapsj-client module, I'm trying to save the thumbnailPhoto into a file
const auth = async () => {
const client = new LdapClient({ url: 'myaddomain' })
await client.bind('someemail#domain.com.br', 'passwaord')
const opts = {
filter: `(sAMAccountName=credential)`,
scope: "sub"
}
const s = await client.search(myBaseDN, opts)
console.log('thumbnailPhoto', s[0].thumbnailPhoto)
}
The console.log() outputs something like '����JFIF��C...'
I cannot figure out how to save this binary into a file. When I try several approaches, as explained here, does not work. It seems the data from AD is not in the same "format".
I tried to convert it into a Buffer and then, to base64
const buffer = Buffer.from(s[0].thumbnailPhoto, 'binary')
var src = "data:image/png;base64," + Buffer.from(s[0].thumbnailPhoto).toString('base64')
But the output is not a valid base64.
I need to extract text data from web url (http://www.africau.edu/images/default/sample.pdf)
I used two node_module.
1) crawler-Request
it('Read Pdf Data using crawler',function(){
const crawler = require('crawler-request');
function response_text_size(response){
response["size"] = response.text.length;
return response;
}
crawler("http://www.africau.edu/images/default/sample.pdf",response_text_size).then(function(response){
// handle response
console.log("Reponse =" + response.size);
});
});
What happen for this it will not print anything on console.
2) pfd2json/pdfparser
it('Read Data from url',function(){
var request = require('request');
var pdf = require('pfd2json/pdfparser');
var fs = require('fs');
var pdfUrl = "http://www.africau.edu/images/default/sample.pdf";
let databuffer = fs.readFileSync(pdfUrl);
pdf(databuffer).then(function(data){
var arr:Array<String> = data.text;
var n = arr.includes('Thursday 02 May');
console.log("Print Array " + n);
});
});
Failed: ENOENT: no such file or directory, open 'http://www.africau.edu/images/default/sample.pdf'
I am able to access data from local path but not able to extract it from url.
The issue here is that you are using the fs module (File System) to read a file on a distant server.
You also mistyped the pdf2json module, which should give you an error ?
You did require the request module. This module will make it possible to access that distant file. Here's one way to do this :
it('Read Data from url', function () {
var request = require('request');
var PDFParser = require('pdf2json');
var pdfUrl = 'http://unec.edu.az/application/uploads/2014/12/pdf-sample.pdf';
var pdfParser = new PDFParser(this, 1);
// executed if the parser fails for any reason
pdfParser.on("pdfParser_dataError", errData => console.error(errData.parserError));
// executed when the parser finished
pdfParser.on("pdfParser_dataReady", pdfData => console.log(pdfParser.getRawTextContent()));
// request to get the pdf's file content then call the pdf parser on the retrieved buffer
request({ url: pdfUrl, encoding: null }, (error, response, body) => pdfParser.parseBuffer(body));
});
This will make it possible to load the distant .pdf file in your program.
I'd recommend looking at the pdf2json documentation if you want to do more. This will simply output the textual content of the .pdf file when the parser has completed reading data.
I developed a desktop application with nwjs (nodejs / html / css ), now i want to put the app for the production so i need to prevent stealing my assets (my images are very valuables), nwjs provide a tool to compile (encrypt) the js files but not the asset so i thought about encrypting my assets with a js then encrypt the js with nwjs tool, i am not very familiare with node modules and dealing with files in js so i struggled with this task !
This code is what i tried to do but i did not reach my goal ?
encrypt
let crypto;
try {
crypto = require('crypto');
} catch (err) {
console.log('crypto support is disabled!');
}
var algorithm = 'aes-256-ctr',
password = 'secret';
var fs = require('fs');
var r;
// encrypt content
var encrypt = crypto.createCipher(algorithm, password);
// decrypt content
var decrypt = crypto.createDecipher(algorithm, password);
// write file
var w;
var path = require('path');
var dirPath = './Files/'; //directory path
var fileType = '.' + 'jpg'; //file extension
var files = [];
fs2.readdir(dirPath, function (err, list) {
if (err) throw err;
for (var i = 0; i < list.length; i++) {
if (path.extname(list[i]) === fileType) {
r = fs.createReadStream('Files/' + list[i]);
w = fs.createWriteStream('encFiles/' + list[i].replace('.jpg', ''));
console.log(list[i]); //print the file
// start pipe
r.pipe(encrypt).pipe(w);
}
}
});
decrypt
'use strict';
var crypt = require('crypto'),
algorithm = 'aes-256-ctr',
password = 'secret';
var fs = require('fs');
var zlib = require('zlib');
var toArray = require('stream-to-array');
// input file
var r = fs.createReadStream('./encFiles/an example file');
// decrypt content
var decrypt = crypt.createDecipher(algorithm, password);
//b64 module so i could put the base64 data to img html element
const B64 = require('b64');
const encoder = new B64.Encoder();
// start pipe
var stream = r.pipe(decrypt);
var d = stream.pipe(encoder);
d.pipe(process.stdout);
var data;
toArray(stream, function(err, arr) {
console.log(err,arr);
data = Buffer.concat(arr);
console.log(data);
});
console.log(data);
thank you for giving me comments on the code or other IDEAS
So the solution was so simple, I used the nw-js code protection feature to protected the script in which I decrypt the assets (images in my case) (this script contains the key of decryption), so you could implement the encryption/decryption with any method you want and protect the decryption script which is going to be shipped with you product (in my case the desktop app).
Since you are building a desktop app, you may want to look at cryptojs for this. I would still strongly suggest that you watermark images and hide them when your application looses focus. Even with that, screenshots can be taken without leaving your application.