Ok so I have attempted to use multiple XML libraries that NodeJS have to offer and I can't seem to work out how to have an NodeJS read the XML file from a website.
I can pull the file using http.request, http.get and all of that but then to have NodeJS be able to actually do anything with the data in the XML file is another story.
I'm sure I must be missing something as when ever I turn the XML to JS with xml-stream; it can not use it from a website; my code runs when I host the file however I am using an api and they only use XML.
Current code:
var http = require('http');
var XmlStream = require('xml-stream');
var options = { host: 'cloud.tfl.gov.uk',
path: '/TrackerNet/LineStatus'};
var twitter = { host: 'api.twitter.com',
path: '/1/statuses/user_timeline.rss?screen_name=nwhite89'}
var request = http.get(options).on('response', function(response) {
response.setEncoding('utf8');
var xml = new XmlStream(response);
xml.on('updateElement: item', function(item) {
item.title = item.title.match(/^[^:]+/)[0] + ' on ' +
item.pubDate.replace(/ +[0-9]{4}/, '');
});
xml.on('text: item > pubDate', function(element) {
element.$text = element.$text;
});
xml.on('data', function(data) {
process.stdout.write(data);
});
});
What I don't understand is using Twitter works fine outputs at xml.on("data") part however using options (cloud.tfl.gov.uk) nothing outputs even if I put console.log("hi") inside the data function it dosn't get executed.
I know that the url is correct outputting console.log(xml) or console.log(response) after creating the variable xml outputs that it has connected. Any help would be greatly appreciated with this I have been stuck on this for a good 2 days now.
There is a byte order mark before the <?xml tag, which xml-stream trips up on a bit and stops it from being able to read the encoding in the tag. That means you need to provide it yourself.
Instead of this:
response.setEncoding('utf8');
var xml = new XmlStream(response);
Just do this:
response.setEncoding('utf8');
var xml = new XmlStream(response, 'utf8');
And really, setting the encoding on the stream is optional.
var xml = new XmlStream(response, 'utf8');
works just fine.
More info here: http://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
If you look at the buffer emitted from response rather that xml, the buffer starts with
<Buffer ef bb bf 3c 3f 78 6d ...>
The first 3 bytes are the byte order mark for utf8, and afterwards you have the start of the tag. xml-stream expects the <?xml tag to only have whitespace between it and the start of the file, but byte order marks don't count as whitespace.
Related
I'm trying to write in a text file, but not at the end like appendFile() do or by replacing the entiere content...
I saw it was possible to chose where you want to start with start parameter of fs.createwritestream() -> https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options
But there is no parameter to say where to stop writting, right ? So it remove all the end of my file after I wrote with this function.
const fs = require('fs');
var logger = fs.createWriteStream('result.csv', {
flags: 'r+',
start: 20 //start to write at the 20th caracter
})
logger.write('5258,525,98951,0,1\n') //example a new line to write
Is there a way to specify where to stop writting in the file to have something like:
....
data from begining
....
5258,525,98951,0,1
...
data till the end
...
I suspect you mean, "Is it possible to insert in the middle of the file." The answer to that is: No, it isn't.
Instead, to insert, you have to:
Determine how big what you're inserting is
Copy the data at your insertion point to that many bytes later in the file
Write your data
Obviously when doing #2 you need to be sure that you're not overwriting data you haven't copied yet (either by reading it all into memory first or by working in blocks, from the end of the file toward the insertion point).
(I've never looked for one, but there may be an npm module out there that does this for you...)
You could read/parse your file at first. Then apply the modifications and save the new file.
Something like:
const fs = require("fs");
const fileData = fs.readFileSync("result.csv", { encoding: "utf8" });
const fileDataArray = fileData.split("\n");
const newData = "5258,525,98951,0,1";
const index = 2; // after each row to insert your data
fileDataArray.splice(index, 0, newData); // insert data into the array
const newFileData = fileDataArray.join("\n"); // create the new file
fs.writeFileSync("result.csv", newFileData, { encoding: "utf8" }); // save it
I'm trying to convert ansi color codes from console output into HTML. I have a found a script to do this but I cant seem to make it parse the strings inside node js. I have tried to JSON.stringify it to also include special chars but its not working.
forever list
[32minfo[39m: Forever processes running
[90mscript[39m [37mforever[39m [37mpid[39m [37mid[39m
[90mdata[39m: [37m [39m [37muid[39m [90mcommand[39m
I get output like this back from ssh2shell in node js. I have a script:
https://github.com/pixelb/scripts/blob/master/scripts/ansi2html.sh
This is supposed to convert the above to html and add the appropriate color codes. It works fine with normal terminal output for example:
npm install --color=always | ansi2html.sh > npminstall.html
This is the raw output on the linux machine piped to a file. It seems the JS strings are missing these escapes when they are shown in console.log but they are also missing newlines there. Perhaps its because im concatenating them directly into the string and its removing special chars?
total 24
-rwxr-xr-x 1 admin admin 17002 May 13 02:52 ^[[0m^[[38;5;34mansi2html.sh^[[0m
drwxr-xr-x 4 admin admin 4096 May 13 00:00 ^[[38;5;27mgit^[[0m
-rw-r--r-- 1 admin admin 0 May 13 02:57 ls.html
Hopefully some of this makes sense.
Thanks
There are a couple of filters that SSH2shell applies to the output from commands. The first removes non-standard ASCII from the response and then the colour formatting codes are removed.
In v1.6.0 I have added pipe()/unpipe(), the events for both and exposed the stream.on('data', function(data){}) event so you can access the stream output directly without SSH2shell interacting with it in any way.
This should resolve the problem of not getting the right output from SSH2shell by giving you access to the raw data.
var fs = require('fs')
var host = {
server: {
host: mydomain.com,
port: 22,
userName: user,
password: password:)
},
commands: [
"`Test session text message: passed`",
"msg:console test notification: passed",
"ls -la"
],
}
//until npm published use the cloned dir path.
var SSH2Shell = require ('ssh2shell')
//run the commands in the shell session
var SSH = new SSH2Shell(host),
callback = function( sessionText ){
console.log ( "-----Callback session text:\n" + sessionText);
console.log ( "-----Callback end" );
},
firstLog = fs.createWriteStream('first.log'),
secondLog = fs.createWriteStream('second.log'),
buffer = ""
//multiple pipes can be added but they wont be bound to the stream until the connection is established
SSH.pipe(firstLog).pipe(secondLog);
SSH.on('data', function(data){
//do something with the data chunk
console.log(data)
})
SSH.connect(callback)
tried this ?
https://github.com/hughsk/ansi-html-stream
var spawn = require('child_process').spawn
, ansi = require('ansi-html-stream')
, fs = require('fs')
var npm = spawn('npm', ['install', 'browserify', '--color', 'always'], {
cwd: process.cwd()
})
var stream = ansi({ chunked: false })
, file = fs.createWriteStream('browserify.html', 'utf8')
npm.stdout.pipe(stream)
npm.stderr.pipe(stream)
stream.pipe(file, { end: false })
stream.once('end', function() {
file.end('</pre>\n')
})
file.write('<pre>\n');
I want to write some data in an existing local XML file with Javascript with some text from an Html page. Is it possible to change content of nodes?
Here is XML sample:
<Notepad>
<Name>Player1</Name>
<Notes>text1</Notes>
</Notepad>
I will get some more text from input and want to add it after "text1", but can't find a solution.
function SaveNotes(content,player)
{
var xml = "serialize.xml";
var xmlTree = parseXml("<Notepad></Notepad>");
var str = xmlTree.createElement("Notes");
$(xmlTree).find("Notepad").find(player).append(str);
$(xmlTree).find("Notes").find(player).append(content);
var xmlString = (new XMLSerializer()).serializeToString(xmlTree);
}
Here is the code to manipulate xml content or xml file :
[Update]
Please check this Fiddle
var parseXml;
parseXml = function(xmlStr) {
return (new window.DOMParser()).parseFromString(xmlStr, "text/xml");
};
var xmlTree = parseXml("<root></root>");
function add_children(child_name, parent_name) {
str = xmlTree.createElement(child_name);
//strXML = parseXml(str);
$(xmlTree).find(parent_name).append(str);
$(xmlTree).find(child_name).append("hello");
var xmlString = (new XMLSerializer()).serializeToString(xmlTree);
alert(xmlString);
}
add_children("apple", "root");
add_children("orange", "root");
add_children("lychee", "root");
you can use it for searching in xml as well as adding new nodes with content in it. (And sorry i dont know how to load xml from client side and display it.)
but this fiddle demo will be helpful in adding content in xml and searching in it.
Hope it helps :)
If you want to achieve this on the client side you can parse your xml into a document object:
See
https://developer.mozilla.org/en-US/docs/Web/Guide/Parsing_and_serializing_XML
and
http://www.w3schools.com/xml/tryit.asp?filename=tryxml_parsertest2
And then manipulate it like you would the DOM of any html doc, e.g. createElement, appendChild etc.
See https://developer.mozilla.org/en-US/docs/Web/API/Document/createElement
Then to serialize it into a String again you could use https://developer.mozilla.org/en-US/docs/Web/API/Element/outerHTML
Persisting the data
Writing to a local file is not possible in a cross-browser way. In IE you could use ActiveX to read/write file.
You could use cookies to store data on the client side, if your data keeps small enough.
In HTML5 you could use local storage, see http://www.w3schools.com/html/html5_webstorage.asp
Try to use these two package one to convert to json and when is finish the other to come back
https://www.npmjs.com/package/xml2json
https://www.npmjs.com/package/js2xmlparser
I am using PDFKit and NodeJS to dynamically generate PDF documents, and I would like to attach said document to an email. At this moment I'm using Mandril-API via NPM.
I can generate the PDF without issue and display it in the browser via:
doc.pipe( res );
I can send an email without issue, but I have failed miserably at getting the proper PDF content. I am fairly certain that I am 99% of the way there - but I'm missing something. I have done a ton of reading and testing using Google/StackOverflow etc but I'm stuck.
I am getting content that when I do a base64 decode I get:
%PDF-1.3 % 7 0 obj << /Predictor 15
I have managed to get my PDF attachment to have a valid size of 445KB
but this is the content of the email:
--_av-Ti-H6i8tBBHL4BgoXnyC2Q Content-Type: application/pdf Content-Transfer-Encoding: base64 Content-Disposition: attachment;
filename="mytestPDF.pdf"
PDF1370obj/Predictor15/Colors1/BitsPerComponent8/Columns100e
ndobj60obj/Type/XObject/Subtype/Image/BitsPerComponent8/Widt
h100/Height19/Filter/FlateDecode/DecodeParms70R/ColorSpace/I
ndexed/DeviceRGB25580R/Length1751streamxdSSNEhGIRTRkWbY/nHaO
MJln7t+vv89ylF111PlYNB9Nm6e9DENsd9FxLFUbOjrgt+ErRgWtj9vPCTBH
oohMHl9oZ7IdpC/hxusjTHFFMcxhwIxPlbNorOB+bH8exrrA1DUnzKzq/UXI
xT456nxtB59fQNiIrBT2apETJZieZvltpeThrObiZ4ydtY0koKJ2Epb940A1
iXyehONQVXiZr8jRP/NJ3bmjHA0sygAou4Q=
Although I've messed around for hours on this, my best hunch is that I have line break/new line errors in my PDF content. The way I'm getting my PDF content is by creating an array called buffers, then: doc.on('data', buffers.push.bind(buffers));
I'm supposing that I need to be adding /n or /r etc...but I've been working with NodeJS and AngularJS for a month or so now and I know almost EVERYTHING I do wrong is because I'm over-complicating the matter...so I turn to you folks and hope that there is a simple method to attach the content from the new PDFDocument I create with PDFKit to an email using NodeJS.
Thank you in advance...please forgive my rambling, but I started this about 8 hours ago (it's now 3:25am my time). :)
I recently had the same problem with posting an email pdf attachment to mandrill through node.js but managed to solve it.
Here is what I did:
generatePdf(inputData, function (err, doc) {
if (err) return callback(err);
var bufferChunks = [];
doc.on('readable', function() {
// Store buffer chunk to array
bufferChunks.push(doc.read());
});
doc.on('end', function() {
var pdfBuffer = Buffer.concat(bufferChunks),
pdfBase64String = pdfBuffer.toString('base64');
// This string is perfectly ok to use as an attachment to the mandrillAPI
sendMandrillEmailWithAttachment(pdfBase64String);
});
});
I hope this helps. Ping if you need additional help =)
I'm trying to write a string to a socket (socket is called "response"). Here is the code I have sofar (I'm trying to implement a byte caching proxy...):
var http = require('http');
var sys=require('sys');
var localHash={};
http.createServer(function(request, response) {
var proxy = http.createClient(80, request.headers['host'])
var proxy_request = proxy.request(request.method, request.url, request.headers);
proxy_request.addListener('response', function (proxy_response) {
proxy_response.addListener('data', function(x) {
var responseData=x.toString();
var f=50;
var toTransmit="";
var p=0;
var N=responseData.length;
if(N>f){
p=Math.floor(N/f);
var hash="";
var chunk="";
for(var i=0;i<p;i++){
chunk=responseData.substr(f*i,f);
hash=DJBHash(chunk);
if(localHash[hash]==undefined){
localHash[hash]=chunk;
toTransmit=toTransmit+chunk;
}else{
sys.puts("***hit"+chunk);
toTransmit=toTransmit+chunk;//"***EOH"+hash;
}
}
//remainder:
chunk=responseData.substr(f*p);
hash=DJBHash(chunk);
if(localHash[hash]==undefined){
localHash[hash]=chunk;
toTransmit=toTransmit+chunk;
}else{
toTransmit=toTransmit+chunk;//"***EOH"+hash;
}
}else{
toTransmit=responseData;
}
response.write(new Buffer(toTransmit)); /*error occurs here */
});
proxy_response.addListener('end', function() {
response.end();
});
response.writeHead(proxy_response.statusCode, proxy_response.headers);
});
request.addListener('data', function(chunk) {
sys.puts(chunk);
proxy_request.write(chunk, 'binary');
});
request.addListener('end', function() {
proxy_request.end();
});
}).listen(8080);
function DJBHash(str) {
var hash = 5381;
for(var i = 0; i < str.length; i++) {
hash = (((hash << 5) + hash) + str.charCodeAt(i)) & 0xffffffff;
}
if(hash<-1){
hash=hash*-1;
}
return hash;
}
The trouble is, I keep getting a "content encoding error" in Firefox. It's as if the gizipped content isn't being transmitted properly. I've ensured that "toTransmit" is the same as "x" via console.log(x) and console.log(toTransmit).
It's worth noting that if I replace response.write(new Buffer(toTransmit)) with simply response.write(x), the proxy works as expected, but I need to do some payload analysis and then pass "toTransmit", not "x".
I've also tried to response.write(toTransmit) (i.e. without the conversion to buffer) and I keep getting the same content encoding error.
I'm really stuck. I thought I had this problem fixed by converting the string to a buffer as per another thread (http://stackoverflow.com/questions/7090510/nodejs-content-encoding-error), but I've re-opened a new thread to discuss this new problem I'm experiencing.
I should add that if I open a page via the proxy in Opera, I get gobblydeegook - it's as if the gzipped data gets corrupted.
Any insight greatly appreciated.
Many thanks in advance,
How about this?
var responseData = Buffer.from(x, 'utf8');
from: Convert string to buffer Node
Without digging very deep into your code, it seems to me that you might want to change
var responseData=x.toString();
to
var responseData=x.toString("binary");
and finally
response.write(new Buffer(toTransmit, "binary"));
From the docs:
Pure Javascript is Unicode friendly but not nice to binary data. When
dealing with TCP streams or the file system, it's necessary to handle
octet streams. Node has several strategies for manipulating, creating,
and consuming octet streams.
Raw data is stored in instances of the Buffer class. A Buffer is
similar to an array of integers but corresponds to a raw memory
allocation outside the V8 heap. A Buffer cannot be resized.
So, don't use strings for handling binary data.
Change proxy_request.write(chunk, 'binary'); to proxy_request.write(chunk);.
Omit var responseData=x.toString();, that's a bad idea.
Instead of doing substr on a string, use slice on a buffer.
Instead of doing + with strings, use the "concat" method from the buffertools.
Actually, now new Buffer() is deprecated sence node.js v10+, so better to use
Buffer.from(,)
from
response.write(new Buffer(toTransmit));
do
response.write(Buffer.from(toTransmit,'binary'));