I want to create Node.js module which provides direct disk access on Windows. It must be able to read, write and search physical drive or virtual device. Because built-in module fs depends on native code written in C++ which uses system function CreateFile which supports \\.\PhysicalDriveX string as an argument to gain access to physical drive, I've tried to do the same in Node.js. My code works fine, application successfully opens disk for read and write access, but there are problems with read and write commands.
When I want to read whole sector, or multiple sectors from disk, it works and displays bytes correctly. But, when I try to read half of a sector or a few bytes, it displays error:
Error: EINVAL: invalid argument, read
at Error (native)
at Object.fs.readSync (fs.js:731:19)
It was not an unsolvable problem to me. I've improved my read function, so it extends buffer to match ceiled requested sector size (i.e. if it needs to read disk from half of the first sector to half of the third sector, it will read whole first, second and third sector and then it will slice the buffer). In order to make it easier to use, I fixed the size of my buffer to 512 bytes, so the only argument of my read and write functions are sector number from which data will be read to buffer or in which data will be written from buffer. Function read works properly and I can get bytes from any sector of the drive.
The real problem is my write function. It can write data only to the first sector. If I try to write data to any other sector expect the first one, I get the following error:
Error: EPERM: operation not permitted, write
at Error (native)
at Object.fs.writeSync (fs.js:786:20)
I tried everything to get rid of this problem, but I coldn't. I tried to change drive, to change buffer size (extend it to cluster size instead of sector size), to use fs.write instead of fs.writeSync, I also tried to search online for solution, but I coldn't find an answer. In order to find out why it doesn't work, I debugged my program using built-in Node.js debugger. I found out that the thread suddenly jumps from fs.readSync to Error function without any reasonable explanation and the process terminates.
How can I properly use fs.writeSync function to write any sector to a physical drive? Am I doing something wrong, or there is a problem with Node.js?
We released a native add on for Node.js yesterday with helpers and documentation for doing direct IO on Node.js (cross-platform):
https://github.com/ronomon/direct-io
I think in your case you might need to look into special write restrictions on Windows (and the need for FSCTL_LOCK_VOLUME so that you can write to all sectors and not just the master boot record).
You also need aligned buffers.
The module will help you with both of these requirements.
Related
This is a topic where I can't seem to find the answer on the Node.js docs (I know it's possible because of libraries like exif), nor can I find an answer on the internet without everyone saying to just use a library.
I don't want to use a library, so I want to do this natively and learn more about reading file metadata, and maybe eventually updating the metadata too while building my own mini-tool.
If I run something like fs.statSync() I can get generic metadata that returns in the Stats object; but, in my case, I'm looking for all the other metadata, NOT just the basic file info like size, birthtime, etc.
I want the other metadata like dimensions, date taken, and especially things you'd see in image, video, or audio files.
Maybe there's something like:
const deepMetaData = fs.readFileSync().getMetaDataAsString();
console.info(/Date Taken/.test(deepMetaData)); // true
or
const deepMetaData = fs.createReadStream().buffer().toString();
const dateTaken = deepMetaData.match(/Date Taken: (\d{4}-\d{2}-\d{2})/)[1];
console.info(dateTaken);
If I need to work with buffers, streams, whatever, instead of a string output, that's cool too. Ideally something synchronous. So if there's a simple example someone could provide of how to read that kind of meta data without a library, I'll at least be able to look up the methods used from that to understand more later and leverage the docs associated with whatever approach. Thank you!
Nodejs fs functions like fs.statSync() provide OS level metadata on the file only (such as createDate, modificationDate, file size, etc...). These are properties of the file in the file system. These do NOT have anything at all to do with the actual data of the file itself.
When you talk about EXIF (for a photo), this is parsed from the file data itself. To know about that type of data, you must read and parse at least the beginning of the file and you must be able to recognize and understand all the different file formats that you might encounter. For photos, this would include JPEG, PNG, HEIC, GIF, etc... Each of those have different file formats and will require unique code for understanding the metadata embedded in the file.
Nodejs does not have support for any of that built-in.
So, it will take custom code for each file type. If you further want to include other types of files like videos, you need to extend your list of different file types you can read, parse and understand. For the depth of files you're talking about this is a big job, particular when it comes to testing against all the different variants of files and metadata that exist out in the wild.
I personally would be fine with implementing my own code for one particular file type like JPEG, but if I was tasked with supporting dozens of types of files and particularly if tasked with supporting the wide range of video file formats, I'd immediately seek out help from existing libraries that have already done all the time consuming work to research, write and test how to properly read and understand all the variants.
I know it's possible because of libraries like exif
This is an example of a library that reads the beginning of the image file, parses it according to the expected format and knows how to interpret all the possible tags that can be in the EXIF header and what they all mean.
So if there's a simple example someone could provide of how to read that kind of meta data without a library
Go study the code for the EXIF library and see how it works. If you're going to implement it yourself, that's how you have to do it. I'm still not sure why you'd avoid using working libraries if they already exist. That is one of the biggest advantages of the nodejs ecosystem - you can build on all the open source code that already exists without reimplementing it all from scratch yourself and spend your coding time on parts of your problem that someone else has not already implemented.
how would one read that metadata using node?
You literally have to read the data from the file (usually at the start of the file). You can use any of the mechanisms that the fs module provides. For example, you can use fs.createReadStream() and then stream in the file, parsing and interpreting it as data arrives and then stop the stream when you get past the end of the metadata. Of, you can open a file handle using fs.open() and use fs.read() to read chunks of the file until you have read enough to have all the metadata.
You HAVE an example sitting right in front of you of code that does this in the EXIF library on NPM that you already seem to know about. Just go examine its code. The code is ALL there.
I'm just looking for a simple answer on getting that info, even if it's a blob of strings.
This is perhaps your main problem. There is no simple answer to get that info and it doesn't just exist as a blob of strings. These files are sometimes binary files (for space efficiency reasons). You have to learn how to read and parse binary data. Go study the code in the EXIF library and see what it is already doing and you can learn from that. There is no better example to start with.
But, for a simple example using the heic filetype, this will grab the first 5000 characters of the file's metadata, which can then be searched:
const fileDescriptor = fs.openSync(absPathToHeicPhoto);
const charCount = 5000;
const buffer = Buffer.alloc(charCount);
const headerBytes = fs.readSync(fileDescriptor, buffer, 0, charCount);
const bufferAsStr = buffer.toString('utf8', 0, charCount);
console.info(/\d{4}:\d{2}:\d{2}/.test(bufferAsStr));
FYI, I looked at the code for this EXIF library on NPM and it's poorly implemented. It uses fs.readFile() to load the ENTIRE image into RAM (even though it only needs a fraction of the data at the start of the file). This is a poor implementation for this reason (memory and disk inefficient).
But, it does have a method called processImage and one called extractExifData that process the binary data of the file to parse out the EXIF info. These are links to its actual code. You can start learning there.
FYI, as a photographer, I use a command line program called exiftool that will dump exif info to stdout or to a file for many images. As a different approach, you could just run that tool from your nodejs program (using the child_process module and capture its output and use that output, letting it do the hard work you just operate on the generated output.
I need to provide text information in a file, read it in java-script and then work with the output. The output is not supposed to go to a web page, but is a plugin for another program (RPG Maker MV).
I'm trying to load a text-file (*.txt) and output it with document.write() I'm currently checking the code in Firefox. I get why this is prevented from working, but ultimately I do not want to run it in a Browser anyway. However I need to try the code before implementing it.
My text file looks like this:
What is the Capital of France?
-Berlin
-Paris [X]
-Koppenhagen
Who came up with the Theory of Evolution?
-Charles Darwin [X]
-Thomas Eddison
-Nicolas Flamel
The java-script should do something like:
var mytext=fs.readFileSync('quiz.txt');
document.write(mytext);
I could also use other file types than *.txt of course. Important is, that the quiz-file can be adapted without messing with the original java-script file.
Unfortunately, you can't use the same code to read in file in NodeJS and from code running in a browser as they use very different APIs.
NodeJS relies on the fs module while the browser environment relies on the File APIs specified by W3C.
I am writing a Media-HUD that runs totally on local notecard files stored within the prim's inventory, with notecards named like index.html, style.css, icon.svg etc.
My hope is to use the LSL HttpServer functions, and the script's URL to create a totally self-contained media based HUD that is easy to edit like editing any web page.
This is completely possible on its own, however there is a limitation in that, the pages must fit into the memory allocated to the LSL script. Under mono this is only 64kb.
I want to remove this limitation, by somehow, perhaps from javascript, reading in each 'file' from a notecard line by line in the users browser itself (thusly, getting around the memory limit by only bringing one notecard line into memory at a time).
Is there a way to do this? generate a entire file in javascript procedurally by loading in the strings making it up line by line, and then serve it as though it were a whole file? I'm not sure how feasible this is.
Any idea's/guidance greatly appreciated!
You could do this through Javascript using XMLHttpRequest. jQuery's wrapper for this is called Ajax. You could request each line individually, which would be slightly slower, or read in a number of lines at a time, at the script's leisure. http_request is not throttled so either works. Note that the loader has to be sent in a single response, because the LSL server has no way of pushing data "piecemeal" like an actual server does.
Notes:
llGetNotecardLine only returns the first 255 bytes per line.
llHTTPResponse must be called within ~20 seconds of the request, so you can't feasibly read more than 20 lines from a notecard at a time.
I'm not sure how this would work for non-DOM filetypes. All files would need to be embed-able in the HTML using the Javascript DOM. To my knowledge, Javascript can't arbitrarily create an external file and serve it to itself. Obviously it would not work for non-text filetypes, but you could certainly load in the rest of the HTML, CSS, and HTML5 SVG. Basically, if it can go in a single HTML file, you could load it in through Javascript.
I have no experience with React but it gives a good example of what is possible on the UI side with loading things in purely through Javascript.
So less than 64 thousand characters in memory at most per script. I will give you some advise that might make your task feasable:
External resources
Minimize the amount of code you have to have in your notecards by sourcing popular libraries from the web Bootstrap, React and such.
You will have to rely on their mirror's availability thought. But it will greatly reduce the amount of memory needed to provide pretty and functional pages.
Minify your code
Use tools like Uglify or Closure Compiler to make your javascript lighter. Though you must be careful, as these tools will fit all your code a single long line by the default, and you can't read lines longer than 255 characters with LSL. Luckily you can customize your options in these tools to limit the amount of characters per line.
Divide and conquer
Since a single script can't handle much memory, make dedicated scripts. One could serve resources (act as a file server, providing your html and js) while the other would receive API request calls, to handle the application's logic.
I am new to wala. Trying to configure it by following steps given at wala: gettingstarted.
However, I have to face errors on each example I am trying to run. Please look at following errors:
1- Example SWTTypeHierarchy
PANIC: roots.size()=0
2- Example PDFTypeHierarchy
spawning process [C:/Program Files (x86)/Graphviz2.38/bin/dot.exe,
-Tpdf, -o, D:\WALAOut\th.pdf, -v, D:\WALAOut\temp.dt] process terminated with exit code 0 read 919 bytes from error stream
3- Js frontend com.ibm.wala.cast.js.rhino.test-JUnit
I am not able to use launcher instead when I open it following error is there
" [JRE]: Unknown JRE type specified:
org.eclipse.jdt.internal.launching.macosx.MacOSXType"
I am not sure which step went wrong which have completely damage the execution. Please help
It is hard to help you with these generic problems. I will give it a try based on the limited information these error messages contain.
1) have you configured your wala.properties file?
2) you are missing dot for this visualization. Hence the error. You may also want to use http://pages.cs.wisc.edu/~ghost/gsview/ to view your graphs.
3) seems like a problem with the location of the jre, how did you configure your properties file?
The WALA examples are great to learn from, but sometimes getting one of them to run is more time consuming, than using the slicer example in their wiki page:
http://wala.sourceforge.net/wiki/index.php/UserGuide:Slicer
This should get you started more easily.
I've built a custom logging utility which displays a Log Message and the DateTime. I would like to add the line number in the source code which called the function.
Is there a way to determine which line of the HTML source a particular javascript function was fired?
Having written a logging library (log4javascript) myself, I've considered this same problem and here are my thoughts:
The problem is that in order to get the information you want, you need an Error object that was created on the line in question. Creating an Error within your logging utility will only directly give you the filename and line number for the particular line in your logging utility code rather than for the line of code that made the logging call. The only way round this I can think of is parsing the stack property of the Error (or the message property in Opera), which has several problems:
the stack trace is only available in Mozilla, recent WebKit and Opera browsers
the stack trace is a string that varies from browser to browser, and may change format again without notice in future browsers, thus breaking the parsing code
throwing an Error and parsing its stack trace for every log call will add a significant performance overhead.
For the purposes of log4javascript, I decided it wasn't worth implementing, but for your own needs you may decide it's worthwhile.