Develop Reference

Why is serialisation necessary for sending data?

I've been reading up on JSON and serialization, from what I understand JSON is a format often used for transferring data over a network e.g. from/to a web server or storing the data to disk.
The data could be strings, numbers, objects etc. I haven't found a clear explanation for why the serialization is needed, for instance when sending a string to a web server or saving it to disk, isn't the string already stored as a series of bits and bytes by the computer, isn't this the most basic form for the data? so why can't these be sent/stored as they are?
Why does it need to be stringified into JSON i.e. serialised, which turns it into a string?
To be clear, I'm asking why it's needed and a simple clear explanation for that.
Thanks

Broadly speaking serialization does two important, mostly independent jobs:
collects all the information into a single "chunk" (stream) of data that's self-contained and
turns all the information into an agreed-on format (usually optimized for either compactness or ease of parsing)
#1 is important because a single object with many properties and sub-object can be spread all over the memory of a running program.
For example a JavaScript runtime could have a dedicated memory pool for strings constants. Then an object that uses some constant as a key would just reference into that pool from its data structure. That means that the object is no longer in a single self-contained block in memory: it's spread out over multiple areas. This kind of spreading-out is actually the norm: objects don't usually contain complex data directly and depending on the language even "primitive" values such as number could be stored as references to another place in memory.
#2 is important mostly because the format used to quickly access data in-memory might not be suitable to transfer (because it might contain unnecessary redundancy or memory pointers that don't make any sense when transferred to another computer, which partially ties to reason #1).
An example of that would be a map (or dictionary): the in-memory representation will usually involve multiple buckets that hold hashed-values and some kind of collision-handling structure inside those buckets (a linked list or a tree, for example). That structure helps with efficient access to separate keys, but transferring that structure directly over the wire is pointless: it's very easy to re-build and there's no guarantee that the receiving end uses the exact same way to represent a map. So instead we just send each key and the associated value and let the receiving end deal with re-constructing any data structures it needs for efficient access.

The simple reason is that data can be stored differently in memory on different computers, or even by programs on the same computer written in different programming languages.
Serialization formats like JSON provide a defined way for exchanging data between computers or programs.

Not everyone knows how to parse or interpret those series of bits. Sometimes you need some general structure, some format, that can be passed around so that other people understand what it is you're trying to tell them.

Use web worker to stringify

So I have an app that needs to JSON.stringify its data to put into localStorage, but as the data gets larger, this operation gets outrageously expensive.
So, I tried moving this onto a webWorker so it's off the main thread, but I'm now learning posting an object to a webWorker is even more expensive than stringifying it.
So I guess I'm asking, is there any way whatsoever to get JSON.stringify off the main thread, or at least make it less expensive?
I'm familiar with fast-json-stringify, but I don't think I can feasibly provide a complete schema every time...

You have correctly observed that passing object to web worker costs as much as serializing it. This is because web workers also need to receive serialized data, not native JS objects, because the instance objects are bound to the JS thread they were created in.
The generic solution is applicable to many programming problems: chose the right data structures when working with large datasets. When data gets larger it's better sacrifice simplicity of access for performance. Thus do any of:
Store data in indexedDB
If your large object contains lists of the same kind of entry, use indexed DB for reading and writing and you don't need to worry about serialization at all. This will require refactor of your code, but this is the correct solution for large datasets.
Store data in ArrayBuffer
If your data is mostly fixed-size values, use an ArrayBuffer. ArrayBuffer can be copied or moved to web worker pretty much instantly and if your entries are all same size, serialization can be done in parallel. For access, you may write simple wrappers classes that will translate your binary data into something more readable.

Nodejs Couchbase deserialize date property from document

I'm saving in Couchbase a document which has javascript Date values, and wish to get it exactly the same, not as string '2016-01-02T12:13:14Z'.
Found a way to achieve this using plain Javascript, by using the second parameter of JSON.parse , but Couchbase does the deserialization internally and can't really use this.
Is there any way to disable the Couchbase deserialization, and to avoid doing JSON.stringify + JSON.parse and neither deep-walking the object?
bucket.get(key, (err, result) => {
if (err) {
//deal with error here
} else {
//here "result.value" is already deserialized
done(result.value);
}
});

As you probably know, JSON and Date object handling can be a bit specific depending on what you're trying to do. We tend to stick to the defaults. What you're looking for is a fairly advanced use case. At the moment, we don't have direct support for changing the way we do that parsing.
However, there is an interface for this. It's called a "transcoder" and it lets you be very specific about how you want to handle converting incoming data to what is stored. I don't have an example that shows quite what you want to do, but a good place to start looking for this at a lower layer is shown in the tests.
It might be easier, however, to just treat whatever you're storing differently at the application level. From my read of the revivier parameter you pointed to, that'd be only at time of retrieval. There's nothing stopping you from wrapping that get() and then mutating the object before passing it along to the next layer at very low cost I do believe.

Overwrite variable content javascript

If you have a variable set with sensitive data:
var secretPassword = 'myPa$sW0rd';
and you overwrite it:
secretPassword = '0000000000';
Does a javascript engine allocate new memory for the new data? is the data myPa$sW0rd potentially somewhere in unallocated memory still?
My main question is this: is there a way to guarantee that you overwrite the data? (ie zero it out or securely delete the data). I wonder if looping through the characters in the string and resetting them that way would do it.
I assume it's not likely you would be able to access the data without some bug in the javascript engine.

Does a javascript engine allocate new memory for the new data?
yes
is the data myPa$sW0rd potentially somewhere in unallocated memory still?
yes if no garbage collection cleaned it up
My main question is this: is there a way to guarantee that you overwrite the data? (ie zero it out or securely delete the data).
Not really, except if you play with the garbage collector. See related post
I wonder if looping through the characters in the string and resetting them that way would do it.
No
Edit : As pointed out by doldt, there is no real security threat even if the previous data is still somewhere in memory.

Try to use an array of fixed size and assign 0 values to all elements on clean-up. This does not reallocate a new object.
However, it will be difficult to control where the value is passed around once you use it as a string.

Custom JSON.stringify fails to Stringify object as whole, but works when iterated one level deep

Hoping someone can spot the error, because I'm having trouble
Alright, I built my own JSON.stringify for just custom large objects. It may not be exactly to specification for some edge case things, but's only meant for stringify on large objects that I'm building myself.
Well, it works, and works well for most objects, but I have an Object I'm trying to stringify and it's failing and printing this before exiting:
node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
undefined
Not very helpful. The object is fine because the regular call to JSON.stringify(object) works fine, and when I iterate over the object with for (var x in obj) if (obj.hasOwnProperty(x)) { myStringify(obj); } that works fine, but if I call it on the top level of the object, it goes to hell... It doesn't really make sense to me, and the only thing I can think of is the level if recursion is somehow breaking something...
The Parser : https://gist.github.com/958776 - The stringify function I'm calling
ObjectIterator.js : https://gist.github.com/958777 - Mostly to provide the asynchronous iteration
Edit So, I iterated over the object one level deep and compared the resulting string to the string of JSON.stringify(sameLevelDeep) and they're equal. Since the output is equal, I'm not sure that it's how I'm parsing something, but possible that it's such a large object or the amount of recursion is so high?
Edit 2 So, I "fixed" the problem, I guess. Instead of every 25th iteration being pushed to the next event loop, I push every fifth. I'm not sure why this would make a difference but it does... I guess the question is now "Why does that make a difference"?

Okay well, beyond it being a very specific question helping a very specific person, I would like to take this to a different place, that might also remove your problem and maybe help others.
Since you are not specifying why you are going through this process, I will have to break it down and guess -- and provide a solution for each guessed idea.
1. (Browser) You are trying to use JavaScript to crunch data, and provide the user with a result
Downloading at least several megabytes of raw data ("some of these objects are 5-10million characters") on a webpage to process and display a result is far from optimal, you should probably be doing this operation on the server side and download the pre-calculated result.
Besides, no matter what you are doing, JavaScript does not support threads.
setTimeout(1, function() { JSON.stringify(data); }); shouldn't be much different from what you are doing.
2. (Browser) You are trying to display the downloaded content
You should attempt downloading smaller chunks instead of the whole 10+ million character content using the built-in JSON.stringify method.
3. (Non-browser) You are trying to use JavaScript for an application that requires threading
You should consider using a different programming language for this application.
In summary
I think you are climbing the wrong mountain, you can achieve the same thing walking around it without breaking sweat. If you want to climb a mountain for kicks, there are mountains out there that need it -- but it's not this one.
Translation: Work on the architecture to obsolete the obstacle instead of trying to solve it, if you want to solve a problem there are problems that need a solving -- but it's not this one.