Develop Reference

How do images impact the browser memory relative to JS?

I am curious about how media such as images relate to memory in the browser and the JavaScript heap. Explicitly, how many JS objects would be equivalent to a 2MB image?
Disclaimer － I acknowledge that question is too vague for a precise answer. I will provide some arbitrary constraints but ultimately I’m looking for a general way to look at this problem. Suppose I am building a chat app and considering keeping all the messages ever sent in memory, how might I calculate a tipping point for a reasonable number of messages to keep in memory? How do I factor in that some messages are images?
You may assume
The browser is chrome.
The image is in jpg format and loaded using new Image() in JS and then inserted into DOM.
Each object has about three to five key-value pairs. Key-value pairs are strings or numbers ranging from small ints to strings of about 10 ascii chars.
Suppose the 2MB image is visible on screen.
My intuition tells me that the memory cost of an image of 2MB is way less than the cost of JSObjects * (2MB / size_of_obj_in_bytes), because:
The bytes of the image are not in JS land and therefore somehow compressed and optimised away.
The objects will not exist in silo but references will be created throughout user code creating more memory when passed through functions.
There will be lots of garbage collection overhead.
I don't know for certain that I'm right and I certainly don't know by how much or even how to begin measuring this. And since you can't optimize what you can't measure...
Disclaimer 2 - Premature optimization is the root of all evil etc etc. Just curious about digging deeper into the internals.

Better performant alternatives of JSON on mobile devices

I am building a webgl application. And it requires deserialization of data ~15MB (this is size of a single object, I will have around 10 of those in my application) and the bigger portion (90%) of this data is a few arrays of floating point numbers and these arrays need to be deserialized into Float32Arrays in JavaScript.
Currently I am using JSON. Since my data contains lots of repeating numbers it is highly compressible and I am happy with the network performance. Besides. I am also happy with it's performance on desktop. However loading, deserialization of the data into plain JS arrays and then converting them to Float32Arrays on mobile devices takes a lot of time.
I considered using protobuff but I saw this on https://protobuffers.codeplex.com/
Protocol Buffers are not designed to handle large messages. If you are
dealing in messages larger than a megabyte each, it may be time to
consider an alternate strategy.
So what can I do to improve performance of my application. What SERDES methods should I test?
Please walk me through this process and help me test my alternatives, I'll put more details if you ask anything in the comments section.

If your Object is like one big Array of floats, you could send the raw bytes instead of a JSON-encoded string.
XMLHttpRequest has responseType = "arraybuffer". With that your "parsing-step" is reduced to var floats = new Float32Array(xhr.response).
And it would even reduce the impact of this task to the memory, because you don't need to keep a 15MB big String + an intermediate Array containing maybe about 20MB of doubles, i guess + the resulting Float32Array containing another 10MB (half of the doubles) all at about the same time.
You have 1 ArrayBuffer containing only the raw bytes + a Float32Array that references this data in memory.
If this doesn't work for you, maybe you could explain the nature/structure of the data, that you send around.
Or maybe the code you use in the backend, if the serialization is the Problem.

What is the difference between an array and an ArrayBuffer?

I'm just wondering why everyone uses ArrayBuffer instead of just a normal array, string or stringified JSON for sending messages from the server to the client. Is it more efficient?
Also, just wondering what Uint8Array is, how it is different, where to use the two etc.
I am currently using Node.js with Socket.io, but I am happy to change to pure WebSockets if it is a better approach.

An ArrayBuffer is more than just a simple array. It contains raw binary data. This is very useful for direct memory manipulation and conserving space.
When you create a normal array, you won't get a proper set of contiguous memory in many cases since arrays can contain any combination of different kinds of objects. This is useful when you have dynamic data and multiple types together (frequently happens in JS) but is not so useful when you know the exact layout of memory that you need.
This also allows you to view the data at the byte level. For example, it's pretty common in binary data formats to have a n byte long identifier number, an m byte long field telling you how many bytes are used for this field, and m' bytes of data that actually makes up the data field.
[ identifier ][ bytes of data ][ data ]
With an ArrayBuffer, you have the option of moving through that data on the byte level by using various Views. A regular array doesn't allow you to move through the data with that level of granularity because no guarantees are made about the memory layout.
Finally, because you are telling the compiler/interpreter exactly how much space you're going to use and exactly how you're going to view it, it can do much more advanced optimizations when working with that data. When iterating through that data, it doesn't have to make calculated leaps through memory. Instead, it knows exactly how far to move ahead in memory to find the next data point.
As for what Uint8Array is, it's a typed array. Essentially, it tells the compiler/interpreter that you will be accessing this data exclusively as 8-bit uints which, again, allows it to make better optimizations. Then you can use standard array indexing on it (arr[0], arr[1], etc.) and you'll be getting back the equivalent uint values out of the array.
TL;DR They take less space when the exact data format is known, allows you to move more exactly through your data and gives the compiler/interpreter greater options for optimization.

JavaScript - WebAudio API ScriptProcessorNode InputBuffer getChannelData precision

I'm trying to determine the fundamental frequency of an input signal (from a tone generator, or possibly a musical instrument) using JavaScript's WebAudio API, along with some other SO articles (How to get frequency from fft result?, How do I obtain the frequencies of each value in an FFT?), but I can only seem to determine a frequency within about 5-10Hz.
I'm testing using a signal generator app for iPad positioned next to a high-quality microphone in a quiet room.
For 500Hz, it consistently returns 506; for 600Hz, I get 592. 1kHz is 1001Hz, but 2kHz is 1991. I've also noticed that I can modulate the frequency by 5Hz (at 1Hz increments) before seeing any change in data from the FFT. And I've only been able to get this accurate by averaging together the two highest bins.
Does this mean that there's not enough resolution in the FFT data to accurately determine the fundamental frequency within 1Hz, or have I gone about it the wrong way?
I've tried using both the native FFT libs (like this, for example):
var fFrequencyData = new Float32Array(analyser.frequencyBinCount);
analyser.getFloatFrequencyData(fFrequencyData);
(you can assume I've properly initialized and connected an Analyser node), which only shows a sensitivity/resolution of about
and also using DSP.js' FFT lib, like this:
var fft = new FFT();
fft.forward(e.inputBuffer.getChannelData(0));
fFrequencyData = fft.spectrum;
where e is the event object passed to onaudioprocess.
I seem to have a problem with FFT data (like FFT::spectrum) being null - is that normal? The data from fft.spectrum is naught unless I run it through analyser.getFloatFrequencyData, which I'm thinking overwrites the data with stuff from the native FFT, defeating the purpose entirely?
Hopefully someone out there can help steer me in the right direction - thanks! :)

You would need a very large FFT to get high-quality pitch detection this way. To get a 1Hz resolution, for example, with a 44,100kHz sample rate, you would need a 64k(-ish) FFT size. You're far better off using autocorrelation to do monophonic pitch detection.

How to render dynamic bitmap stream to canvas?

I have written an OpenGL game and I want to allow remote playing of the game through a canvas element. Input is easy, but video is hard.
What I am doing right now is launching the game via node.js and in my rendering loop I am sending to stdout a base64 encoded stream of bitmap data representing the current frame. The base64 frame is sent via websocket to the client page, and rendered (painstakingly slowly) pixel by pixel. Obviously this can't stand.
I've been kicking around the idea of trying to generate a video stream and then I can easily render it onto a canvas through a tag (ala http://mrdoob.github.com/three.js/examples/materials_video.html).
The problem I'm having with this idea is I don't know enough about codecs/streaming to determine at a high level if this is actually possible? I'm not sure if even the codec is the part that I need to worry about being able to have the content dynamically changed, and possibly on rendered a few frames ahead.
Other ideas I've had:
Trying to create an HTMLImageElement from the base64 frame
Attempting to optimize compression / redraw regions so that the pixel bandwidth is much lower (seems unrealistic to achieve the kind of performance I'd need to get 20+fps).
Then there's always the option of going flash...but I'd really prefer to avoid it. I'm looking for some opinions on technologies to pursue, ideas?

Try transforming RGB in YCbCr color space and stream pixel values as:
Y1 Y2 Y3 Y4 Y5 .... Cb1 Cb2 Cb3 Cb4 Cb5 .... Cr1 Cr2 Cr3 Cr4 Cr5 ...
There would be many repeating patterns, so any compressing algorithm will compress it better then RGBRGBRBG sequence.
http://en.wikipedia.org/wiki/YCbCr

Why base64 encode the data? I think you can push raw bytes over a WebSocket
If you've got a linear array of RGBA values in the right format you can dump those straight into an ImageData object for subsequent use with a single ctx.putImageData() call.