Timestamps for custom output-only AudioWorkletProcessor - javascript

I'd like to build a very low-level audio output node using the web audio API's AudioWorkletProcessor. It seems like all of the examples out there implement processors that transform input samples to output samples, but what I'd like to do is produce output only, all based on the timestamp of the samples.
According to MDN, BaseAudioContext.currentTime is not a precise source for this timestamp:
To offer protection against timing attacks and fingerprinting, the precision of audioCtx.currentTime might get rounded depending on browser settings.
In Firefox, the privacy.reduceTimerPrecision preference is enabled by default and defaults to 20us in Firefox 59; in 60 it will be 2ms.
One hacky solution may be to use BaseAudioContext.sampleRate, a running counter, and the size of the output arrays, but that only works if we can assume that every sample is computed without drops and that everything's computed in order, and I'm not sure if those are valid assumptions.
Within a processing frame, is there a reliable way to know the timestamp which correlates to given sample index?

Inside of the AudioWorkletGlobalScope you have access to the currentTime as well as to the currentFrame. They are both available as globals.
https://webaudio.github.io/web-audio-api/#AudioWorkletGlobalScope-attributes
As far as I know they are accurate in any browser which supports the AudioWorklet.

Related

Can Javascript's Date().getTime() be incorrect if the system clock is wrong?

Occasionally we get a report from users that a countdown timer in our application is set too high, or is displaying negative numbers. I figured it was a timezone issue but I've been unable to replicate it by changing my clock settings - even if I change the ones in my computer's BIOS. The only way I was able to replicate it was by actually adding or subtracting 3600 seconds from the timestamp in my code.
I read that Date().getTime() always provides a UTC timestamp. I doubt it's sourced from the internet, so it makes sense to me that it could be wrong if the computer is providing an incorrect time. Unfortunately I can't make this actually happen yet.
I also can't find much information online about this, or gather any data from our users (This is very infrequent and they unsurprisingly become unwilling to help once I mention timezones and system clocks).
So, can Javascript provide me with an incorrect UTC timestamp? If so, is there a strategy to ensure the timestamp is accurate?
Edit -
It struck me that the simplest solution is probably just to send over the server time and use that, since it doesn't need to be especially accurate.
I'd say if you want to dig into implementation - go and check Mozilla code. It's open source. I dig into it because of your post and I finally reach to the file: https://hg.mozilla.org/mozilla-central/file/tip/js/src/vm/Time.cpp
It has some interesting function called NowCalibrate(). Seems that it's doing the trick with fixing time if the system time is off or something.
You can check lines 79 to 140. I will attach portion of the code that initializes the time.
void
PRMJ_NowInit()
{
memset(&calibration, 0, sizeof(calibration));
// According to the documentation, QueryPerformanceFrequency will never
// return false or return a non-zero frequency on systems that run
// Windows XP or later. Also, the frequency is fixed so we only have to
// query it once.
LARGE_INTEGER liFreq;
DebugOnly<BOOL> res = QueryPerformanceFrequency(&liFreq);
MOZ_ASSERT(res);
calibration.freq = double(liFreq.QuadPart);
MOZ_ASSERT(calibration.freq > 0.0);
InitializeCriticalSectionAndSpinCount(&calibration.data_lock, DataLockSpinCount);
// Windows 8 has a new API function we can use.
if (HMODULE h = GetModuleHandle("kernel32.dll")) {
pGetSystemTimePreciseAsFileTime =
(void (WINAPI*)(LPFILETIME))GetProcAddress(h, "GetSystemTimePreciseAsFileTime");
}
}
On the other side I found that every other browser has it's own implementation, but Mozilla can be a real world example of how the initial time can be calculated and calibrated after certain period.

Understanding how midi file is mapped to object in javascript

I'm using this library I found to read a midi file
As there is very little documentation, I have no idea how to read the output object?
question: What does: Channel, data, deltaTime and type keys mean?
In the end I would love to map this js object to some kind of visualization.
Channel: The MIDI format uses the concept of channels to allow different MIDI devices to only listen to specific MIDI events by listening to such a channel. This makes it possible to use a single MIDI file for multiple instruments that should play different notes, etc. So when you have a note on event you should check the channel of the event and only play the instruments that are interested in events that happen in this channel.
Data: Data is a bit arbitrary, but in your example we have an event of type 255 (0xFF) which is a meta event. It has a meta type of 3 (0x03) which means it's a Sequence/Track-name. This was probably assigned by the program that created the MIDI file you use. There's a pretty nifty and concise list of events here: http://www.ccarh.org/courses/253/handout/smf/
deltaTime: Since events in the MIDI file is tempo agnostic it uses the concept of ticks. It's basically a resolution expressed as ticks per quarter note. I think 480 ticks per quarter note is pretty standard, though that is purely based on my own experience, so YMMV. Events can then either be expressed in absolute time (ie. this note on events happens 4800 ticks from the start of the track) or delta time. Delta time is the number of ticks since the last MIDI event happened.
Type: Each MIDI event in a MIDI file has a type to identify what kind of an event it is. This matters since different types of events has different formats (and thus changes the way we decode it, since MIDI is a binary format), where some have a fixed length and others include information on how long the event is (the number of bytes that make up the event).
It's been a couple of years since I last worked with the MIDI format, but I think the above is accurate.

How can dates and random numbers be used for evil in Javascript?

The ADsafe subset of Javascript prohibits the use of certain things that are not safe for guest code to have access to, such as eval, window, this, with, and so on.
For some reason, it also prohibits the Date object and Math.random:
Date and Math.random
Access to these sources of non-determinism is restricted in order to make it easier to determine how widgets behave.
I still don't understand how using Date or Math.random will accomodate malevolence.
Can you come up with a code example where using either Date or Math.random is necessary to do something evil?
According to a slideshow posted by Douglas Crockford:
ADsafe does not allow access to Date or random
This is to allow human evaluation of ad content with confidence that
behavior will not change in the future. This is for ad quality and
contractual compliance, not for security.
I don't think anyone would consider them evil per se. However the crucial part of that quote is:
easier to determine how widgets behave
Obviously Math.random() introduces indeterminism so you can never be sure how the code would behave upon each run.
What is not obvious is that Date brings similar indeterminism. If your code is somehow dependant on current date it will (again obviously) work differently in some conditions.
I guess it's not surprising that these two methods/objects are non-functional, in other words each run may return different result irrespective to arguments.
In general there are some ways to fight with this indeterminism. Storing initial random seed to reproduce the exact same series of random numbers (not possible in JavaScript) and supplying client code with sort of TimeProvider abstraction rather than letting it create Dates everywhere.
According to their website, they don't include Date or Math.random to make it easier to determine how third party code will behave. The problem here is Math.random (using Date you can make a psuedo-random number as well)- they want to know how third party code will behave and can't know that if the third party code is allowed access to random numbers.
By themselves, Date and Math.random shouldn't pose security threats.
At a minimum they allow you to write loops that can not be shown to be non-terminating, but may run for a very long time.
The quote you exhibit seem to suggest that a certain amount of static analysis is being done (or is at least contemplated), and these features make it much harder. Mind you these restrictions aren't enough to actually prevent you from writing difficult-to-statically-analyze code.
I agree with you that it's a strange limitation.
The justification that using date or random would make difficult to predict widget behavior is of course nonsense. For example implement a simple counter, compute the sha-1 of the current number and then act depending on the result. I don't think it's any easier to predict what the widget will do in the long term compared to a random or date... short of running it forever.
The history of math has shown that trying to classify functions on how they compute their value is a path that leads nowhere... the only sensible solution is classifying them depending on the actual results (black box approach).

Is Math.random() cryptographically secure?

How good are algorithms used in Javascript Math.random() in different browsers? Is it okay to use it for generating salts and one-time passwords?
How many bits from one random I can use?
Nope; JavaScript's Math.random() function is not a cryptographically-secure random number generator. You are better off using the JavaScript Crypto Library's Fortuna implementation which is a strong pseudo-random number generator (have a look at src/js/Clipperz/Crypto/PRNG.js), or the Web Crypto API for getRandomValues
Here is a detailed explanation: How trustworthy is javascript's random implementation in various browsers?
Here is how to generate a good crypto grade random number: Secure random numbers in javascript?
It is not secure at all, and in some cases was so predictable you could rebuild internal state of the PRNG, deduct the seed and thus could use it to track people across websites even if they didn't use cookies, hid behind onion routing etc...
2022 edit since this answer still gets upvotes: use Crypto.getRandomValues if you need a cryptographic RNG in JavaScript
http://landing2.trusteer.com/sites/default/files/Temporary_User_Tracking_in_Major_Browsers.pdf a 2008 paper exposing the user tracking possibilities of the browser weak PRNG
http://dl.packetstormsecurity.net/papers/general/Google_Chrome_3.0_Beta_Math.random_vulnerability.pdf a later (2009) Chrome vulnerability, as the problem was already well known
As of March 2013, window.crypto.getRandomValues is an "experimental technology" available since Chrome 11 and Firefox 21 that lets you get cryptographically random values. Also, see getRandomValues from the lastest W3C Web Cryptography API draft.
Description:
If you provide an integer-based TypedArray (i.e. Int8Array,
Uint8Array, Int16Array, Uint16Array, Int32Array, or Uint32Array), the
function is going fill the array with cryptographically random
numbers. The browser is supposed to be using a strong (pseudo) random number generator. The method throws the QuotaExceededError if the requested length is greater than 65536 bytes.
Example:
var array = new Uint32Array(10);
window.crypto.getRandomValues(array);
console.log("Your lucky numbers:");
for (var i = 0; i < array.length; i++) {
console.log(array[i]);
}
Also, an answer to How random is JavaScript's Math.random? refers to Temporary user tracking in major browsers and Cross-domain information leakage and attacks from 2008 which discusses how the JavaScript Math.random() function leaks information.
Update: For current browser support status, check out the Modern.IE Web Crypto API section, which also links to the Chrome, Firefox, and Safari bug reports.
Because you cannot know the exact implementation of the browser (except for closed user groups like for your business intranet) I would generally consider the RNG weak.
Even if you can identify the browser you don't know if the browser itself or any other browser's agent ID is manipulated. If you can you should generate the number on the server.
Even if you include a good PRNG in your JavaScript your server cannot know whether the request from the client originates from an unmodified script. If the number goes into your database and/or is used as a cryptographic tool it is no good idea to trust the data from the client at all. That is true not only for validity (You do validate all data coming from the client, don't you?) but also for general properties like randomness.
Math.random() is not cryptographically secure. Also Veracode will point this occurrence with
CWE-331 (Insufficient Entropy)
We could make use of SecureRandom to implement similar functionality.
new SecureRandom().nextDouble();

Is it possible to find stretches of silence in audio files with Javascript?

I've been working on a tool to transcribe recordings of speech with Javascript. Basically I'm hooking up key events to play, pause, and loop a file read in with the audio tag.
There are a number of advanced existing desktop apps for doing this sort of thing (such as Transcriber -- here's a screenshot). Most transcription tools have a built-in waveform that can be used to jump around the audio file, which is very helpful because the transcriber can learn to visually find and repeat or loop phrases.
I'm wondering if it's possible to emulate a subset of this functionality in the browser, with Javascript. I don't know much about signal processing, perhaps it's not even feasible.
But what I envision is Javascript reading the sound stream from the file, and periodically sampling the amplitude. If the amplitude is very low for longer than a certain threshhold of time, then that would be labled as a phrase break.
Such labeling, I think, would be very useful for transcription. I could then set up key commands to jump to the previous period of silence. So hypothetically (imagining a jQuery-based API):
var audio = $('audio#someid');
var silences = silenceFindingVoodoo(audio);
silences will then contain a list of times, so I can hook up some way to let the user jump around through the various silences, and then set the currentTime to a chosen value, and play it.
Is it even conceivable to do this sort of thing with Javascript?
Yes it's possible with Web Audio API, to be more precise you will need AnalyserNode. To give you a short proof of concept you can get this example, and add following code to drawTimeDomain():
var threshold = 1000;
var sum = 0;
for (var i in amplitudeArray) {
sum += Math.abs(128 - amplitudeArray[i]);
}
var test = (sum < threshold) ? 'silent' : 'sound';
console.log('silent info', test);
You will just need a additional logic to filter silent by milliseconds (e.g. any silent taking more than 500 ms should be seen as real silent )
I think this is possible using javascript (although maybe not advisable, of course). This article:
https://developer.mozilla.org/En/Using_XMLHttpRequest#Handling_binary_data
... discusses how to access files as binary data, and once you have the audio file as binary data you could do whatever you like with it (I guess, anyway - I'm not real strong with javascript). With audio files in WAV format, this would be a trivial exercise, since the data is already organized by samples in the time domain. With audio files in a compressed format (like MP3), transforming the compressed data back into time-domain samples would be so insanely difficult to do in javascript that I would found a religion around you if you managed to do it successfully.
Update: after reading your question again, I realized that it might actually be possible to do what you're discussing in javascript, even if the files are in MP3 format and not WAV format. As I understand your question, you're actually just looking to locate points of silence within the audio stream, as opposed to actually stripping out the silent stretches.
To locate the silent stretches, you wouldn't necessarily need to convert the frequency-domain data of an MP3 file back into the time-domain of a WAV file. In fact, identifying quiet stretches in audio can actually be done more reliably in the frequency domain than in the time domain. Quiet stretches tend to have a distinctively flat frequency response graph, whereas in the time domain the peak amplitudes of audible speech are sometimes not much higher than the peaks of background noise, especially if auto-leveling is occurring.
Analyzing an MP3 file in javascript would be significantly easier if the file were CBR (constant bit rate) instead of VBR (variable bit rate).
As far as I know, JavaScript is not powerful enough to do this.
You'll have to resort to flash, or some sort of server side processing to do this.
With the HTML5 audio/video tags, you might be able to trick the page into doing something like this. You could (hypothetically) identify silences server-side and send the timestamps of those silences to the client as meta data in the page (hidden fields or something) and then use that to allow JavaScript to identify those spots in the audio file.
If you use WebWorker threads you may be able to do this in Javascript, but that would require using more threads in the browser to do this. You could break up the problem into multiple threads and process it, but, it would be all but impossible to synchronize this with the playback. So, Javascript can determine the silent periods, by doing some audio processing, but since you can't link that to the playback well it would not be the best choice.
But, if you wanted to show the waveforms to the user then javascript and canvas can be used for this, but then see the next paragraph for the streaming.
Your best bet would be to have the server stream the audio and it can do the processing and find all the silences. Each of these should then be saved in a separate file, so that you can easily jump between the silences, and by streaming, your server app can determine when to load up the new file, so there isn't a break.
I don't think JavaScript is the tool you want to use to use for processing those audio files - that's asking for trouble. However, javascript could easily read a corresponding XML file which describes where those silences occur in the audio file, adjusting the user interface appropriately. Then, the question is what do you use to generate those XML files:
You can do it manually if you need to demo the capability right away. (Use audacity to see where those audio envelopes occur)
Check out this CodeProject article, which creates a wav processing library in C#. The author has created a function to extract silence from the input file. Probably a good place to start hacking.
Just two of my initial thoughts ... There are ALOT of audio processing APIs out there, but they are written for particular frameworks and application programming languages. Definitely make use of them before trying to write something from scratch ... unless you happen to really love fourier transforms.

Categories

Resources