I'm using the standard Fisher-Yates algorithm to randomly shuffle a deck of cards in an array. However, I'm unsure if this will actually produce a true distribution of all possible permutations of a real-world shuffled deck of cards.
V8's Math.random only has 128-bits of internal state. Since there are 52 cards in a deck, 52 factorial would require 226-bits of internal state to generate all possible permutations.
However, I'm unsure if this applies when using Fisher-Yates since you aren't actually generating each possible but just getting one position randomly out of 52.
function shuffle(array) {
var m = array.length, t, i;
while (m) {
i = Math.floor(Math.random() * m--);
t = array[m];
array[m] = array[i];
array[i] = t;
}
return array;
}
In general, if a pseudorandom number generator admits fewer than 52 factorial different seeds, then there are some permutations that particular PRNG can't choose when it shuffles a 52-item list, and Fisher-Yates can't change that. (The set of permutations a particular PRNG can choose can be different from the set of permutations another PRNG can choose, even if both PRNGs are initialized with the same seed.) See also this question.
Note that although the Math.random algorithm used by V8 admits any of about 2^128 seeds at the time of this writing, no particular random number algorithm is mandated by the ECMAScript specification of Math.random, which states only that that method uses an "implementation-dependent algorithm or strategy" to generate random numbers (see ECMAScript sec. 20.2.2.27).
A PRNG's period can be extended with the Bays-Durham shuffle, which effectively increases that PRNG's state length (see Severin Pappadeux's answer). However, if you merely initialize the Bays-Durham table entries with outputs of the PRNG (rather than use the seed to initialize those entries), it will still be the case that that particular PRNG (which includes the way in which it initializes those entries and selects those table entries based on the random numbers it generates) can't choose more permutations than the number of possible seeds to initialize its original state, because there would be only one way to initialize the Bays-Durham entries for a given seed — unless, of course, the PRNG actually shuffles an exorbitant amount of lists, so many that it generates more random numbers without cycling than it otherwise would without the Bays-Durham shuffle.
For example, if the PRNG is 128 bits long, there are only 2^128 possible seeds, so there are only 2^128 ways to initialize the Bays-Durham shuffle, one for each seed, unless a seed longer than 128 bits extends to the Bays-Durham table entries and not just the PRNG's original state. (This is not to imply that the set of permutations that PRNG can choose is always the same no matter how it selects table entries in the Bays-Durham shuffle.)
EDIT (Aug. 7): Clarifications.
EDIT (Jan. 7, 2020): Edited.
You are right. With 128 bits of starting state, you can only generate at most 2128 different permutations. It doesn't matter how often you use this state (call Math.random()), the PRNG is deterministic after all.
Where the number of calls to Math.random() actually matter is when
each call would draw some more entropy (e.g. from hardware random) into the system, instead of relying on the internal state that is initialised only once
the entropy of a single call result is so low that you don't use the entire internal state over the run of the algorithm
Well, you definitely need RNG with 226bits period for all permutation to be covered, #PeterO answer is correct in this regard. But you could extend period using Bays-Durham shuffle, paying by effectively extending state of RNG. There is an estimate of the period of the B-D shuffled RNG and it is
P = sqrt(Pi * N! / (2*O))
where Pi=3.1415..., N is B-D table size, O is period of the original generator. If you take log2 of the whole expression, and use Stirling formula for factorial, and assume P=2226 and O=2128, you could get estimate for N, size of the table in B-D algorithm. From back-of-the envelope calculation N=64 would be enough to get all your permutations.
UPDATE
Ok, here is an example implementation of RNG extended with B-D shuffle. First, I implemented in Javascript Xorshift128+, using BigInt, which is apparently default RNG in V8 engine as well. Compared with C++ one, they produced identical output for first couple of dozen calls. 128bits seed as two 64bits words. Windows 10 x64, NodeJS 12.7.
const WIDTH = 2n ** 64n;
const MASK = WIDTH - 1n; // to keep things as 64bit values
class XorShift128Plus { // as described in https://v8.dev/blog/math-random
_state0 = 0n;
_state1 = 0n;
constructor(seed0, seed1) { // 128bit seed as 2 64bit values
this._state0 = BigInt(seed0) & MASK;
this._state1 = BigInt(seed1) & MASK;
if (this._state0 <= 0n)
throw new Error('seed 0 non-positive');
if (this._state1 <= 0n)
throw new Error('seed 1 non-positive');
}
next() {
let s1 = this._state0;
let s0 = this._state1;
this._state0 = s0;
s1 = ((s1 << 23n) ^ s1 ) & MASK;
s1 ^= (s1 >> 17n);
s1 ^= s0;
s1 ^= (s0 >> 26n);
this._state1 = s1;
return (this._state0 + this._state1) & MASK; // modulo WIDTH
}
}
Ok, then on top of XorShift128+ I've implemented B-D shuffle, with table of size 4. For your purpose you'll need table more than 84 entries, and power of two table is much easier to deal with, so let's say 128 entries table (7bit index) shall be good enough. Anyway, even with 4 entries table and 2bit index we need to know which bits to pick to form index. In original paper B-D discussed picking them from the back of rv as well as from front of rv etc. Here is where B-D shuffle needs another seed value - telling algorithm to pick say, bits from position 2 and 6.
class B_D_XSP {
_xsprng;
_seedBD = 0n;
_pos0 = 0n;
_pos1 = 0n;
_t; // B-D table, 4 entries
_Z = 0n;
constructor(seed0, seed1, seed2) { // note third seed for the B-D shuffle
this._xsprng = new XorShift128Plus(seed0, seed1);
this._seedBD = BigInt(seed2) & MASK;
if (this._seedBD <= 0n)
throw new Error('B-D seed non-positive');
this._pos0 = findPosition(this._seedBD); // first non-zero bit position
this._pos1 = findPosition(this._seedBD & (~(1n << this._pos0))); // second non-zero bit position
// filling up table and B-D shuffler
this._t = new Array(this._xsprng.next(), this._xsprng.next(), this._xsprng.next(), this._xsprng.next());
this._Z = this._xsprng.next();
}
index(rv) { // bit at first position plus 2*bit at second position
let idx = ((rv >> this._pos0) & 1n) + (((rv >> this._pos1) & 1n) << 1n);
return idx;
}
next() {
let retval = this._Z;
let j = this.index(this._Z);
this._Z = this._t[j];
this._t[j] = this._xsprng.next();
return retval;
}
}
Use example is as follow.
let rng = new B_D_XSP(1, 2, 4+64); // bits at second and sixth position to make index
console.log(rng._pos0.toString(10));
console.log(rng._pos1.toString(10));
console.log(rng.next());
console.log(rng.next());
console.log(rng.next());
Obviously, third seed value of say 8+128 would produce different permutation from what is shown in the example, you could play with it.
Last step would be to make 226bit random value by calling several (3 of 4) times B-D shuffled rng and combine 64bit values (and potential carry over) to make 226 random bits and then convert them to the deck shuffle.
Related
In this thread we see this simple and beautiful algorithm to shuffle an array:
function shuffle<T>(array: T[]): T[] {
return array.sort(() => Math.random() - 0.5);
}
And we can see comments that say this algorithm is biased. But I made a simple script to create an empirical probability distribution of the index that the last element of the array ends after the shuffle:
function shuffle(array) {
return array.sort(() => Math.random() - 0.5);
}
function generateDistribution(iterations = 10_000, arrayLength = 10) {
const testArray = Array(arrayLength - 1).fill("test");
const testTarget = "target";
testArray.push(testTarget);
const results = {};
for (let index = 0; index < iterations; index++) {
countTargetPosition();
}
return generateResult();
function countTargetPosition() {
const shuffled = shuffle(testArray);
shuffled.forEach((value, index) => {
if (value === testTarget) {
results[index] = results[index] + 1 || 1;
}
});
}
function generateResult() {
return Object.entries(results).map(([index, count]) => {
return {
[index]: count / iterations,
};
});
}
}
const result = generateDistribution()
document.write(`<h1>Result</h1>`)
document.write(JSON.stringify(result))
We expect an unbiased algorithm to have a uniform distribution, and the result is very close to that, even for array with 100 elements. Why is this algorithm biased then?
JavaScript doesn't specify a specific algorithm for sort, and this shuffling algorithm can give very biased results depending on the specific sorting algorithm in use. Below, I describe some simple, well-known sorting algorithms that give very biased results; I demonstrate that Firefox and Chrome both give very biased results for an array of length 4; and I give a general argument for why any sorting algorithm will give biased results (though not necessarily as biased as these explicit examples).
Example #1 — selection sort. In selection sort, we first find the least element and put it at index 0, then find the second-least element and put it at index 1, and so on. The important thing to note is that, with the comparison function () => Math.random() - 0.5, each argument to the comparison has an equal chance of being considered "less". So if you find the least element by iterating over the array and comparing each element to the previously-least element, then there's a 50% chance that you'll deem the last element to be the least, a 25% chance that you'll deem the second-to-last element to be the least, a 12.5% chance that you'll deem the third-to-last element to be the least, etc., therefore giving a biased distribution of which element goes first.
Example 2 — insertion sort. In insertion sort, we build up a "sorted" section of the array by taking each element in turn and inserting it into the right place in that sorted section (shifting all greater elements by one to make room for it). This means that the last element has a 50% chance of being deemed the least, a 25% chance of being deemed the second-least, a 12.5% chance of being deemed the third-least, etc.
Examples #3 and #4 — whatever Firefox and Chrome use for a four-element array.
Now, realistically, I wouldn't expect any implementation of sort to use exactly selection sort or insertion sort, since there are other algorithms that are much more efficient for large inputs. But sophisticated modern sorting algorithms, such as Timsort, incorporate multiple different sorting algorithms, adaptively choosing between them based on the size and characteristics of the input (or of parts of the input, since they can combine these algorithms in sophisticated ways). So, as an experiment, I tried out this shuffle algorithm on the array [1, 2, 3, 4] — a short enough array that it seemed likely that the sort implementation might just use insertion sort for the whole array.
Here's the code I used:
const counts = {};
for (let i = 0; i < 1_000_000; ++i) {
const permutation = [1, 2, 3, 4].sort(() => Math.random() - 0.5).join('');
counts[permutation] = (counts[permutation]||0) + 1;
}
const result = [];
for (let permutation in counts) {
result.push(permutation + ': ' + counts[permutation]);
}
result.join('\n')
I tried this in both Firefox and Chrome.
In Firefox, I got results like this:
1234: 125747
1243: 62365
1324: 62299
1342: 31003
1423: 31320
1432: 15635
2134: 125380
2143: 62216
2314: 62615
2341: 31255
2413: 31509
2431: 15608
3124: 62377
3142: 31166
3214: 62194
3241: 31293
3412: 15631
3421: 15782
4123: 31056
4132: 15672
4213: 31231
4231: 15319
4312: 15727
4321: 15600
which doesn't match what I'd expect from insertion sort, so it must be doing something different, but regardless, it shows very clear biases. Some permutations occur 1/64 of the time (15,625 times out of a million, plus/minus random noise), some occur 1/32 of the time (31,250), some occur 1/16 of the time (62,500), and some occur 1/8 of the time (125,000); so some permutations are eight times as common as others.
In Chrome, I got results like this:
1234: 187029
1243: 62380
1324: 15409
1342: 15679
1423: 62476
1432: 15368
2134: 31280
2143: 31291
2314: 15683
2341: 15482
2413: 31482
2431: 15732
3124: 15786
3142: 15692
3214: 47186
3241: 47092
3412: 15509
3421: 46600
4123: 62825
4132: 15595
4213: 31091
4231: 15763
4312: 15624
4321: 171946
which also doesn't match what I'd expect from insertion sort, and is a bit more complicated than the distribution in Firefox (I think I see some 3/16ths (187,500) and 3/64ths (46,875) in there?), but is actually even more heavily biased, with a factor of twelve difference between the most and least common permutations.
Example #5 — any deterministic sorting algorithm. I've given various examples above of fairly extreme biases; but really, any sorting algorithm would be expected to produce some bias, because if the algorithm does worst-case k comparisons on an array of length n, and each comparison has a 50–50 split, then the probability of any given permutation has to be a multiple of 1/2k, whereas an unbiased shuffler has to give each permutation the probability 1/n!, which won't be a multiple of 1/2k if n ≥ 3 (because then n! will be a multiple of 3).
That said, I should acknowledge that these biases might be small enough that they don't matter; after all, even 1.0 / 3.0 doesn't compute exactly 1/3, but rather, rounds it to a binary approximation. Of more direct relevance, a typical implementation of Math.random() holds 64 or 128 bits of internal state, which means that it doesn't even have 21! or 35! distinct internal states, which means that no algorithm that uses Math.random() to shuffle an array of 21 or 35 or more elements can possibly produce each permutation with nonzero probability. So I suppose that some bias is inevitable!
Even if you're using a sort implementation that gives results that you consider good enough, there's just no reason to do it this way, since a Fisher–Yates shuffle is simple to code and is faster than any comparison-based sorting algorithm.
But I made a simple script to create an empirical probability distribution of the index that the last element of the array ends after the shuffle: […]
Note that there can potentially be subtler biases such that not all permutations are equally likely even if the last element is equally likely to end up at any position. Even if the sort implementation is held fixed, you'd want to do a more thorough analysis (probably including a look at its source code) before relying on this shuffling algorithm giving unbiased results.
The reason is that the comparison outcomes are incoherent, and as ruakh detailed, this can bias some sorting algorithms.
A correct solution is to associate a random key to the elements and sort on the key. But this takes O(n) extra space.
I have gone through Google and Stack Overflow search, but nowhere I was able to find a clear and straightforward explanation for how to calculate time complexity.
What do I know already?
Say for code as simple as the one below:
char h = 'y'; // This will be executed 1 time
int abc = 0; // This will be executed 1 time
Say for a loop like the one below:
for (int i = 0; i < N; i++) {
Console.Write('Hello, World!!');
}
int i=0; This will be executed only once.
The time is actually calculated to i=0 and not the declaration.
i < N; This will be executed N+1 times
i++ This will be executed N times
So the number of operations required by this loop are {1+(N+1)+N} = 2N+2. (But this still may be wrong, as I am not confident about my understanding.)
OK, so these small basic calculations I think I know, but in most cases I have seen the time complexity as O(N), O(n^2), O(log n), O(n!), and many others.
How to find time complexity of an algorithm
You add up how many machine instructions it will execute as a function of the size of its input, and then simplify the expression to the largest (when N is very large) term and can include any simplifying constant factor.
For example, lets see how we simplify 2N + 2 machine instructions to describe this as just O(N).
Why do we remove the two 2s ?
We are interested in the performance of the algorithm as N becomes large.
Consider the two terms 2N and 2.
What is the relative influence of these two terms as N becomes large? Suppose N is a million.
Then the first term is 2 million and the second term is only 2.
For this reason, we drop all but the largest terms for large N.
So, now we have gone from 2N + 2 to 2N.
Traditionally, we are only interested in performance up to constant factors.
This means that we don't really care if there is some constant multiple of difference in performance when N is large. The unit of 2N is not well-defined in the first place anyway. So we can multiply or divide by a constant factor to get to the simplest expression.
So 2N becomes just N.
This is an excellent article: Time complexity of algorithm
The below answer is copied from above (in case the excellent link goes bust)
The most common metric for calculating time complexity is Big O notation. This removes all constant factors so that the running time can be estimated in relation to N as N approaches infinity. In general you can think of it like this:
statement;
Is constant. The running time of the statement will not change in relation to N.
for ( i = 0; i < N; i++ )
statement;
Is linear. The running time of the loop is directly proportional to N. When N doubles, so does the running time.
for ( i = 0; i < N; i++ ) {
for ( j = 0; j < N; j++ )
statement;
}
Is quadratic. The running time of the two loops is proportional to the square of N. When N doubles, the running time increases by N * N.
while ( low <= high ) {
mid = ( low + high ) / 2;
if ( target < list[mid] )
high = mid - 1;
else if ( target > list[mid] )
low = mid + 1;
else break;
}
Is logarithmic. The running time of the algorithm is proportional to the number of times N can be divided by 2. This is because the algorithm divides the working area in half with each iteration.
void quicksort (int list[], int left, int right)
{
int pivot = partition (list, left, right);
quicksort(list, left, pivot - 1);
quicksort(list, pivot + 1, right);
}
Is N * log (N). The running time consists of N loops (iterative or recursive) that are logarithmic, thus the algorithm is a combination of linear and logarithmic.
In general, doing something with every item in one dimension is linear, doing something with every item in two dimensions is quadratic, and dividing the working area in half is logarithmic. There are other Big O measures such as cubic, exponential, and square root, but they're not nearly as common. Big O notation is described as O ( <type> ) where <type> is the measure. The quicksort algorithm would be described as O (N * log(N )).
Note that none of this has taken into account best, average, and worst case measures. Each would have its own Big O notation. Also note that this is a VERY simplistic explanation. Big O is the most common, but it's also more complex that I've shown. There are also other notations such as big omega, little o, and big theta. You probably won't encounter them outside of an algorithm analysis course. ;)
Taken from here - Introduction to Time Complexity of an Algorithm
1. Introduction
In computer science, the time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the string representing the input.
2. Big O notation
The time complexity of an algorithm is commonly expressed using big O notation, which excludes coefficients and lower order terms. When expressed this way, the time complexity is said to be described asymptotically, i.e., as the input size goes to infinity.
For example, if the time required by an algorithm on all inputs of size n is at most 5n3 + 3n, the asymptotic time complexity is O(n3). More on that later.
A few more examples:
1 = O(n)
n = O(n2)
log(n) = O(n)
2 n + 1 = O(n)
3. O(1) constant time:
An algorithm is said to run in constant time if it requires the same amount of time regardless of the input size.
Examples:
array: accessing any element
fixed-size stack: push and pop methods
fixed-size queue: enqueue and dequeue methods
4. O(n) linear time
An algorithm is said to run in linear time if its time execution is directly proportional to the input size, i.e. time grows linearly as input size increases.
Consider the following examples. Below I am linearly searching for an element, and this has a time complexity of O(n).
int find = 66;
var numbers = new int[] { 33, 435, 36, 37, 43, 45, 66, 656, 2232 };
for (int i = 0; i < numbers.Length - 1; i++)
{
if(find == numbers[i])
{
return;
}
}
More Examples:
Array: Linear Search, Traversing, Find minimum etc
ArrayList: contains method
Queue: contains method
5. O(log n) logarithmic time:
An algorithm is said to run in logarithmic time if its time execution is proportional to the logarithm of the input size.
Example: Binary Search
Recall the "twenty questions" game - the task is to guess the value of a hidden number in an interval. Each time you make a guess, you are told whether your guess is too high or too low. Twenty questions game implies a strategy that uses your guess number to halve the interval size. This is an example of the general problem-solving method known as binary search.
6. O(n2) quadratic time
An algorithm is said to run in quadratic time if its time execution is proportional to the square of the input size.
Examples:
Bubble Sort
Selection Sort
Insertion Sort
7. Some useful links
Big-O Misconceptions
Determining The Complexity Of Algorithm
Big O Cheat Sheet
Several examples of loop.
O(n) time complexity of a loop is considered as O(n) if the loop variables is incremented / decremented by a constant amount. For example following functions have O(n) time complexity.
// Here c is a positive integer constant
for (int i = 1; i <= n; i += c) {
// some O(1) expressions
}
for (int i = n; i > 0; i -= c) {
// some O(1) expressions
}
O(nc) time complexity of nested loops is equal to the number of times the innermost statement is executed. For example, the following sample loops have O(n2) time complexity
for (int i = 1; i <=n; i += c) {
for (int j = 1; j <=n; j += c) {
// some O(1) expressions
}
}
for (int i = n; i > 0; i += c) {
for (int j = i+1; j <=n; j += c) {
// some O(1) expressions
}
For example, selection sort and insertion sort have O(n2) time complexity.
O(log n) time complexity of a loop is considered as O(log n) if the loop variables is divided / multiplied by a constant amount.
for (int i = 1; i <=n; i *= c) {
// some O(1) expressions
}
for (int i = n; i > 0; i /= c) {
// some O(1) expressions
}
For example, [binary search][3] has _O(log n)_ time complexity.
O(log log n) time complexity of a loop is considered as O(log log n) if the loop variables is reduced / increased exponentially by a constant amount.
// Here c is a constant greater than 1
for (int i = 2; i <=n; i = pow(i, c)) {
// some O(1) expressions
}
//Here fun is sqrt or cuberoot or any other constant root
for (int i = n; i > 0; i = fun(i)) {
// some O(1) expressions
}
One example of time complexity analysis
int fun(int n)
{
for (int i = 1; i <= n; i++)
{
for (int j = 1; j < n; j += i)
{
// Some O(1) task
}
}
}
Analysis:
For i = 1, the inner loop is executed n times.
For i = 2, the inner loop is executed approximately n/2 times.
For i = 3, the inner loop is executed approximately n/3 times.
For i = 4, the inner loop is executed approximately n/4 times.
…………………………………………………….
For i = n, the inner loop is executed approximately n/n times.
So the total time complexity of the above algorithm is (n + n/2 + n/3 + … + n/n), which becomes n * (1/1 + 1/2 + 1/3 + … + 1/n)
The important thing about series (1/1 + 1/2 + 1/3 + … + 1/n) is around to O(log n). So the time complexity of the above code is O(n·log n).
References:
1
2
3
Time complexity with examples
1 - Basic operations (arithmetic, comparisons, accessing array’s elements, assignment): The running time is always constant O(1)
Example:
read(x) // O(1)
a = 10; // O(1)
a = 1,000,000,000,000,000,000 // O(1)
2 - If then else statement: Only taking the maximum running time from two or more possible statements.
Example:
age = read(x) // (1+1) = 2
if age < 17 then begin // 1
status = "Not allowed!"; // 1
end else begin
status = "Welcome! Please come in"; // 1
visitors = visitors + 1; // 1+1 = 2
end;
So, the complexity of the above pseudo code is T(n) = 2 + 1 + max(1, 1+2) = 6. Thus, its big oh is still constant T(n) = O(1).
3 - Looping (for, while, repeat): Running time for this statement is the number of loops multiplied by the number of operations inside that looping.
Example:
total = 0; // 1
for i = 1 to n do begin // (1+1)*n = 2n
total = total + i; // (1+1)*n = 2n
end;
writeln(total); // 1
So, its complexity is T(n) = 1+4n+1 = 4n + 2. Thus, T(n) = O(n).
4 - Nested loop (looping inside looping): Since there is at least one looping inside the main looping, running time of this statement used O(n^2) or O(n^3).
Example:
for i = 1 to n do begin // (1+1)*n = 2n
for j = 1 to n do begin // (1+1)n*n = 2n^2
x = x + 1; // (1+1)n*n = 2n^2
print(x); // (n*n) = n^2
end;
end;
Common running time
There are some common running times when analyzing an algorithm:
O(1) – Constant time
Constant time means the running time is constant, it’s not affected by the input size.
O(n) – Linear time
When an algorithm accepts n input size, it would perform n operations as well.
O(log n) – Logarithmic time
Algorithm that has running time O(log n) is slight faster than O(n). Commonly, algorithm divides the problem into sub problems with the same size. Example: binary search algorithm, binary conversion algorithm.
O(n log n) – Linearithmic time
This running time is often found in "divide & conquer algorithms" which divide the problem into sub problems recursively and then merge them in n time. Example: Merge Sort algorithm.
O(n2) – Quadratic time
Look Bubble Sort algorithm!
O(n3) – Cubic time
It has the same principle with O(n2).
O(2n) – Exponential time
It is very slow as input get larger, if n = 1,000,000, T(n) would be 21,000,000. Brute Force algorithm has this running time.
O(n!) – Factorial time
The slowest!!! Example: Travelling salesman problem (TSP)
It is taken from this article. It is very well explained and you should give it a read.
When you're analyzing code, you have to analyse it line by line, counting every operation/recognizing time complexity. In the end, you have to sum it to get whole picture.
For example, you can have one simple loop with linear complexity, but later in that same program you can have a triple loop that has cubic complexity, so your program will have cubic complexity. Function order of growth comes into play right here.
Let's look at what are possibilities for time complexity of an algorithm, you can see order of growth I mentioned above:
Constant time has an order of growth 1, for example: a = b + c.
Logarithmic time has an order of growth log N. It usually occurs when you're dividing something in half (binary search, trees, and even loops), or multiplying something in same way.
Linear. The order of growth is N, for example
int p = 0;
for (int i = 1; i < N; i++)
p = p + 2;
Linearithmic. The order of growth is n·log N. It usually occurs in divide-and-conquer algorithms.
Cubic. The order of growth is N3. A classic example is a triple loop where you check all triplets:
int x = 0;
for (int i = 0; i < N; i++)
for (int j = 0; j < N; j++)
for (int k = 0; k < N; k++)
x = x + 2
Exponential. The order of growth is 2N. It usually occurs when you do exhaustive search, for example, check subsets of some set.
Loosely speaking, time complexity is a way of summarising how the number of operations or run-time of an algorithm grows as the input size increases.
Like most things in life, a cocktail party can help us understand.
O(N)
When you arrive at the party, you have to shake everyone's hand (do an operation on every item). As the number of attendees N increases, the time/work it will take you to shake everyone's hand increases as O(N).
Why O(N) and not cN?
There's variation in the amount of time it takes to shake hands with people. You could average this out and capture it in a constant c. But the fundamental operation here --- shaking hands with everyone --- would always be proportional to O(N), no matter what c was. When debating whether we should go to a cocktail party, we're often more interested in the fact that we'll have to meet everyone than in the minute details of what those meetings look like.
O(N^2)
The host of the cocktail party wants you to play a silly game where everyone meets everyone else. Therefore, you must meet N-1 other people and, because the next person has already met you, they must meet N-2 people, and so on. The sum of this series is x^2/2+x/2. As the number of attendees grows, the x^2 term gets big fast, so we just drop everything else.
O(N^3)
You have to meet everyone else and, during each meeting, you must talk about everyone else in the room.
O(1)
The host wants to announce something. They ding a wineglass and speak loudly. Everyone hears them. It turns out it doesn't matter how many attendees there are, this operation always takes the same amount of time.
O(log N)
The host has laid everyone out at the table in alphabetical order. Where is Dan? You reason that he must be somewhere between Adam and Mandy (certainly not between Mandy and Zach!). Given that, is he between George and Mandy? No. He must be between Adam and Fred, and between Cindy and Fred. And so on... we can efficiently locate Dan by looking at half the set and then half of that set. Ultimately, we look at O(log_2 N) individuals.
O(N log N)
You could find where to sit down at the table using the algorithm above. If a large number of people came to the table, one at a time, and all did this, that would take O(N log N) time. This turns out to be how long it takes to sort any collection of items when they must be compared.
Best/Worst Case
You arrive at the party and need to find Inigo - how long will it take? It depends on when you arrive. If everyone is milling around you've hit the worst-case: it will take O(N) time. However, if everyone is sitting down at the table, it will take only O(log N) time. Or maybe you can leverage the host's wineglass-shouting power and it will take only O(1) time.
Assuming the host is unavailable, we can say that the Inigo-finding algorithm has a lower-bound of O(log N) and an upper-bound of O(N), depending on the state of the party when you arrive.
Space & Communication
The same ideas can be applied to understanding how algorithms use space or communication.
Knuth has written a nice paper about the former entitled "The Complexity of Songs".
Theorem 2: There exist arbitrarily long songs of complexity O(1).
PROOF: (due to Casey and the Sunshine Band). Consider the songs Sk defined by (15), but with
V_k = 'That's the way,' U 'I like it, ' U
U = 'uh huh,' 'uh huh'
for all k.
For the mathematically-minded people: The master theorem is another useful thing to know when studying complexity.
O(n) is big O notation used for writing time complexity of an algorithm. When you add up the number of executions in an algorithm, you'll get an expression in result like 2N+2. In this expression, N is the dominating term (the term having largest effect on expression if its value increases or decreases). Now O(N) is the time complexity while N is dominating term.
Example
For i = 1 to n;
j = 0;
while(j <= n);
j = j + 1;
Here the total number of executions for the inner loop are n+1 and the total number of executions for the outer loop are n(n+1)/2, so the total number of executions for the whole algorithm are n + 1 + n(n+1/2) = (n2 + 3n)/2.
Here n^2 is the dominating term so the time complexity for this algorithm is O(n2).
Other answers concentrate on the big-O-notation and practical examples. I want to answer the question by emphasizing the theoretical view. The explanation below is necessarily lacking in details; an excellent source to learn computational complexity theory is Introduction to the Theory of Computation by Michael Sipser.
Turing Machines
The most widespread model to investigate any question about computation is a Turing machine. A Turing machine has a one dimensional tape consisting of symbols which is used as a memory device. It has a tapehead which is used to write and read from the tape. It has a transition table determining the machine's behaviour, which is a fixed hardware component that is decided when the machine is created. A Turing machine works at discrete time steps doing the following:
It reads the symbol under the tapehead.
Depending on the symbol and its internal state, which can only take finitely many values, it reads three values s, σ, and X from its transition table, where s is an internal state, σ is a symbol, and X is either Right or Left.
It changes its internal state to s.
It changes the symbol it has read to σ.
It moves the tapehead one step according to the direction in X.
Turing machines are powerful models of computation. They can do everything that your digital computer can do. They were introduced before the advent of digital modern computers by the father of theoretical computer science and mathematician: Alan Turing.
Time Complexity
It is hard to define the time complexity of a single problem like "Does white have a winning strategy in chess?" because there is a machine which runs for a single step giving the correct answer: Either the machine which says directly 'No' or directly 'Yes'. To make it work we instead define the time complexity of a family of problems L each of which has a size, usually the length of the problem description. Then we take a Turing machine M which correctly solves every problem in that family. When M is given a problem of this family of size n, it solves it in finitely many steps. Let us call f(n) the longest possible time it takes M to solve problems of size n. Then we say that the time complexity of L is O(f(n)), which means that there is a Turing machine which will solve an instance of it of size n in at most C.f(n) time where C is a constant independent of n.
Isn't it dependent on the machines? Can digital computers do it faster?
Yes! Some problems can be solved faster by other models of computation, for example two tape Turing machines solve some problems faster than those with a single tape. This is why theoreticians prefer to use robust complexity classes such as NL, P, NP, PSPACE, EXPTIME, etc. For example, P is the class of decision problems whose time complexity is O(p(n)) where p is a polynomial. The class P do not change even if you add ten thousand tapes to your Turing machine, or use other types of theoretical models such as random access machines.
A Difference in Theory and Practice
It is usually assumed that the time complexity of integer addition is O(1). This assumption makes sense in practice because computers use a fixed number of bits to store numbers for many applications. There is no reason to assume such a thing in theory, so time complexity of addition is O(k) where k is the number of bits needed to express the integer.
Finding The Time Complexity of a Class of Problems
The straightforward way to show the time complexity of a problem is O(f(n)) is to construct a Turing machine which solves it in O(f(n)) time. Creating Turing machines for complex problems is not trivial; one needs some familiarity with them. A transition table for a Turing machine is rarely given, and it is described in high level. It becomes easier to see how long it will take a machine to halt as one gets themselves familiar with them.
Showing that a problem is not O(f(n)) time complexity is another story... Even though there are some results like the time hierarchy theorem, there are many open problems here. For example whether problems in NP are in P, i.e. solvable in polynomial time, is one of the seven millennium prize problems in mathematics, whose solver will be awarded 1 million dollars.
I have an array of N HTMLCanvasElements that come from N frames of a video, and I want to compute the "median canvas" in the sense that every component (r, g, b, opacity) of every pixel is the median of the corresponding component in all the canvases.
The video frames are 1280x720, so that the pixels data for every canvas (obtained with canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height).data) is a Uint8ClampedArray of length 3.686.400.
The naive way to compute the median is to:
prepare a result Uint8ClampedArray of length 3.686.400
prepare a temporary Uint8ClampedArray of length N
loop from 0 to 3.686.399
a) loop over the N canvases to fill the array
b) compute the median of the array
c) store the median to the result array
But it's very slow, even for 4 canvases.
Is there an efficient way (or existing code) to do that? My question is very similar to Find median of list of images, but I need to to this in JavaScript, not Python.
Note: for b), I use d3.median() which doesn't work on typed arrays, as far as I understand, so that it implies converting to numbers, then converting back to Uint8Clamped.
Note 2: I don't know much of GLSL shaders, but maybe using the GPU would be a way to get faster results. It would require to pass data from the CPU to the GPU though, which takes time if done repeatedly.
Note 3: the naive solution is there: https://observablehq.com/#severo/compute-the-approximate-median-image-of-a-video
You wrote
I use d3.median() which doesn't work on typed arrays…
Although that is not exactly true it leads into the right direction. Internally d3.median() uses the d3.quantile() method which starts off like this:
export default function quantile(values, p, valueof) {
values = Float64Array.from(numbers(values, valueof));
As you can see, this in fact does make use of typed arrays, it is just not your Uint8ClampedArray but a Float64Array instead. Because floating-point arithmetic is much more computation-intensive than its integer counterpart (including the conversion itself) this has a dramatic effect on the performance of your code. Doing this some 3 million times in a tight loop kills the efficiency of your solution.
Since you are retrieving all your pixel values from a Uint8ClampedArray you can be sure that you are always dealing with integers, though. That said, it is fairly easy to build a custom function median(values) derived from d3.median() and d3.quantile():
function median(values) {
// No conversion to floating point values needed.
if (!(n = values.length)) return;
if (n < 2) return d3.min(values);
var n,
i = (n - 1) * 0.5,
i0 = Math.floor(i),
value0 = d3.max(d3.quickselect(values, i0).subarray(0, i0 + 1)),
value1 = d3.min(values.subarray(i0 + 1));
return value0 + (value1 - value0) * (i - i0);
}
On top of getting rid of the problematic conversion on the first line this implementation additionally applies some more micro-optimizations because in your case you are always looking for the 2-quantile (i.e. the median). That might not seem much at first, but doing this multiple million times in a loop it does make a difference.
With minimal changes to your own code you can call it like this:
// medianImageData.data[i] = d3.median(arr); Instead of this use line below.
medianImageData.data[i] = median(arr);
Have a look at my working fork of your Observable notebook.
This problem I was asked yesterday. I had to write a code to split the array into two parts such that the difference between the sum of these two part would be minimum.
Here is the code I wrote with the complexity O(n)
function solution(a) {
let leftSum = 0;
let rightSum = a.reduce((acc, value) => acc + value ,0);
let min = Math.abs(rightSum - leftSum);
a.forEach((item, i) => {
leftSum += a[i];
rightSum -= a[i];
const tempMin = Math.abs(rightSum - leftSum);
if(tempMin < min) min = tempMin;
})
return min;
}
But then I was asked if the input array is of length 10 million, how would I solve this problem in a distributed environment?
I am new to distributed programming, need help in this.
If you have N node,s then split the array into N sequential subarrays; this will give you N sequential sums. Take a pass to determine which subarray contains the desired split point. The difference between the "before" and "after" sums is your bias target for the next phase ...
Now divide that "middle" array into N pieces. Again, you look for the appropriate split point, except that now you know the exact result you'd like (since you have the array sum and your missing difference).
Repeat that second paragraph until you can fit the entire subarray into one node and that's the fastest way to finish the computation for your project.
You can speed this up somewhat by keeping a cumulative sum at each value; this will allow you to find the appropriate split point somewhat faster at each stage, as you can use a binary or interpolation search for every stage after the first.
Given an array of length N, and given M available nodes, divide the array into chunks of size N/M. Each node computes the sum of its chunk, and reports back. The total is computed by adding the partial sums. Then the total and the partial sums are distributed to each of the nodes. Each node determines the best split point within its chunk (the local minimum), and reports back. The global minimum is computed from the local minimums.
For example, if the array has 10 million entries, and 200 nodes are available, the chunk size is 50000. So each node receives 50000 numbers, and reports back the sum. The total of the array is computed by adding the 200 partial sums. Then each node is given the total, along with the 200 partial sums. The information at each node now consists of
a chunk number
the 50000 array entries for that chunk
the array total
the 200 partial sums
From that information, each node can compute its local minimum. The global minimum is computed from the 200 local minimums.
In the ideal case, where network bandwidth is infinite, network latency is zero, and any number of nodes can be used, the chunk size should be sqrt(N). So each node receives sqrt(N) array elements, and then receives sqrt(N) partial sums. Under those ideal conditions, the running time is O(sqrt(N)) instead of O(N).
Of course, in the real world, it makes no sense to try to distribute a problem like this. The amount of time (per array element) to send the array elements over the network is significant. Much larger than the amount of time (per array element) needed to solve the problem on a single computer.
Assume the array is stored sequentially over several nodes N_1, ..., N_k. A simple distributed version of your original algorithm could be the following.
On each N_i, calculate the sum s_i of the subarray stored on N_i and send it to a control node M
On node M, using s_1, ..., s_k, calculate leftSum_i and rightSum_i for the left subarray boundary of each N_i and send them back to N_i
On each N_i, using leftSum_i and rightSum_i, conduct search to find the minimum min_i and send it back to M
On node M, calculate the global minimum min from min_i, ... min_k
A side note: your original algorithm can be optimized to keep only the value rightSum - leftSum rather than two separate values leftSum and rightSum. The distributed version can also be optimized correspondingly.
I am trying to generate a random number between 1 and a maximum. This I don't have a problem doing so and do so with the following:
var max = 200;
var randomNumber = Math.floor(Math.random() * max) + 1;
However in an ideal situation I would like to generate a number between 1 and my maximum however the lower numbers have a higher probability of occurring. I want the variable to be biased towards 1. However my maths skills aren't strong enough to work this out, it would be great if someone could point me in the right direction.
Thank you,
Josh
a simple way will be to just square the result of Math.random(). Since the result of the function is between 0 and 1 , the square will also be in the range [0, 1], but values , for example , 0.5 from it will be mapped to lower ones - 0.25 . You can experiment with powers above 1 until you find an acceptable function.
I got a code in java which does what you want.
You should choose your own probabilities for the int[] prob arrays.
I think it wont be that hard to translate this to js or build smth. equal.
int[] probs;
void initRandom(int n, int[] probabilities)
{
int i,j,begin=0,end=0,sum=0;
int[] probs;
// sum of all propabilitys must be 100%
for(i=0;i<probabilities.length;i++) sum+=probabilities[i];
probs=new int[sum];
// fills numbers from 0 till n-1 in regard to their probabilities
// to the probability array.
for(i=0;i<n;i++)
{
begin=end;
end+=probabilities[i];
for(j=begin;j<end;j++) probs[j]=i;
}
}
int genRandom()
{
return probs[smallRand(probs.length-1)];
}
This is a very general question. First consider this link here
http://en.wikipedia.org/wiki/List_of_probability_distributions#Supported_on_a_bounded_interval
It shows some probability functions which are bounded, which I believe is what you are looking for (since min=1 and max=max).
You can also chose a semi-infine interval, and just ignore all value above your maximum. I think, this could also be acceptable, depending on your application.
Next, chose one of those probabilty functions, that suits you best. For the sake of simplicity, I chose the triangular distribution
The distribution functions are (PDF and CDF)
f(x) = 2/(2*max-1-max^2)*(x-max)
F(x) = 2/(2*max-1-max^2)*(0.5*x^2-max*x-0.5+max)
so I can generate from a uniform distribution on 0-1 a biased distribution by inverting the CDF like
var urand = Math.random();
var a = 2/(2*max-1-max^2);
var randomNumber = max-Math.sqrt(max*max-2*(max-urand/a-0.5));
Cheers
R
The following function i made up gives you a near-one-biased random number
function rand(max) {
var r = Math.random();
r = 1/(101-100 * r);
return Math.floor(r * max) - 1;
}
It only uses simple arithmetics, thus, it should be quite fast.