Develop Reference

Use the audiocontext as audio input in p5.js

I am implementing a synthesizer which uses the nodes of the audio-api to generate sound and my goal is to visualize it using p5.
I currently have a script that analyzes audio with fft and visualizes the frequencies with bars. My audio input at the moment is a locally saved song but I need to change it, so it uses the audiocontext as input.
Currently I can get the audiocontext with p5's own method getAudioContext() but then I have no clue how to set it as input for the visualization.
I know the API has a createBuffer()-Method but I haven't found a way to use it as input for p5.
var fft;
var button;
var song;
var slider;
var audiocontext;
var out;
var prue;
var source;
function preload(){
song = loadSound("src/media/Chopin - Nocturne op.9 No.2.mp3");
button = createButton("Play");
button.mousePressed(togglePlaying);
slider = createSlider(0,1,0.5,0.01);
this.audiocontext = getAudioContext();
}
function setup() {
createCanvas(windowWidth,windowHeight);
fft = new p5.FFT(0.8);
source = context.createBufferSource();
widthBand = (width / 128);
source.connect(context.destination);
}
function draw() {
background(61);
var spectrum = fft.analyze();
noStroke();
for (var i = 0; i<spectrum.length; i++) {
var amp = spectrum[i];
var y = map(amp, 0, 256, height, 0);
fill(i, 255, 255);
rect(i*widthBand,y,widthBand-2, height - y );
}
//Set Volume according to slider
audiocontext.setVolume(slider.value());
}
//Play/Pause Button
function togglePlaying(){
if(!song.isPlaying()){
song.play();
button.html("Pause");
}else if(song.isPlaying()){
song.pause();
button.html("Play");
}
}
Any help would be very appreciated!

Audiocontext is not an input himself but contains one or more input nodes (and output and connections and ...). P5 creates own Audiocontext and operates inside of that.
So, option one: build your app using p5 functionality only. It's a powerful library, all the needed tools (e.g. AudioIn(), MonoSynth() etc.) should be available.
Option two: initialize p5 first and then use p5 created audiocontext to add extra nodes, which can later be used by p5.
var cnv, fft, audiocontext, osc;
//p5 setup.
function setup() {
cnv = createCanvas();
fft = new p5.FFT(0.8);
audiocontext = getAudioContext(); //if p5.Audiocontext doesn't exist
// then new is created. Let's make
// it global.
myCustomSetup(); //now we can create our own input nodes, filters...
fft.setInput(osc); //after which we can connect fft to those created
//nodes
}
function myCustomSetup() {
//p5 audiocontext is usable here, allowing to use full WebAudioApi
//and connect all nodes, created here or by some p5 function.
osc = audiocontext.createOscillator();
}

Firebase Javascript + P5.js: Asynchronous function getting in the way of redrawing the canvas

I'm trying to create an application where circles are drawn onto the canvas through reading information from a Firebase database that stores the x and y coordinates of the circles. Executing the code below however, simply produces nothing, without any sign of the circles, because the function drawCricles runs asynchronously, and thus the command background(40) clears everything before the circles can be drawn.
Here is my code:
function setup() {
createCanvas(windowWidth, windowHeight);
background(40);
stroke(80);
smooth();
frameRate(60);
}
function drawCircles() {
firebase.database().ref("circles").once("value", function(snapshot) {
var snapshotVal = snapshot.val();
var circleCount = snapshotVal.numCircles;
for (var j = 0; j < circleCount; j++) {
firebase.database().ref("circles" + j).once("value", function(snapshot) {
var snapshotValue = snapshot.val();
fill(143, 2, 2);
ellipse(snapshotValue.xPos, 50, 50);
});
}
});
}
function draw() {
stroke(80);
background(40);
stroke(0);
drawCircles();
}

It seems your problem is simply 60 frames per second, which is causing a foot-race condition. Firebase .once will execute async when it completes fetching, and P5 won't wait for it to fetch because it'll stick with its framerate timing.
In this specific case I have multiple recommendations which will hopefully get you very close to the result you desire.
1 - Restructure your code
There's two issues with the current structure of your code.
Case 1 : Your current code would lead me to think your circles are updated in realtime in the database, and you need to stay up to date, so you keep fetching their latest position. If this is the case, you should use .on("value") instead of .once("value") and let firebase send you the updates whenever circles change, instead of asking it 60 times a second to save roundtrip request time. If this is the case : See my solution 1 below.
Case 2 : If your circles aren't updated real-time in the database, and you just want the whole list of circles, then you're fetching the list 60 times a second for no reason. You should instead fetch the list using .once upon setup and iterate through that list in draw() later. See solution 2 below.
2 - Restructure your database
In either case, your current database model requires you to keep fetching in a loop. Which means you're making as many requests as your circleCount. This is bad for your usage, simply because each request takes additional trip time, and we're trying to reduce the time it takes, so that it would be closer to real-time. (or match the framerate)
Currently your circles are seemingly saved as circles1 circles2 etc all at root, because you're using .ref("circles" + j) to retrieve them. Make it so that you save your circles like this : .ref("circles/" + j) that would mean that each circle is now saved in circles. like circles/circle1 circles/circle2 etc.
The benefit of this is that now you don't need the additional requests to firebase get all circles. Firebase has incredibly convenient things like forEach to iterate through all children with a single request.
3 - Clear background in your firebase callback
Currently, you clear the background in a frame-rate specific manner. This means that if each of your firebase calls take longer than 1/60th of a second (16 miliseconds) you will have cleared the background and move on. Chances of you achieving this speed is very low even after we structure our database. so instead, I would recommend first using 30fps, which would also reduce the number of calls you will make to firebase to 30 calls per second.
Solution 1
If your circles are updated in the database (say for example by some other game-player or someone else, and you want your code to always display the latest xPos)
var latestCirclePositionsSnapshot;
function setup() {
createCanvas(windowWidth, windowHeight);
background(40);
stroke(80);
smooth();
frameRate(60);
firebase.database().ref("circles").on("value", function(snapshot) {
// got a new value from database, so let's save this in a global variable.
latestCirclePositionsSnapshot = snapshot;
// we will keep drawing this update until we get a new one from the database.
});
}
function draw() {
drawCircles();
}
function clearBackground () {
stroke(80);
background(40);
}
function drawCircles() {
clearBackground();
stroke(0);
latestCirclePositionsSnapshot.forEach(function(circleSnapshot) {
// circleData will be the actual contents of each circle
var circleData = circleSnapshot.val();
fill(143, 2, 2);
ellipse(circleData.xPos, 50, 50);
});
}
Basically this will keep drawing the last circle positions we got from firebase until we get a new one. (so P5 will keep refreshing at 60fps, but your firebase updates will be as realtime as firebase can run and fetch from firebase etc.)
Solution 2
If you don't have real-time updates in your database, and all you'd like is to draw circles by getting data from firebase once (say for example to plot some dots based on some data)
var circlePositions;
var gotPositions = false;
function setup() {
createCanvas(windowWidth, windowHeight);
background(40);
stroke(80);
smooth();
frameRate(60);
firebase.database().ref("circles").once("value", function(snapshot) {
// got the circle values from the database
// let's store them and we'll keep drawing them forever.
circlePositions = snapshot;
gotPositions = true;
});
}
function draw() {
drawCircles();
}
function clearBackground () {
stroke(80);
background(40);
}
function drawCircles() {
clearBackground();
stroke(0);
if (gotPositions) {
circlePositions.forEach(function(circleSnapshot) {
// circleData will be the actual contents of each circle
var circleData = circleSnapshot.val();
fill(143, 2, 2);
ellipse(circleData.xPos, 50, 50);
});
} else {
// Display some text here like "LOADING DATA FROM SERVERS..."
}
}
Hope these help :) It's good to see another fellow fan of Processing & Firebase.

I looked into the docs and here is an example of how they suggest to deal with a fetched data. In your case, try to separate fetching and drawing, cache your data using some global variable:
var circles = [];
function fetchData() {
firebase.database().ref("circles").once("value",
function(snapshot) {
var snapshotVal = snapshot.val();
var circleCount = snapshotVal.numCircles;
circles = [];
for (var j = 0; j < circleCount; j++) {
firebase.database().ref("circles" + j).once("value", function(snapshot) {
circles.push(snapshot.val());
});
}
});
}
function setup() {
createCanvas(windowWidth, windowHeight);
background(40);
stroke(80);
smooth();
frameRate(60);
fetchData();
}
function drawCircles() {
circles.forEach(function (snapshotValue) {
var snapshotValue = snapshot.val();
fill(143, 2, 2);
ellipse(snapshotValue.xPos, 50, 50);
});
}
function draw() {
stroke(80);
background(40);
stroke(0);
drawCircles();
}
If you need to display always relevant data, try to call fetchData function using setInterval, like:
function setup() {
createCanvas(windowWidth, windowHeight);
background(40);
stroke(80);
smooth();
frameRate(60);
setInterval(fetchData, 5000); //will call fetchData every 5000 ms
}

The problem is not that drawCircles() is asynchronous -- the problem is that draw() is called at frameRate(), and background() clears the screen with a solid color each fraction of a second when draw loops: see draw reference and background. If you remove the line background(40) from draw() then it will not clear the screen each frame, and drawn circles will accumulate as desired. This is simpler than redrawing all the Firebase data each frame.
The following sketch that demonstrates the concept: background() is only called during setup(), not during draw(), so the screen area is colored once, then progressively overwritten with circles that accumulate.
function setup() {
createCanvas(400, 200);
frameRate(5)
background(40);
}
function drawCircles() {
fill(143, 2, 2);
ellipse(random(width), 50, 50);
}
function draw() {
// background(40);
drawCircles();
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.16/p5.js"></script>
<html>
<head>
</head>
<body>
</body>
</html>
If you want to accumulate some things and wipe away others each frame, then you need to accumulate your circles on a createGraphics buffer. Each frame redraw that buffer of circles onto the canvas, then draw ephemeral elements (like a mouse indicator etc.) on top.
Here is an example: each frame the canvas is cleared with background(), then the pg buffer is drawn onto the canvas, then a white circle is drawn at the mouse. Because background clears the screen, the white circle does not leave trails behind from frame to frame -- but the red circles are drawn to an un-cleared graphics buffer, so they persist.
var pg;
function setup() {
createCanvas(400, 200);
pg = createGraphics(400, 200);
background(40);
}
function drawCircles() {
pg.fill(143, 2, 2);
pg.ellipse(random(pg.width), 50, 50);
}
function draw() {
background(40);
drawCircles();
image(pg,0,0);
fill(255);
ellipse(mouseX,mouseY,50,50);
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.16/p5.js"></script>
<html>
<head>
</head>
<body>
</body>
</html>

Exporting a video in p5.js

I am creating a simple animation program in p5.js. When a user clicks the save button, I want to download a video of the animation.
I have an object called frames where each key is labelled frame_1, frame_2 and so on. The value associated with each key is an array of line segments that makes up that frame.
I am trying to think of an approach to take this data and create an mp4 video. p5.js has a built in save function that I thought might be helpful but it is not a full solution on its own. I could save each frame as an individual image and then somehow stitch those images together on the client side but I have yet to find a solution to this.
Any other approaches would be great as well. The only requirement is that it is done client side.

Since p5.js is built on the Canvas API, in modern browsers, you can use a MediaRecorder to do this job.
const btn = document.querySelector('button'),
chunks = [];
function record() {
chunks.length = 0;
let stream = document.querySelector('canvas').captureStream(30),
recorder = new MediaRecorder(stream);
recorder.ondataavailable = e => {
if (e.data.size) {
chunks.push(e.data);
}
};
recorder.onstop = exportVideo;
btn.onclick = e => {
recorder.stop();
btn.textContent = 'start recording';
btn.onclick = record;
};
recorder.start();
btn.textContent = 'stop recording';
}
function exportVideo(e) {
var blob = new Blob(chunks);
var vid = document.createElement('video');
vid.id = 'recorded'
vid.controls = true;
vid.src = URL.createObjectURL(blob);
document.body.appendChild(vid);
vid.play();
}
btn.onclick = record;
// taken from pr.js docs
var x, y;
function setup() {
createCanvas(300, 200);
// Starts in the middle
x = width / 2;
y = height;
}
function draw() {
background(200);
// Draw a circle
stroke(50);
fill(100);
ellipse(x, y, 24, 24);
// Jiggling randomly on the horizontal axis
x = x + random(-1, 1);
// Moving up at a constant speed
y = y - 1;
// Reset to the bottom
if (y < 0) {
y = height;
}
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.7/p5.min.js"></script>
<button>start recording</button><br>

ccapture works well with p5.js to achieve the goal of recording what's displaying on a canvas.
Here is a demo of ccapture working with p5.js. The source code comes with the demo.
This method won't output laggy videos because it is not recording what you see on the screen, which can be laggy. Instead, it writes every frame into the video and tells the videos to play at a fixed frame rate. So even if it takes seconds to calculate just one frame, the output video will play smoothly without showing any delay between frames.
However, there is one caveat though. This method only works with Chrome.

As you specified in the comments that a gif would also work, here is a solution:
Below is a sample p5 sketch that records canvas animation and turns it into a gif, using gif.js.
Works in browsers supporting: Web Workers, File API and Typed Arrays.
I've provided this code so you can get an idea of how to use this library because not much documentation is provided for it and I had a hard time myself figuring it out.
var cnv;
var gif, recording = false;
function setup() {
cnv = createCanvas(400, 400);
var start_rec = createButton("Start Recording");
start_rec.mousePressed(saveVid);
var stop_rec = createButton("Stop Recording");
stop_rec.mousePressed(saveVid);
start_rec.position(500, 500);
stop_rec.position(650, 500);
setupGIF();
}
function saveVid() {
recording = !recording;
if (!recording) {
gif.render();
}
}
var x = 0;
var y = 0;
function draw() {
background(51);
fill(255);
ellipse(x, y, 20, 20);
x++;
y++;
if (recording) {
gif.addFrame(cnv.elt, {
delay: 1,
copy: true
});
}
}
function setupGIF() {
gif = new GIF({
workers: 5,
quality: 20
});
gif.on('finished', function(blob) {
window.open(URL.createObjectURL(blob));
});
}
More Info :
This sketch starts recording frames when you click start_rec and stops when you hit stop_rec, in your sketch you might want to control things differently, but keep in mind that addFrame only adds one frame to the gif so you need to call it in the draw function to add multiple frames, you can pass in an ImageElement, a CanvasElement or a CanvasContext along with other optional parameters.
In the gif.on function, you can specify a callback function to do whatever you like with the gif.
If you want to fine tune settings of the gif, like quality, repeat, background, you can read more here. Hope this helps!

JavaScript: initialize different image if an error occurs

I have a folder with the contents like the following (giving a minimal example)
ex1_1.png
ex1_2.png
ex2_1.png
page.html
and I've written the following JavaScript code. The HTML file has two canvas elements that are designed to draw ex1_1.png and ex2_1.png and each canvas element has an associated "Next" button. If the first one is clicked it erases the first canvas element and draws ex_2.png. What I want is for the Next button to cycle through all my images, going back to the start when the last image is exceeded. The following JavaScript accomplishes this, except for the part where it cycles back. When it reaches the image with source ex1_3.png (which doesn't exist in the folder), I get a crash, but on the draw() command--which tells me that for whatever reason, it's not cycling the source back to ex1_1.png before attempting to draw.
To the best of my ability to debug this, something is going wrong with the img.onerror part, or how its implemented with the global variable window.indicator. When I cycle through using the next button, the indicator shows true then false if I print the value from within the img.onerror function. But if I print from within the next() function, it never shows false. This sounds like some kind of an issue with the window.indicator keeping its value globally.
// Variable to indicate whether the most recently generated image
// was valid.
window.indicator = true;
// Give the file base-name and index as stored in the local address.
// Return the corresponding Image() object.
function initImg(name, ind) {
var img = new Image();
// The file is local, the image is always of the form
// baseName_i.png
window.indicator = true;
img.src = name + "_" + ind + ".png";
img.onerror = function() {
// Find the appropriate canvas state element, and
// update its state back to 1.
for (i = 0; i < canStates.length; i++) {
var n = canStates[i][0].getAttribute("id");
n = n.split("_")[1];
if (name == canStates[i][0].getAttribute("id")) {
canStates[i][1] = 1;
}
window.indicator = false;
}
};
return img;
}
// Give the canvas context and image objects. Draw the image to
// the context, no return value.
function draw(ctx, img) {
// Check that the image is loaded before writing. Keep
// checking every 50 milliseconds.
if (!img.complete) {
setTimeout( function() {
draw(ctx, img);
}, 50);
}
// Clear the current image and draw the new.
ctx.clearRect(0,0, 200,200);
ctx.drawImage(img, 0,0, 200,200);
}
// Give the string canvas id and string base-name, create the
// canvas object and draw the first image to the canvas.
function slideShow(canId, name) {
var can = document.getElementById(canId);
can.width = 300;
can.height = 300;
var ctx = can.getContext('2d');
var img = initImg(name, 1);
draw(ctx, img);
}
// Next button function. Give the name of the canvas, draw the
// next image or cycle to the start.
function next(button) {
var name = button.getAttribute("name");
// Find the appropriate canvas element.
for (i = 0; i < canStates.length; i++) {
var id = canStates[i][0].getAttribute("id");
id = id.split("_")[1];
if (id == name) {
// Use the states to produce an image, and update the
// states.
canStates[i][1] += 1;
var img = initImg(name, canStates[i][1]);
if (!window.indicator) {
img = initImg(name, 1);
}
// Draw to the canvas.
draw(canStates[i][0].getContext('2d'),img);
}
}
}
// Create a global variable tracking all states of "Next" buttons.
// Stored as a list, each element is a list, the left coordinate is
// a canvas and the right coordinate is its state (image index).
// Also initialize all slide shows.
// The variable r stores the canvases and states, initialized
// outside the function in order to pass-by-reference so as to act
// as a global variable.
var canStates = new Array();
window.onload = function() {
var cans = document.getElementsByTagName("canvas");
for (i=0; i < cans.length; i++) {
var c = cans[i];
var n = c.getAttribute("id").split("_")[1];
slideShow("can_"+n, n);
}
for (i = 0; i < cans.length; i++) {
canStates[i] = [cans[i],1];
}
}
I could switch strategies completely here. I've heard that PHP is a decent way to server-side look at the files in a directory, and I could use that, but I don't know how to make a PHP script execute when a browser is loaded, or how to take its results and hand them over to the JavaScript file.

Problems getting context while using a javascript library to handle image sizing

I'm using a good library on here to handle some large images coming in through the iphone camera in order to avoid the whole subsampling drama here.
My draw code:
function imageLoaded(img, frontCamera) {
element = document.getElementById("canvas1");
var mpImg= new MegaPixImage(img);
// read the width and height of the canvas- scaled down
width = element.width; //188 94x2
height = element.height; //125
//used for side by side comparison of images
w2 = width / 2;
// stamp the image on the left of the canvas
if (frontCamera) {
mpImg.render(element, {maxWidth:94, maxHeight:125});} else{
mpImg.render(element, {maxWidth:94, maxHeight:125});}
//at this point, i want to grab the imageData drawn to the canvas using
//MegaPixImage and continue to do some more image processing, which normally
//would happen by declaring ctx=element.getContext("2d");
//more stuff here
}
The image is drawing fine,...but I cannot seem to find a way of then doing image processing on that image subsequently. How would I get a new context after having drawn that image on the canvas?
Maybe I would either have to run further image processing from within that library so I have context access or strip the context drawing out of the library.
Thanks for the help!

I had a similar issue, and actually found a helpful function to detect subsampling and only use MegaPixImage when subsampling was found.
In my case, for local file reading (iPhone camera, in your case), I called a handleFileSelect function when a <input type="file"> value is changed (i.e. when a file is selected to populate this input). Inside this function, I called a general populateImage JS function that draws the image data to the canvas.
Here's the handleFileSelect function and input binding:
$("#my_file_input").bind('change', function (event) {
handleFileSelect(event);
});
function handleFileSelect(event) {
var reader,
tmp,
file = event.target.files[0];
try {
reader = new FileReader();
reader.onload = function (e) {
tmp = e.target.result.toString();
// In my case, some image data (from Androids, mostly) didn't contain necessary image data, so I added it in
if (tmp.search("image/jpeg") === -1 && tmp.search("data:base64") !== -1) {
tmp = tmp.replace("data:", "data:image/jpeg;");
}
populateImage(tmp);
};
reader.onerror = function (err) {
// Handle error as you need
};
reader.readAsDataURL(file);
} catch (error) {
// Handle error as you need
}
}
Then, my populateImage function (called in the reader.onload function above):
function populateImage(imageURL) {
var tmpImage = new Image();
$(tmpImage).load(function () {
var mpImg, mpImgData;
// If subsampling found, render using MegaPixImage fix, grab image data, and re-populate so we can use non-subsampled image.
// Note: imageCanvas is my canvas element.
if (detectSubsampling(this)) {
mpImg = new MegaPixImage(this);
mpImg.render(imageCanvas, {maxWidth: 94, maxHeight: 125});
mpImgData = imageCanvas.toDataURL("image/jpg");
populateImage(mpImgData);
return;
}
// Insert regular code to draw image to the canvas
// Note: ctx is my canvas element's context
ctx.drawImage(tmpImage, 0, 0, 94, 125); // Or whatever x/y/width/height values you need
});
$(tmpImage).error(function (event) {
// Handle error as you need
});
tmpImage.src = imageURL;
}
And last but not least, the detectSubsampling function. Note that this method was found from another source and isn't my own.
function detectSubsampling(img) {
var iw = img.naturalWidth,
ih = img.naturalHeight,
ssCanvas,
ssCTX;
if (iw * ih > 1024 * 1024) { // Subsampling may happen over megapixel image
ssCanvas = document.createElement('canvas');
ssCanvas.width = ssCanvas.height = 1;
ssCTX = ssCanvas.getContext('2d');
ssCTX.drawImage(img, -iw + 1, 0);
// Subsampled image becomes half smaller in rendering size.
// Check alpha channel value to confirm image is covering edge pixel or not.
// If alpha value is 0 image is not covering, hence subsampled.
return ssCTX.getImageData(0, 0, 1, 1).data[3] === 0;
}
return false;
}
This may be more than you bargained for, but like I said, I ran into a similar issue and this solution proved to work across all browsers/devices that were canvas supported.
Hope it helps!