How to let Pix2Pix with p5.js run? - javascript

I would like to try the ml5.js Pix2Pix example for p5.js. If I just copy the code, update the paths, and try to let it run on my local server, it doesn't work.
Same here:
// Copyright (c) 2019 ml5
//
// This software is released under the MIT License.
// https://opensource.org/licenses/MIT
/* ===
ml5 Example
Pix2pix Edges2Pikachu example with p5.js using callback functions
This uses a pre-trained model on Pikachu images
For more models see: https://github.com/ml5js/ml5-data-and-training/tree/master/models/pix2pix
=== */
// The pre-trained Edges2Pikachu model is trained on 256x256 images
// So the input images can only be 256x256 or 512x512, or multiple of 256
const SIZE = 256;
let inputImg, inputCanvas, outputContainer, statusMsg, pix2pix, clearBtn, transferBtn, modelReady = false,
isTransfering = false;
function setup() {
// Create a canvas
inputCanvas = createCanvas(SIZE, SIZE);
inputCanvas.class('border-box').parent('canvasContainer');
// Display initial input image
inputImg = loadImage('https://ml5js.github.io/ml5-examples/javascript/Pix2Pix/Pix2Pix_promise/images/input.png', drawImage);
// Selcect output div container
outputContainer = select('#output');
statusMsg = select('#status');
// Select 'transfer' button html element
transferBtn = select('#transferBtn');
// Select 'clear' button html element
clearBtn = select('#clearBtn');
// Attach a mousePressed event to the 'clear' button
clearBtn.mousePressed(function() {
clearCanvas();
});
// Set stroke to black
stroke(0);
pixelDensity(1);
// Create a pix2pix method with a pre-trained model
pix2pix = ml5.pix2pix('https://github.com/ml5js/ml5-library/blob/main/examples/p5js/Pix2Pix/Pix2Pix_callback/models/edges2pikachu.pict', modelLoaded);
}
// Draw on the canvas when mouse is pressed
function draw() {
if (mouseIsPressed) {
line(mouseX, mouseY, pmouseX, pmouseY);
}
}
// Whenever mouse is released, transfer the current image if the model is loaded and it's not in the process of another transformation
function mouseReleased() {
if (modelReady && !isTransfering) {
transfer()
}
}
// A function to be called when the models have loaded
function modelLoaded() {
// Show 'Model Loaded!' message
statusMsg.html('Model Loaded!');
// Set modelReady to true
modelReady = true;
// Call transfer function after the model is loaded
transfer();
// Attach a mousePressed event to the transfer button
transferBtn.mousePressed(function() {
transfer();
});
}
// Draw the input image to the canvas
function drawImage() {
image(inputImg, 0, 0);
}
// Clear the canvas
function clearCanvas() {
background(255);
}
function transfer() {
// Set isTransfering to true
isTransfering = true;
// Update status message
statusMsg.html('Applying Style Transfer...!');
// Select canvas DOM element
const canvasElement = select('canvas').elt;
// Apply pix2pix transformation
pix2pix.transfer(canvasElement, function(err, result) {
if (err) {
console.log(err);
}
if (result && result.src) {
// Set isTransfering back to false
isTransfering = false;
// Clear output container
outputContainer.html('');
// Create an image based result
createImg(result.src).class('border-box').parent('output');
// Show 'Done!' message
statusMsg.html('Done!');
}
});
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
<script src="https://unpkg.com/ml5#latest/dist/ml5.min.js" type="text/javascript"></script>
<h1>Pix2Pix Edges2Pichaku Example</h1>
<p>1. Wait until the model is loaded</p>
<p>2. Press your mouse to draw a Pikachu on the left side of the canvas.</p>
<p>3. A colored Pikachu image will automatically appear on the right side of the canvas in ~2 seconds. You could also click the "Transfer" button to generate an new image.</p>
<p>4. You could click the "Clear" button to clear the canvas and draw again.</p>
<p id="status">Loading Model... Please wait...</p>
<div class="flex">
<div>
<div id="canvasContainer"></div>
<div id="btnContainer" class="flex flex-space-between">
<button id="clearBtn">Clear</button><br />
<button id="transferBtn" class="btn">Transfer</button>
</div>
</div>
<div id="transferContainer">
</div>
<div id="output"></div>
</div>
Here would be also a jsFiddle:
https://jsfiddle.net/L6oaydrm/
Has anyone an idea how to let it run? Would be very thankful.

I think I was able to get the 'callback' example to work locally with some tinkering:
Download files from the example: https://github.com/ml5js/ml5-library/tree/main/examples/p5js/Pix2Pix/Pix2Pix_callback
Adjust the index.html to load ml5.min.js from the unpkg.com URL in your code.
Create a new function:
function startTransfer(){
// Create a pix2pix method with a pre-trained model
pix2pix = ml5.pix2pix('./models/edges2pikachu.pict', modelLoaded);
}
Replace all calls to transfer() except the first one in modelLoaded() with startTransfer().
Start a simple local web server; for me: python -m http.server worked.
The example appeared to work. I could draw on the canvas, and the ML model would redraw the Pikachu image factoring in the new lines I added. Note, sometimes the initial transfer is run before the template image (input.png) is loaded, and the result is a garbled yellow / red pixels; clicking 'Transfer' fixes this.
Basically, it always will reload the model into the ml5 library; I don't know of the performance implications of this, but it was redrawing relatively quickly in my browser. The file will be cached in the browser, so that isn't a concern, but I'm not sure of the internals of the ml5.js lib and what ml5.pix2pix(...) does.
I've put my revised code (including some other tweaks to the JS) up on https://jsfiddle.net/lecrte/jvohcw8r/16/ ... but it won't work there because the assets aren't available relative to the HTML file, and we can't load the edges2pikachu.pict direct from github.com due to CORS issues.

Related

Dom doesn't update right away when using convertToBlob on an Offscreen canvas

I have an application that needs to generate a couple thousand images. The way that I'm doing that is with a set of preloaded pngs (acting as transparent layers) and an offscreen canvas. I draw the images onto the canvas, convert it to a blob, and then write the image to a div using a custom class called Images.
I want to show a loading bar and clear old images, first, but there is a 3-5 second delay before the dom updates even though the "empty()" and "show()" code is at the beginning of the click request.
Is there something I'm doing wrong with regard to the asynchrony or promises that is causing the dom to not update immediately?
Here's some of the code:
// Generate Images
$("#generate_images").click(function(){
// Show loading bar
$("#progress_bar .progress-bar").css("width", "0%");
$("#progress_bar").show(0);
// Show loading spinner
$("#loading").show(0);
$("#images").empty();
console.log("Generating Images");
$.each(images, function(id, image){
// Sort traits (png layers)
var traits = Object.values(image.traits).sort((a, b) => {
return a.z_index - b.z_index;
});
images[id].layers = [];
$.each(traits, function(trait_idx, trait){
images[id].layers.push(preloaded_images[trait.variant_id])
});
});
// Create canvas
var canvas = new OffscreenCanvas(1200, 1200);
var context = canvas.getContext("2d");
console.log("Generating canvases.");
$.each(images, function(id, image){
// Clear the canvas
context.clearRect(0, 0, canvas.width, canvas.height);
// Draw each image layer
$.each(images.layers, function(src, layer){
context.drawImage(layer, 0, 0);
});
// Add imageData to screen
canvas.convertToBlob().then(function(blob) {
// Do something with the blob like render to the screen
});
});
});
Any thoughts on making this more efficient would also be appreciated.
Generating a thousand images is not something a browser is necessarily meant to do, but you need to give the DOM a chance to update so add a timeout
Collect what you want to do in an array
Do not loop, but instead do
function generate() {
if (arrayCounter >= array.length) return
canvas.convertToBlob().then(function(blob) {
if (success) {
arrayCounter++
generate()
}
}
}
collect()
generate()

Canvas Element In Closure Is Being Changed Unexpectedly

I am working on creating a webpage that will allow users to add color filters to images they upload. In order to do this I wish to keep an original copy of the image stored in a canvas. Then whenever a user changes one of the color values, using a range slider I want to copy the original image into a new canvas edit it and then plot the new image.
The reason I don't want to apply changes directly to the original image is that they can be difficult to reverse. For example if the user set the blue channel of an image to 0 then unless I have the original image the user would not be able to undo that effect.
My current solution was to create a closure holding a canvas element. This closure is initialized when the user uploads their photo. The code for my closure is below
var scaleCanvas = (function() {
var scaledCvs = document.createElement('canvas'); //Creates a canvas object
var scaledCtx = scaledCvs.getContext("2d"); //Gets canvas context
// All the functions in this section can access scaledCanvas
return {
init: function(cvs) {
scaledCvs = cvs;
console.log("Image Stored")
}, //close init
getImg: function() {
return (scaledCvs);
} //Close getImg
}; //close return
})();
Once the canvas is initialized it can be accessed using the scaleCanvas.getImg() function. An example of this would be during my function adjustFilters() which is called by a range slider element as below
<div class="slidecontainer">
<label>Blue Channel</label>
<input type="range" min="0" max="150" value="100" class="slider" id="blue" onchange="adjustFilters();"/>
</div>
Once the blue slider is changed I load the canvas that I want to use to display the image. Then I load my original image using scaleCanvas.getImg(). Then I apply the effects I am interested in. Then I draw the image. What is strange though is that it appears the scaledCvs variable in my closure is being changed. I say this because if I select a value for "blue" as 0 then it is impossible for me to recover the original blue channel. This should not happen unless the scaledCvs in my closure is being altered somehow.
function adjustFilters() {
var canvas = document.getElementById("dispImg"); //gets the canvas element
canvas.style.display = "block"; //Makes the canvas visible
var context = canvas.getContext("2d"); //Using 2D context
// look up the size the canvas is being displayed at
const width = canvas.clientWidth;
const height = canvas.clientHeight;
var imgCV = scaleCanvas.getImg(); //Gets the original canvas image
var imgCtx = imgCV.getContext('2d');
var blue = document.getElementById("blue").value / 100; //Gets value of the blue slider
// Adjusts blue channel
adjustColor(imgCtx, imgCV.height, imgCV.width, "blue", blue)
// Draws the Image
context.clearRect(0, 0, canvas.width, canvas.height);
context.drawImage(imgCV, 0, 0);
}
I am looking for suggestions on how to get my code to operate as expected. i.e I need the variable scaledCvs in my closure to not be adjusted except for when the init function is called.
Thank You
The reason for the observed behavior is that canvas elements are passed by reference therefore
init: function(cvs) {
scaledCvs = cvs;
console.log("Image Stored")
}, //close init
Does not create a copy of cvs it just creates a reference to it. In order to create a copy of the actual canvas you would need to follow the protocol below
init: function(cvs) {
//Create a Copy of the provided canvas (cvs)
scaledCvs.width=cvs.width;
scaledCvs.height=cvs.height;
scaledCtx.drawImage(cvs,0,0);
}, //close init
For the entire code to work properly you would also need to modify getImage() so that it is not passed a reference of scaledCvs. The final closure definition would be
var scaleCanvas = (function() {
var scaledCvs=document.createElement('canvas'); //Creates a canvas object
var scaledCtx=scaledCvs.getContext("2d"); //Gets canvas context
// All the functions in this section can access scaledCanvas
return {
init: function(cvs) {
//Create a Copy of the provided canvas (cvs)
scaledCvs.width=cvs.width;
scaledCvs.height=cvs.height;
scaledCtx.drawImage(cvs,0,0);
}, //close init
getImg: function() {
//Since canvas are passed by reference in order to protect the saved canvas I have to create a copy before passing it out
var cpCanvas = document.createElement('canvas');
var context = cpCanvas.getContext("2d");
cpCanvas.height = scaledCvs.height;
cpCanvas.width = scaledCvs.width;
context.drawImage(scaledCvs,0,0);
return(cpCanvas);
} //Close getImg
}; //close return
})();
All other code can remain the same.

Canvas drawImage without using load event

As an introduction to canvas and the way they worked, I wanted to create a small quizz. Using only colors, I haven't any very strong problems that I can't solve by using google.
Now I want to show a background picture to the quizz but more precisely, I want that any question have a different picture as a background.
So I did this:
My main loop is a event binding on a function with click working like that:
qcm.canvas.addEventListener('click', function(evt) {
main(qcm, evt);
}, false);
Then, my main loop will use values in qcm object to know what to draw. In any case, I'll clear my canvas and then drawing back the canvas skeletton (which at the moment contain only the background picture assignement).
This is my clear canvas function, using only a clearRect.
function clearCanvas(qcm)
{
//Suppression du contenu du canvas
qcm.context.clearRect(0, 0, qcm.canvas.scrollWidth, qcm.canvas.scrollHeight);
// Rechargement du décor du canvas
initCanvas(qcm);
}
the InitCanvas function which should draw the background is this one:
function initCanvas(qcm)
{
//Background color
//qcm.context.fillStyle = QcmConfiguration.Canvas_BackColor;
//qcm.context.fillRect(0,0,qcm.canvas.scrollWidth, qcm.canvas.scrollHeight);
if (qcm.backgroundImage != undefined)
{
var background = new Image();
background.addEventListener('load', function()
{
qcm.context.drawImage(background, 0, 0, qcm.canvas.scrollWidth, qcm.canvas.scrollHeight);
}, false);
background.src = qcm.backgroundImage;
}
}
When my canvas end "init" phase, it the draw the frame I want to show, (question, answer, score ... depending on the qcm status).
For example, here is a part of the main loop function:
clearCanvas(qcm);
log(QcmLogsType.Initialisation, "Chargement du start panel");
drawStartPanel(qcm);
Where my drawStartPanel is a function which just draw a "Play" button:
function drawStartPanel(qcm)
{
// Dessin du rectangle colorée pour le bouton play
qcm.context.fillStyle = QcmConfiguration.Start_BackButtonColor;
qcm.context.fillRect((qcm.canvas.scrollWidth / 2) - 25, (qcm.canvas.scrollHeight / 2) - 25, 50, 50);
// Dessin du triangle "play"
qcm.context.fillStyle = QcmConfiguration.Start_ForeButtonColor;
qcm.context.beginPath();
qcm.context.moveTo((qcm.canvas.scrollWidth / 2) - 10, (qcm.canvas.scrollHeight / 2) - 10);
qcm.context.lineTo((qcm.canvas.scrollWidth / 2) - 10, (qcm.canvas.scrollHeight / 2) + 10);
qcm.context.lineTo((qcm.canvas.scrollWidth / 2) + 12, qcm.canvas.scrollHeight / 2);
qcm.context.closePath();
qcm.context.fill();
}
But when I execute my code, my start button seems to be "under" the picture, which mean that the picture load event seems to execute after the drawStartpanel function while the call is before...
This way I think that the main option is to put the picture as display:none in the html code and then draw it on the canvas, but I don't really like this idea, because I later want to try to do something which will go get pictures on a database, which will not allow to do something like that.
So I'm looking:
Either for a way to force the load event manually
Or a way to not use the load event to show my picture
(I didn't post the entire code as it's 800 lines but will if needed :) )
Simply move your drawStartPanel function in the load callback like this :
function initCanvas(qcm, callback) {
// Background color
// qcm.context.fillStyle = QcmConfiguration.Canvas_BackColor;
// qcm.context.fillRect(0,0,qcm.canvas.scrollWidth, qcm.canvas.scrollHeight);
if (qcm.backgroundImage !== 'undefined') {
var background = new window.Image();
background.onload = function () {
qcm.context.drawImage(background, 0, 0, qcm.canvas.scrollWidth, qcm.canvas.scrollHeight);
// Here call the start button drawing function assuming it always takes the qcm object as an argument
callback(qcm);
}
background.src = qcm.backgroundImage;
}
}
Calling it like this :
clearCanvas(qcm);
log(QcmLogsType.Initialisation, "Chargement du start panel");
initCanvas(qcm, drawStartPanel);
Going further
Another way of getting the drawings synchronously applied is to use Javascript Promises

How to make HTML5 Canvas "Press/Click to play" start screen

I've finished the basic mechanic of a game and also done with the end screen. Now I plan to have a png made with Photoshop where the title and instructions are. And when I click/press enter I should be able to start the game like normal.
Been searching for quite sometimes but most of the answer seems to aim towards framework or complex menu.
My program also starts at this point
window.addEventListener('load', function () {
canvas1 = document.getElementById("gameCanvas1");
canvasContext1 = canvas1.getContext("2d");
canvas2 = document.getElementById("gameCanvas2");
canvasContext2 = canvas2.getContext("2d");
...
}
Use a game state manager to hold the current game state function then just listen to the mouse and key events during the splash screen state. The game state just holds the function you need to call once per frame to run the game or slash screen or end game screen.
function splashIO (event) { // function to start the game when IO is correct
// check for the correct events
if(event.type === "click" || (event.type === "keydown" && event.code === "Enter")){
// remove events
canvas.removeEventListener("click",splashIO);
canvas.removeEventListener("keydown",splashIO);
gameStates.current = gameStates.startGame;
}
}
// holds the game state and game state functions
const gameStates = {
current : undefined,
splash () { // display splash ===================================
// display splash and wait for new state
},
setupSplash () { // setup splash screen ==========================
canvas.addEventListener("click", splashIO);
canvas.addEventListener("keydown", splashIO);
gameStates.current = gameStates.splash();
gameStates.current(); // call the first frame
},
startGame () { // setup game =====================================
gameStates.current = gameStates.game(); //set up state function
gameStates.current(); // call the first frame
},
game () { // plays the main game ===============================
// play game
}
}
// main animation loop
function mainLoop () {
gameStates.current(); // run current game state
requestAnimationFrame(mainLoop);
}
gameStates.current = gameStates.setupSplash; // set current state to splash screen
// wait for page to load then start the animation
window.addEventListener('load', function () {
requestAnimationFrame(mainLoop); // start the animation
}

Save canvas to image via toDataURL failed

I create a test code below and you can manipulate it on Jsfiddle:
http://jsfiddle.net/Stallman41/57hvX/31/
HTML:
<canvas id="test_canvas" style="background-color : #FFFF00" ; width="500px"
; height="340px"></canvas>
<br>
<button id="test_put_btn">Put an image</button>
<br>
<button id="save_dataURL">Save to dataURL</button>
<br>
<button id="draw_back">Final step: draw 3 images back.</button>
<br>
<img id="first_img"; width="100px" ; height="100px" ;></img>
<img id="second_img"; width="100px" ; height="100px" ></img>
<img id="third_img"; width="100px" ; height="100px" ;></img>
Javascript:
var drawing_plate;
var context;
var dataURL_arr = new Array();
$(document).ready(function () {
drawing_plate = document.getElementById("test_canvas");
context = drawing_plate.getContext('2d');
$("#test_canvas").bind("mousedown", Touch_Start);
$("#test_canvas").bind("mousemove", Touch_Move);
$("#test_canvas").bind("mouseup", Touch_End);
}); //document ready.
function Touch_Start(event) {
event.preventDefault();
touch = event;
touch_x = touch.pageX;
touch_y = touch.pageY;
line_start_x = touch.pageX - 0;
line_start_y = touch.pageY - 0;
context.beginPath();
context.moveTo(line_start_x, line_start_y);
}
function Touch_Move(event) {
event.preventDefault();
touch = event; //mouse
line_end_x = touch.pageX - 0;
line_end_y = touch.pageY - 0;
context.lineTo(line_end_x, line_end_y);
context.stroke();
}
$("#test_put_btn").click(function () {
var test_img = new Image();
test_img.src = "http://careerscdn.sstatic.net/careers/gethired/img/careers2- ad-header-so-crop.png";
context.drawImage(test_img, 0, 0);
});
$("#save_dataURL").click(function () {
dataURL_arr.push(drawing_plate.toDataURL("image/png"));
});
$("#draw_back").click(function () {
var f_image= $("#first_img")[0];
var s_image= $("#second_img")[0];
var t_image= $("#third_img")[0];
f_image.onload= function()
{
f_image.src= dataURL_arr[0];
}
f_image.src= dataURL_arr[0];
s_image.onload= function()
{
s_image.src= dataURL_arr[0];
}
s_image.src= dataURL_arr[0];
t_image.onload= function()
{
t_image.src= dataURL_arr[0];
}
t_image.src= dataURL_arr[0];
});
I develop a drawing plate on Android system, saving the drawings to a dataURL string. They can draw something on the canvas and put images on the canvas. And I need to let the users see their drawings on small icons.
I use canvas.toDataURL("image/png") to save the base64 string. And I choose <img> as the small icon container. However, what I got is only the drawings can be shown on the icon, and usually, when I write img.src= canvas.toDataURL("image/png"); the image shows nothing!
I investigate the issue for long time.
1. I think the problem might be the dataURL string is too long?
2. The support of the OS: Android?
The code in Jsfiddle here shows a similar procedure on my Android PhoneGap development.
First , you just draw something on the canvas, and press Press an image, and then Save to dataURL. But you should do the process three times. In this condition, the string array contains the base64 string generated by the drawings and the image.
In the final, you press Final step: draw 3 images back., nothing will be shown on the image icon.
In conclusion:
In my experience, as I write img.src= canvas.toDataURL("image/png"); (no matter the img is an dom element or var img = new Image();). It can't always work: sometimes it works... but sometimes not...(I work on Android 4.0.1, phonegap 1.7.0)
Second, especially if I store lots of base64 strings to an array, assigning them to lots of image DOM element, it definitely fails.
Third, if the user only draw something on the canvas, it can always work.( Except the example code in the Jsfiddle, but it works on my Android system...)
But if he draw an image context.drawImage(~) the image wouldn't show the pic.
Too much confusions...
I need to let the user can view their drawings in small icon, any alternative?
Some References:
1
2
3
I just stumbled across this question.
Click Put an image, then click Save to dataURL, then check your JavaScript console for something like:
SecurityError: DOM Exception 18
It's a browser security feature. Because you've inserted an image from a different domain, it counts as a cross-origin request.
If you eliminate the security error, you can export the canvas to a data URL.
Another thing in your code.
The image you try to draw onto the canvas into your test_put_btn onclick event handler, your image will never show up (or it will sometimes work accidentally) because you don't wait for your image to be loaded to draw it onto the canvas.
You have to handle the "onload" event of your image and draw it into the handler to permit the drawing of your image.
Before your test_img.src statement, you have to put :
test_img.onload = function()
{
context.drawImage(test_img, 0, 0);
};
Plus, the image you try to access is not accessible --> For me it does not work

Categories

Resources