Develop Reference

concatenating two tf.data.Dataset gives "this.lastRead.then is not a function" error in tensorflow.js

I am trying to append a new row in tf.data.Dataset in tensorflow.js, and after searching i figured that the way to do this is by turning the new row which is originally a json object into a dataset object then concatenate it with the pervious one, but i ended up facing this error
"this.lastRead.then is not a function"
I tried to debug it so i tried to concatenate the same dataset with it self and faced the same problem:
csvUrl = 'https://storage.googleapis.com/tfjs-examples/multivariate-linear-regression/data/boston-housing-train.csv';
const a = tf.data.csv(
csvUrl, {
columnConfigs: {
medv: {
isLabel: true
}
}
});
const b = a.concatenate(a);
await b.forEachAsync(e => console.log(e));
and got the same error message, can you help me out?

Currently there is a bug that prevents to concatenate datasets. Until it is deployed in a new version, the dataset iterators can be utilized to do a concatenation. Here is an example:
const csvUrl =
'https://storage.googleapis.com/tfjs-examples/multivariate-linear-regression/data/boston-housing-train.csv';
async function run() {
const csvDataset = tf.data.csv(
csvUrl, {
columnConfigs: {
medv: {
isLabel: true
}
}
});
const numOfFeatures = (await csvDataset.columnNames()).length - 1;
// Prepare the Dataset for training.
const flattenedDataset =
csvDataset
.map(({xs, ys}) =>
{
// Convert xs(features) and ys(labels) from object form (keyed by
// column name) to array form.
return {xs:Object.values(xs), ys:Object.values(ys)};
})
//.batch(10);
const it = await flattenedDataset.iterator()
const it2 = await flattenedDataset.iterator()
const xs = []
const ys = []
// read only the data for the first 5 rows
// all the data need not to be read once
// since it will consume a lot of memory
for (let i = 0; i < 5; i++) {
let e = await it.next()
let f = await it2.next()
xs.push(e.value.xs.concat(f.value.xs))
ys.push(e.value.ys.concat(f.value.ys))
}
const features = tf.tensor(xs)
const labels = tf.tensor(ys)
console.log(features.shape)
console.log(labels.shape)
}
await run();
The only thing to keep in mind is that the above will load all the tensors in memory which cannot be ideal depending on the size of the dataset. A data generator can be used to reduce the memory footprint. here is a very detailed answer to show how to do it

apparently this was a bug in the tensorflow.js library. The bug has been fixed in this pull request: https://github.com/tensorflow/tfjs/pull/5444
Thank you.
Edit
The pull request has been merged and now it is in the version 3.9.0

Netlify Serverless Function returning 404

I am trying to set up a simple serverless function on Netlify just to test out usage of environment variables. I have defined the following two environment variables in Netlify for my site:
Variable Name
Value
ALPHABET_SEPARATION
2
CHARS_BETWEEN
3
I have also updated my functions directory as follows:
Functions directory: myfunctions
I am using continuous deployment from github. As I do not know the use of npm at present and finding it convenient to directly test the production deploy, I have defined a subdirectory called myfunctions inside my root directory and have placed my javascript file containing the "serverless" function inside it on my local machine. I have built in logic so that the "serverless" function gets called only when a "netlify" flag is set, otherwise, an alternate function gets executed client-side. Basically it works as follows:
const deploy = "netlify" //Possible valid values are "local" and "netlify"
async function postRandomString() {
const stringToUpdate = "THISISATESTSTRING"
var stringToPost = "DUMMYINITIALVALUE";
if (deploy === "local") {
stringToPost = updateString(stringToUpdate); //updateString is a function defined elsewhere and executes client-side;
}
else if (deploy === "netlify") {
const config = {
method: 'GET',
headers: {
'Accept': 'application/json',
}
};
const res = await fetch(`myfunctions/serverUpdateString?input=${stringToUpdate}`, config);
const data = await res.json();
stringToPost = data.retVal;
console.log(data.retVal);
}
else {
stringToPost = "##ERROR##";
}
postString(stringToPost); //postString is a function defined elsewhere and executes client-side;
}
The serverless function file serverUpdateString.js is coded as follows (it basically sets a character at a certain position (determined by CHARS_BETWEEN) in the string to an alphabetical character which is a certain number (determined by ALPHABET_SEPARATION) of places in the alphabet after the first character of the string (don't ask why - the point is that it never even receives/handles the request):
exports.handler = async function (event) {
const { CHARS_BETWEEN, ALPHABET_SEPARATION } = process.env;
const charsBetween = CHARS_BETWEEN;
const alphabetSeparation = ALPHABET_SEPARATION;
const initString = event.queryStringParameters.input;
const rootUnicode = initString.charCodeAt(0);
const finalUnicode = "A".charCodeAt(0) + (rootUnicode - "A".charCodeAt(0) + alphabetSeparation) % 26;
const finalChar = String.fromCharCode(finalUnicode);
const stringArray = initString.split("");
stringArray[charsBetween + 1] = finalChar;
const stringToReturn = stringArray.join("");
const response = {
statusCode: 200,
retVal: stringToReturn,
}
return JSON.stringify(response);
}
When I run it, I get a 404 error for the GET request:
In the above image, script.js:43 is the line const res = await fetch(myfunctions/serverUpdateString?input=ATESTSTRIN, config); in the calling file, as shown in the first code block above.
What am I doing incorrectly? Surely Netlify should be able to pick up the serverless function file given that I have specified the folder alright and have placed it at the right place in the directory structure? I have given the whole code for completeness but the problem seems quite elementary. Look forward to your help, thanks.

I got assistance from Netlify forums. Basically the following changes needed to be made:
The fetch request -- line 43 in the calling code (script.js) -- needed to be changed to
const res = await fetch(`https://netlifytestserverless.netlify.app/.netlify/functions/serverUpdateString?input=${stringToUpdate}`, config);
The return statement in the lambda function needed to be changed to:
const response = {
statusCode: 200,
body: JSON.stringify(stringToReturn),
}
Other minor changes such as using parseInt with the environment variables.
The code works now.

At an object's instantiation time how does one handle best the asynchronous initialization of one of its properties?

I've never created a Javascript module/library before so this is a bit new to me so apologizes for my lack of knowing what to google.
I'm creating a library that will hold information from a URL that is provided by a user. I want to parse the URL's path (the part that comes after the domain) as well as retain a header value that's provided by the URL's response.
It's basic but here's what I have so far:
function Link(someURL) {
this.url = someURL;
this.urlPath = "";
this.uuid = "";
this.getPath = function (someURL) {
// do regexp parsing and return everything after the domain
};
this.getUUID = function (someURL) {
// fetch the URL and return what is in the response's "uuid" header
}
}
Ideally, I'd the module to automatically get all the information upon construction:
var foo = new Link("http://httpbin.org/response-headers?uuid=36d09ff2-4b27-411a-9155-e82210a100c3")
console.log(foo.urlPath); // should return "uuid"
console.log(foo.uuid); // should return the contents in the "uuid" header in the response
How do I ensure the this.urlPath and this.uuid properties get initialized along with this.url? Ideally, I'd only fetch the URL once (to prevent rate limiting by the target server).

After a lot of trial and error, I ended up doing something more like this:
class Link {
constructor (url_in) {
const re = RegExp("^https://somedomain.com\/(.*)$");
this.url = re[0];
this.linkPath = re[1];
}
async getUUID() {
const res = await fetch("https://fakedomain.com/getUUID?secret=" + this.linkPath);
this.uuid = res.uuid;
}
async getJSON() {
const res = await fetch("https://fakedomain.com/getJSON?uuid=" + this.uuid);
this.json = await res.json();
}
async initialize() {
await this.getUUID();
await this.getJSON();
}
}
const someLinkData = new Link("https://reallydumbdomain.com/2020/10/4/blog");
someLinkData.initialize()
.then(function() {
console.log(this.json); // this now works
});
I think a future iteration of this will require me to send a promise with the initialize function but for now, this works.

Websockets in Sapper

I have a readable store in Svelte that looks like this:
const state = {};
export const channels = readable(state, set => {
let st = state;
let socket = new WebSocket("ws://127.0.0.1:5999");
socket.onmessage = function (event) {
var datastr = event.data.split(':');
st[datastr[0]].value = datastr[1];
st[datastr[0]].timestamp = Date.now();
set(st)
};
return () => {
socket.close()
}
});
When I import it to my Svelte App works. But if I put that App.svelte as my index.svelte running on Sapper, it doesnt work at first. It says error 500 websocket is not defined. Once I reload the page in the browser start to work...
I have try to parse a function that creates the store instead:
export const getChannel = () => {
// here my store
return {...store}
}
and then creating the store inside a onMount() like this:
onMount( ()=> {
const channel = getChannel();
});
But doesnt seem to do the trick... What do I miss?
Note: If a just replace the store by a simple writable, and create the websocket onMount(), it works without any problem. I just only wanted to put all the communication inside the store as a readable...

In Sapper, code in components (or imported into components) is executed in Node during server-side rendering unless it's put inside onMount (which doesn't run on the server, because there's no 'mounting' happening) or an if (process.browser) {...} block, or something equivalent.
That includes things like references to $channels causing channels.subscribe(...) to be called during initialisation.
Since there's no WebSocket global in Node, creating that subscription will fail. The simplest workaround is probably a simple feature check:
const state = {};
export const channels = readable(state, (set) => {
if (typeof WebSocket === 'undefined') return;
let st = state;
let socket = new WebSocket("ws://127.0.0.1:5999");
socket.onmessage = function (event) {
var datastr = event.data.split(":");
st[datastr[0]].value = datastr[1];
st[datastr[0]].timestamp = Date.now();
set(st);
};
return () => {
socket.close();
};
});

How would I assign a Javascript variable to another Javascript variable? (a little confusing, i know)

So, I have had a problem recently!
I was trying to assign a value to a variables value (a little confusing, i know). I was trying to make a library system for my Discord bot, which uses JavaScript (Node.js)!
So, here's a part of the code that I was struggling with:
flist.forEach(item1 => {
liblist.forEach(item => {
eval(item1 = require(item));
});
});
OK, so basically item1 has to be replaced with a library file's name (if a file is named cmdh, I use that as a variable name to assign require(item) to it)
Edit: the item1 is the file name, by the way.
Edit 2: Here's a part of the file util.js:
const Discord = require("discord.js");
const fs = require("fs");
function genEmbed(title, desc, col) {
return new Discord.MessageEmbed().setTitle(title).setDescription(desc).setColor(col);
};
function log(inp) {
console.log(`[UTIL] ${inp}`);
};
function loadDef() {
log("Loading libraries...");
dirlist = [];
liblist = [];
flist = [];
fs.readdir("./libraries", (err, dirs) => {
if(err) console.error(err);
dirs.forEach(dir => {
dirlist.push("./libraries/" + dir);
});
dirlist.forEach(dir => {
fs.readdir(dir, (err, files) => {
if(err) console.error(err);
files.forEach(file => {
if(file.includes(".")) {
liblist.push(require(dir + "/" + file));
filename = file.split(".")[0];
flist.push(filename);
} else {
log(`${file} is a directory, ignoring...`)
};
});
});
});
});
flist.forEach(item1 => {
liblist.forEach(item => {
eval(item1 = require(item));
});
});
log("Libraries loaded!");
};
module.exports = {
genEmbed,
loadDef
};

If you are trying to assign a new value for each element in an array and return a new array with the updated values, you should use map instead of forEach.
If you want to return an array with the required libraries from liblist and store the result in flist, your code should look as follows:
flist = liblist.map(item => require(item));
Essentially, map takes each value from an array, applies a function to it (in your case, require), and then returns a new array with the results of the function.
It should be noted that you also need to replace fs.readdir with fs.readdirSync, otherwise liblist will be empty when the code is called.
After our discussion on the chat I understood that you want to set the variables as globals in your main file. However, this is a bit hackish. To do that you need to modify the global object.
Note: I would advise you to statically require via const libraryName = require('./libraries/fileName.js') instead, as this way of dynamically defining libraries has no advantage in this case and only makes the code more complicated. Dynamically requiring libraries only makes sense when you, for example, add listeners via your libraries, but then you do not need to store them as globals.
Your full code for the utils file should now look as follows:
const Discord = require("discord.js");
const fs = require("fs");
function genEmbed(title, desc, col) {
return new Discord.MessageEmbed().setTitle(title).setDescription(desc).setColor(col);
};
function log(inp) {
console.log(`[UTIL] ${inp}`);
};
function loadDef() {
log("Loading libraries...");
libraries = {};
const dirlist = fs.readdirSync("./libraries").map(dir => "./libraries/" + dir)
dirlist.forEach(dir => {
const files = fs.readdirSync(dir)
files.forEach(file => {
if(file.includes(".")) {
filename = file.split(".")[0];
libraries[filename] = require(dir + "/" + file);
} else {
log(`${file} is a directory, ignoring...`)
};
});
});
log("Libraries loaded!");
return libraries;
};
module.exports = {
genEmbed,
loadDef
};
Now you still need to modify the code in your main file, as follows:
const libraries = loadDef()
global = Object.assign(global, libraries) // assign libraries to global variables