So I followed some tutorial on converting a tensorflow model (downloaded from tensorflow hub) to a tfjs model with binary and json files. When I try to use loadLayersModel() it throws a error. When I try to use loadGraphModel(), it loads and runs but the predictions dont seem to work or be meaningful in anyway. How can I tell which method to load models? I need some direction on how to troubleshoot or some recommended workflow particularly in tensorflowJS. I am using this in React Native with expo project. not sure if that matters.
TensorflowHub : https://tfhub.dev/google/aiy/vision/classifier/food_V1/1
my code:
import * as tf from '#tensorflow/tfjs'
import { bundleResourceIO, decodeJpeg } from '#tensorflow/tfjs-react-native'
import * as FileSystem from 'expo-file-system';
const modelJSON = require('./TFJS/model.json')
const modelWeights = require('./TFJS/group1-shard1of1.bin')
export const loadModel = async () => {
const model = await tf.loadLayersModel(
bundleResourceIO(modelJSON, modelWeights)
).catch((e) => {
console.log("[LOADING ERROR] info:", e)
})
return model
}
export const transformImageToTensor = async (uri)=>{
//read the image as base64
const img64 = await FileSystem.readAsStringAsync(uri, {encoding:FileSystem.EncodingType.Base64})
const imgBuffer = tf.util.encodeString(img64, 'base64').buffer
const raw = new Uint8Array(imgBuffer)
let imgTensor = decodeJpeg(raw)
const scalar = tf.scalar(255)
//resize the image
imgTensor = tf.image.resizeNearestNeighbor(imgTensor, [192, 192])
//normalize; if a normalization layer is in the model, this step can be skipped
const tensorScaled = imgTensor.div(scalar)
//final shape of the rensor
const img = tf.reshape(tensorScaled, [1,192,192,3])
return img
}
export const getPredictions = async (image)=>{
await tf.ready()
const model = await loadModel()
const tensor_image = await transformImageToTensor(image)
// const predictions = await makePredictions(1, model, tensor_image)
const prediction = await model.predict(tensor_image)
console.log(prediction)
console.log(prediction.datasync()[0])
return prediction
}
If its a layers model, you you 'loadLayersModel and if its a graph model, you use 'loadGraphModel - they are NOT interchangable and you CANNOT load a model using different method.
So if it loads using loadGraphModel, it is a graph model - as simple as that.
the predictions dont seem to work or be meaningful in anyway
This does not help - what do you expect and what do you actually get?
const embed = new Discord.MessageEmbed();
got('https://www.reddit.com/r/memes/random/.json')
.then(response => {
const [list] = JSON.parse(response.body);
const [post] = list.data.children;
const permalink = post.data.permalink;
const memeUrl = `https://reddit.com${permalink}`;
const memeImage = post.data.url;
const memeTitle = post.data.title;
const memeUpvotes = post.data.ups;
const memeNumComments = post.data.num_comments;
embed.setTitle(`${memeTitle}`);
embed.setURL(`${memeUrl}`);
embed.setColor('BLACK');
embed.setImage(memeImage);
embed.setFooter(`👍 ${memeUpvotes} | 💬 ${memeNumComments}`);
message.channel.send(embed);
})
Here's the code. What I want it to do is to only show memes with more than 4k upvotes, how can I achieve it?
You may apply a really simple approach , add an if statement
if(parseInt(post.data.ups) > 4000) {
// Code here
}
parseInt() is a function that returns an integral value you can work with, rest seems self explanatory 😄.
I'm running some Node.js code to scrape a website and return some text from this part of the html:
And here's the code I'm using to get it
const fs = require('mz/fs');
const xpath = require('xpath');
const parse5 = require('parse5');
const xmlser = require('xmlserializer');
const dom = require('xmldom').DOMParser;
const axios = require('axios');
(async () => {
const response = await axios.get('https://www.aritzia.com/en/product/sculpt-knit-tank-%28arjun-knit-top%29/66139.html?dwvar_66139_color=17388');
const html = response.data;
const document = parse5.parse(html.toString());
const xhtml = xmlser.serializeToString(document);
const doc = new dom().parseFromString(xhtml);
const select = xpath.useNamespaces({"x": "http://www.w3.org/1999/xhtml"});
const nodes = select("//x:div[contains(#class, 'pdp-product-brand')]/*/text()", doc);
console.log(nodes.length ? nodes[0].nodeValue : nodes.length)
})();
The code above works as expected -- it prints Babaton.
But when I swap out the xpath above for one that includes a instead of * (i.e. //x:div[contains(#class, 'pdp-product-brand')]/a/text()) it instead tells me that nodes.length === 0.
I would expect it to give the same result because the div that it's pointing to does in fact have a child anchor tag (see screenshot above). I'm just confused why it doesn't work with a and was wondering if anybody else knew the answer. Thanks!
im trying to get a larger discord bot of mine to save all custom emojis it grabs from another shard to a cache so I can serve better response times for each shard. To give a theoretical example, my bot spawns 4 shards and only one shard serves the guild that contains all the custom emojis that I want to use across all shards. I am using this function to grab the emojis, but I need to await each one, and it can make my response times up to 15 seconds as there are many emojis I need to grab:
function findEmoji(id) {
const temp = this.emojis.get(id);
if (!temp) return null;
const emoji = Object.assign({}, temp);
if (emoji.guild) emoji.guild = emoji.guild.id;
emoji.require_colons = emoji.requiresColons;
return emoji;
}
async function grabEmoji(emojiID) {
const emojiArray = await client.shard.broadcastEval(`(${findEmoji}).call(this, '${emojiID}')`);
const foundEmoji = emojiArray.find(emoji => emoji);
if (!foundEmoji) return;
const raw = await client.rest.makeRequest('get', Discord.Constants.Endpoints.Guild(foundEmoji.guild).toString(), true);
const guild = new Discord.Guild(client, raw);
const emoji = new Discord.Emoji(guild, foundEmoji);
return await emoji;
}
// then when I send the message, I call the function with the said ID of the emoji I want:
await grabEmoji("530800350656200705");
On the other hand, when I remove await, it will give me listener errors (maxListeners reached) or whatever, and then display "null".
Here is what I have tried, but I havent been able to get it to work.
const emojiMap = new Map();
createMap();
async function createMap() {
let woodenPick = await grabEmoji("601256699629797379"),
stonePick = await grabEmoji("601256431076769803"),
ironPick= await grabEmoji("601257055285673987"),
goldPick = await grabEmoji("601256566670491658"),
diamondPick= await grabEmoji("601256973798998016"),
emeraldPick = await grabEmoji("601256896577404938"),
rubyPick = await grabEmoji("601256312696733696"),
ultimatePick= await grabEmoji("629817042954092545"),
sandstonePick = await grabEmoji("629817043142705152"),
aquaPick = await grabEmoji("629817733902761985"),
techPick = await grabEmoji("502940161085014046"),
stone = await grabEmoji("502940717883064321"),
coal = await grabEmoji("502940528149659649"),
iron =await grabEmoji("502940160942669824"),
gold = await grabEmoji("493801062856392705"),
diamond= await grabEmoji("493805522466766849"),
obsidian =await grabEmoji("493801062671581184"),
emerald = await grabEmoji("630846535025819649"),
ruby =await grabEmoji("502940161001259018"),
lapis = await grabEmoji("502940160988807188"),
redstone= await grabEmoji("632411168601931822"),
silver = await grabEmoji("632413243503149087"),
neonite = await grabEmoji("632413243708801024"),
pulsatingStar= await grabEmoji("632404511759138816"),
sapphire = await grabEmoji("642799734192341013"),
developerBadge = await grabEmoji("642799734209118221"),
staffBadge = await grabEmoji("642799734209118221"),
donatorBadge = await grabEmoji("642799734247129089"),
contributorBadge = await grabEmoji("642799734247129089");
emojiMap.set(['woodenPick', woodenPick])
emojiMap.set(['stonePick', stonePick])
emojiMap.set(['ironPick', ironPick])
emojiMap.set(['goldPick', goldPick])
emojiMap.set(['diamondPick', diamondPick])
emojiMap.set(['emeraldPick', emeraldPick])
emojiMap.set(['rubyPick', rubyPick])
emojiMap.set(['ultimatePick', ultimatePick])
emojiMap.set(['sandstonePick', sandstonePick])
emojiMap.set(['aquaPick', aquaPick])
emojiMap.set(['techPick', techPick])
emojiMap.set(['stone', stone])
emojiMap.set(['coal', coal])
emojiMap.set(['iron', iron])
emojiMap.set(['gold', gold])
emojiMap.set(['diamond', diamond])
emojiMap.set(['obsidian', obsidian])
emojiMap.set(['emerald', emerald])
emojiMap.set(['ruby', ruby])
emojiMap.set(['lapis', lapis])
emojiMap.set(['redstone', redstone])
emojiMap.set(['silver', silver])
emojiMap.set(['neonite', neonite])
emojiMap.set(['pulsatingStar', pulsatingStar])
emojiMap.set(['sapphire', sapphire])
emojiMap.set(['developerBadge', developerBadge])
emojiMap.set(['staffBadge', staffBadge])
emojiMap.set(['donatorBadge', donatorBadge])
emojiMap.set(['contributorBadge', contributorBadge])
}
client.on('message', ... //rest of the code continues for my command handler.
//grab emojis
let woodenPick = emojiMap.get('woodenPick')
let stonePick = emojiMap.get('stonePick')
let ironPick = emojiMap.get('ironPick')
let goldPick = emojiMap.get('goldPick')
let diamondPick = emojiMap.get('diamondPick')
let emeraldPick = emojiMap.get('emeraldPick')
let rubyPick = emojiMap.get('rubyPick')
let ultimatePick = emojiMap.get('ultimatePick')
let sandstonePick = emojiMap.get('sandstonePick')
let aquaPick = emojiMap.get('aquaPick')
let techPick = emojiMap.get('techPick')
let stone = emojiMap.get('stone')
let coal = emojiMap.get('coal')
let iron = emojiMap.get('iron')
let gold = emojiMap.get('gold')
let diamond = emojiMap.get('diamond')
let obsidian = emojiMap.get('obsidian')
let emerald = emojiMap.get('emerald')
let ruby = emojiMap.get('ruby')
let lapis = emojiMap.get('lapis')
let redstone = emojiMap.get('redstone')
let silver = emojiMap.get('silver')
let neonite = emojiMap.get('neonite')
let pulsatingStar = emojiMap.get('pulsatingStar')
let sapphire = emojiMap.get('sapphire')
let developerBadge= emojiMap.get('developerBadge')
let staffBadge = emojiMap.get('staffBadge')
let donatorBadge = emojiMap.get('donatorBadge')
let contributorBadge = emojiMap.get('contributorBadge')
Doing that returns undefined as seen here:
Does anyone have any ideas? I'm directly saving the emoji object to the map thinking I can just use it later.
Getting discord.js's abstraction of an Emoji object in order to display it in a message is extremely unnecessary, but I can't blame you as discord.js tends to nudge its users towards these kinds of practices.
You already know the emoji names and IDs. There is no other information that you need to get from your other shards that your bot doesn't already have. In Discord messages, custom emoji are represented like this:
Custom Emoji - <:NAME:ID> -> <:mmLol:216154654256398347>
Custom Emoji (Animated) - <a:NAME:ID> -> <a:b1nzy:392938283556143104>
source: discord api docs reference
Hence, you don't need to make any requests, any broadcast evals, or anything of the sort: you only need static data. Like this:
let emojiMap = {
woodenPick: "601256699629797379",
stonePick: "601256431076769803",
//etc
};
I recommend a util function for putting the emoji in a message:
function getEmoji(name) {
return `<:${name}:${emojiMap[name]}>`;
}
Use it like this:
await msg.react(emojiMap.woodenPick); //might need to be :name:id
//etc, probably use an array for that (or Object.keys(emojiMap))
//make embed
let description = `${getEmoji("woodenPick")} --> Wooden Pickaxe\netc...`;
I'm trying to scrape the https://twist.moe/ page, but cheerio doesn't show me the content of the page. Apparently it uses some sucuri protection.
When using cheerio it shows me:
<html><head><title>You are being redirected...</title>
<noscript>Javascript is required. Please enable javascript before you are allowed to see this page.</noscript>
<script>var s={},u,c,U,r,i,l=0,a,e=eval,w=String.fromCharCode,sucuri_cloudproxy_js='',S='dj0iNXN1Y3VyIi5jaGFyQXQoMCkrJzkwJy5zbGljZSgxLDIpKyI2c3VjdXIiLmNoYXJBdCgwKSsnYlMzJy5jaGFyQXQoMikrJ3BKMCcuY2hhckF0KDIpKydUZCcuc2xpY2UoMSwyKSsiNiIuc2xpY2UoMCwxKSArICAnJyArIAoiZiIgKyAgJycgKycnKyI5aiIuY2hhckF0KDApICsgImQiICsgIiIgK1N0cmluZy5mcm9tQ2hhckNvZGUoMHg2MykgKyAnYicgKyAgJzgyJy5zbGljZSgxLDIpKyJmIiArICJmc2VjIi5zdWJzdHIoMCwxKSArICdpPDEnLmNoYXJBdCgyKSsnSGxJMycuc3Vic3RyKDMsIDEpICsnQ2kyZicuc3Vic3RyKDMsIDEpICsnOGEnLnNsaWNlKDEsMikrJzknICsgICJhIi5zbGljZSgwLDEpICsgICcnICsgCiI2Ii5zbGljZSgwLDEpICsgU3RyaW5nLmZyb21DaGFyQ29kZSgxMDIpICsgImZzdSIuc2xpY2UoMCwxKSArICIiICsiOGgiLmNoYXJBdCgwKSArICdsQjQnLmNoYXJBdCgyKSsiOXN1Ii5zbGljZSgwLDEpICsgImUiLnNsaWNlKDAsMSkgKyAiNCIuc2xpY2UoMCwxKSArIFN0cmluZy5mcm9tQ2hhckNvZGUoOTgpICsgIiIgKyc3JyArICAnMCcgKyAgJyc7ZG9jdW1lbnQuY29va2llPSdzc3VjdXInLmNoYXJBdCgwKSsgJ3UnLmNoYXJBdCgwKSsnYycuY2hhckF0KDApKyd1JysncnN1YycuY2hhckF0KDApKyAnaScrJ18nLmNoYXJBdCgwKSsnY3N1Y3UnLmNoYXJBdCgwKSAgKydsc3VjdScuY2hhckF0KDApICArJ29zdScuY2hhckF0KDApICsndXMnLmNoYXJBdCgwKSsnZHN1Y3UnLmNoYXJBdCgwKSAgKydwJysnc3VjdXInLmNoYXJBdCg0KSsgJ3N1Y3VvJy5jaGFyQXQoNCkrICd4c3VjJy5jaGFyQXQoMCkrICd5c3VjdXJpJy5jaGFyQXQoMCkgKyAnc3VfJy5jaGFyQXQoMikrJ3UnKycnKyd1JysnaScrJ2RzdWN1Jy5jaGFyQXQoMCkgICsnX3N1Y3VyaScuY2hhckF0KDApICsgJ3N1Y3VyaWInLmNoYXJBdCg2KSsnc3VjdWQnLmNoYXJBdCg0KSsgJzQnKydzMycuY2hhckF0KDEpKyc5cycuY2hhckF0KDApKycwc3VjdScuY2hhckF0KDApICArJzQnKydzdWN1ZScuY2hhckF0KDQpKyAnNycuY2hhckF0KDApKyI9IiArIHYgKyAnO3BhdGg9LzttYXgtYWdlPTg2NDAwJzsgbG9jYXRpb24ucmVsb2FkKCk7';L=S.length;U=0;r='';var A='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';for(u=0;u<64;u++){s[A.charAt(u)]=u;}for(i=0;i<L;i++){c=s[S.charAt(i)];U=(U<<6)+c;l+=6;while(l>=8){((a=(U>>>(l-=8))&0xff)||(i<(L-2)))&&(r+=w(a));}}e(r);</script></head><body>
</body></html>
From chrome I accessed dev-tools and found two cookie values
"sucuri_cloudproxy_uuid_645833be2=0fa8e64535001a7393d98096a1bf40a5"
"sucuri_cloudproxy_uuid_f735b3372=77a9f80992e5d2cc9ffda8e165f8dcfb"
Both values I pass them to the header, but still shows me the same output.
const axios = require('axios');
const cheerio = require('cheerio');
const url = require('./urls');
const util = require('../utils');
const animeList = async() =>{
const headers = {
cookie: "sucuri_cloudproxy_uuid_645833be2=0fa8e64535001a7393d98096a1bf40a5; sucuri_cloudproxy_uuid_f735b3372=77a9f80992e5d2cc9ffda8e165f8dcfb;"
}
const {data} = await axios.get('https://twist.moe/' , {headers});
const body = await data;
const $ = cheerio.load(body);
const promises = [];
console.log($.html());
};
animeList();
Can someone show me how to get all the HTML content of the page https://twist.moe/?
Problem solved using the npm package cloudscraper which is Node.js library to bypass Cloudflare's anti-DDOS page. Cloudscraper can also identify and automatically bypass Sucuri.
const animeList = async() =>{
const res = await cloudscraper.get(`${url.BASE_URL}`);
const body = await res;
const $ = cheerio.load(body);
const promises = [];
console.log($.html());
};