Develop Reference

Looking for a solution to stream dynamic changes in JSON data on to browser without refreshing

I have a node.js / next.js api built that essentially does a bunch of stuff after the user submits text into a form on the front end. One of the things it does is write stage completion messages periodically to a JSON file to signify the completion of certain stages.
my api looks something like this
import dbConnect from '../../../lib/dbConnect'
import Demo from '../../../models/Demo'
import fs from 'fs'
import shell from 'shelljs';
export default async function handler(req, res) {
const {
method,
body,
} = req
await dbConnect()
switch (method) {
case 'GET':
try {
const demos = await Demo.find({})
res.status(200).json({ success: true, data: demos })
} catch (error) {
res.status(400).json({ success: false })
}
break
case 'POST':
try {
const initialjson = '[]'
const timestamp = Date.now();
// stage 1
if (shell.exec('./initial_checks.sh').code !== 0) {
shell.echo('Sorry stage failed');
shell.exit(1);
};
const objSuccess1 = JSON.parse(initialjson);
objSuccess1.push("Stage 1 complete", + timestamp);
const finalJSONSuccess1 = JSON.stringify(objSuccess1);
fs.writeFileSync('success-stage.json', finalJSONSuccess1);
// stage 2
if (shell.exec('./secondary_checks.sh').code !== 0) {
shell.echo('Sorry stage failed');
shell.exit(1);
};
const objSuccess2 = JSON.parse(initialjson);
objSuccess2.push("Stage 2 complete", + timestamp);
const finalJSONSuccess2 = JSON.stringify(objSuccess2);
fs.writeFileSync('success-stage.json', finalJSONSuccess2);
const demo = await Demo.create(
req.body
)
res.status(201).json({ success: true, data: demo })
} catch (error) {
res.status(400).json({ success: false })
}
break
default:
res.status(400).json({ success: false })
break
}
}
I am using socket.io, my server.js file is
server.js
const app = require("express")();
const server = require("http").Server(app);
const io = require("socket.io")(server);
const next = require("next");
const dev = process.env.NODE_ENV !== "production";
const nextApp = next({ dev });
const nextHandler = nextApp.getRequestHandler();
let port = 3000;
const fs = require('fs')
const data = fs.readFileSync('success-stage.json', 'utf8')
io.on("connect", (socket) => {
socket.emit("now", {
message: data
});
});
nextApp.prepare().then(() => {
app.all("*", (req, res) => {
return nextHandler(req, res);
});
server.listen(port, (err) => {
if (err) throw err;
console.log("> Ready on port: " + port);
});
});
and here is the pages/index.js file
import { useEffect, useRef, useState } from "react";
import io from "socket.io-client";
export default function IndexPage() {
const socket = useRef();
const [hello, setHello] = useState();
useEffect(() => {
socket.current = io();
socket.current.on("now", (data) => {
setHello(data.message);
});
}, []);
return <h1>{hello}</h1>;
}
so at this point we are seeing the 2nd message from my JSON file match what is rendered on the frontend when I build my application. It looks like this
["Stage 2 complete",1664289144513]
I am wondering how I can stream this data onto the front end for clients without having to refresh the page? I need it to show the current stage's success message... There are 5 total stages, so i guess i am looking for a way to either stream data or maybe to revalidate the browser window like every second without having to refresh... is this possible?
Any help would be greatly appreciated... Thanks in advance for your time everyone...

You've already got a solution implemented that can handle this. What you're describing is exactly what sockets are for -- bidirectional communication between the client and server without refreshing the page.
Just create a new socket listener on the frontend for a new topic, maybe "stageStatus", and then emit messages to that topic on the backend at various stages in the process. That's it!

Undefined end-point http://localhost:8010/undefined/

I am trying to send data from front-end to back-end. Here is my front:
import axios from 'axios'
const api = axios.create({
baseURL: `${process.env.BASE_FRONT_URL}`, // process.env.BASE_FRONT_URL = http://localhost:8010
})
export const postTip = async (payload) => {
try {
const { data } = await api.post(`post-tip`, payload);
return data;
} catch (e) {
return [];
}
};
And here is back-end:
const router = require('express').Router();
const tipController = require('../controllers/tips/tipController')
router.post('post-tip', tipController.postTip);
That function tipController.postTip actually just receive and shows data, but when I trigger this end-point I get error: POST http://localhost:8010/undefined/post-tip 404 (Not Found). So, what's wrong with end-point and how can I make it work? Also, I have no idea, where does this undefined come from? Am I missing something?

I have found my mistake. Actually process.env.BASE_FRONT_URL was really undefined, so, I made it like that:
import axios from 'axios'
const api = axios.create({
baseURL: 'http://localhost:8084',
})
export const postTip = async (payload) => {
try {
const { data } = await api.post(`post-tip`, payload);
return data;
} catch (e) {
return [];
}
};
But the most important thing is that on back-end I have to use the same port (8084)

How to implement cookie authentication | SvelteKit & MongoDB

The question stands as-is - how to implement cookie authentication in a SvelteKit & MongoDB app? Meaning how to properly use hooks, endpoints, establish a DB connection and show it on a boilerplate-ish project.

After SvelteKit Project Initialisation
#1 Install additional dependencies
npm install config cookie uuid string-hash mongodb
I prefer config over vite's .env variables due to all the leaks and problems regarding it
cookie is used to properly set cookies
uuid is used to generate complex cookie IDs
string-hash is a simple yet secure hashing for passwords stored in your DB
mongodb is used to establish a connection to your DB
#2 Set up config
In root, create a folder called config. Inside it, create a file called default.json.
config/default.json
{
"mongoURI": "<yourMongoURI>",
"mongoDB": "<yourDatabaseName>"
}
#3 Set up base DB connection code
Create lib folder in src. Inside it, create db.js file.
src/lib/db.js
import { MongoClient } from 'mongodb';
import config from 'config';
export const MONGODB_URI = config.get('mongoURI');
export const MONGODB_DB = config.get('mongoDB');
if (!MONGODB_URI) {
throw new Error('Please define the mongoURI property inside config/default.json');
}
if (!MONGODB_DB) {
throw new Error('Please define the mongoDB property inside config/default.json');
}
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
let cached = global.mongo;
if (!cached) {
cached = global.mongo = { conn: null, promise: null };
}
export const connectToDatabase = async () => {
if (cached.conn) {
return cached.conn;
}
if (!cached.promise) {
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true
};
cached.promise = MongoClient.connect(MONGODB_URI, opts).then((client) => {
return {
client,
db: client.db(MONGODB_DB)
};
});
}
cached.conn = await cached.promise;
return cached.conn;
}
The code is taken from next.js implementation of MongoDB connection establishment and modified to use config instead of .env.
#4 Create hooks.js file inside src
src/hooks.js
import * as cookie from 'cookie';
import { connectToDatabase } from '$lib/db';
// Sets context in endpoints
// Try console logging context in your endpoints' HTTP methods to understand the structure
export const handle = async ({ request, resolve }) => {
// Connecting to DB
// All database code can only run inside async functions as it uses await
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
// Getting cookies from request headers - all requests have cookies on them
const cookies = cookie.parse(request.headers.cookie || '');
request.locals.user = cookies;
// If there are no cookies, the user is not authenticated
if (!cookies.session_id) {
request.locals.user.authenticated = false;
}
// Searching DB for the user with the right cookie
// All database code can only run inside async functions as it uses await
const userSession = await db.collection('cookies').findOne({ cookieId: cookies.session_id });
// If there is that user, authenticate him and pass his email to context
if (userSession) {
request.locals.user.authenticated = true;
request.locals.user.email = userSession.email;
} else {
request.locals.user.authenticated = false;
}
const response = await resolve(request);
return {
...response,
headers: {
...response.headers
// You can add custom headers here
// 'x-custom-header': 'potato'
}
};
};
// Sets session on client-side
// try console logging session in routes' load({ session }) functions
export const getSession = async (request) => {
// Pass cookie with authenticated & email properties to session
return request.locals.user
? {
user: {
authenticated: true,
email: request.locals.user.email
}
}
: {};
};
Hooks authenticate the user based on cookies and pass the desired variables (in this example it is the user's email etc.) to context & session.
#5 Create register.js & login.js Endpoints inside auth folder
src/routes/auth/register.js
import stringHash from 'string-hash';
import * as cookie from 'cookie';
import { v4 as uuidv4 } from 'uuid';
import { connectToDatabase } from '$lib/db';
export const post = async ({ body }) => {
// Connecting to DB
// All database code can only run inside async functions as it uses await
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
// Is there a user with such an email?
const user = await db.collection('testUsers').findOne({ email: body.email });
// If there is, either send status 409 Conflict and inform the user that their email is already taken
// or send status 202 or 204 and tell them to double-check on their credentials and try again - it is considered more secure
if (user) {
return {
status: 409,
body: {
message: 'User with that email already exists'
}
};
}
// Add user to DB
// All database code can only run inside async functions as it uses await
await db.collection('testUsers').insertOne({
name: body.name,
email: body.email,
password: stringHash(body.password)
});
// Add cookie with user's email to DB
// All database code can only run inside async functions as it uses await
const cookieId = uuidv4();
await db.collection('cookies').insertOne({
cookieId,
email: body.email
});
// Set cookie
// If you want cookies to be passed alongside user when they redirect to another website using a link, change sameSite to 'lax'
// If you don't want cookies to be valid everywhere in your app, modify the path property accordingly
const headers = {
'Set-Cookie': cookie.serialize('session_id', cookieId, {
httpOnly: true,
maxAge: 60 * 60 * 24 * 7,
sameSite: 'strict',
path: '/'
})
};
return {
status: 200,
headers,
body: {
message: 'Success'
}
};
};
If you want to take it a step further, don't forget to create Schemas with Mongoose!
src/routes/auth/login.js
import stringHash from 'string-hash';
import * as cookie from 'cookie';
import { v4 as uuidv4 } from 'uuid';
import { connectToDatabase } from '$lib/db';
export const post = async ({ body }) => {
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
const user = await db.collection('testUsers').findOne({ email: body.email });
if (!user) {
return {
status: 401,
body: {
message: 'Incorrect email or password'
}
};
}
if (user.password !== stringHash(body.password)) {
return {
status: 401,
body: {
message: 'Unauthorized'
}
};
}
const cookieId = uuidv4();
// Look for existing email to avoid duplicate entries
const duplicateUser = await db.collection('cookies').findOne({ email: body.email });
// If there is user with cookie, update the cookie, otherwise create a new DB entry
if (duplicateUser) {
await db.collection('cookies').updateOne({ email: body.email }, { $set: { cookieId } });
} else {
await db.collection('cookies').insertOne({
cookieId,
email: body.email
});
}
// Set cookie
const headers = {
'Set-Cookie': cookie.serialize('session_id', cookieId, {
httpOnly: true,
maxAge: 60 * 60 * 24 * 7,
sameSite: 'strict',
path: '/'
})
};
return {
status: 200,
headers,
body: {
message: 'Success'
}
};
};
#6 Create Register.svelte and Login.svelte components
src/lib/Register.svelte
<script>
import { createEventDispatcher } from 'svelte';
// Dispatcher for future usage in /index.svelte
const dispatch = createEventDispatcher();
// Variables bound to respective inputs via bind:value
let email;
let password;
let name;
let error;
const register = async () => {
// Reset error from previous failed attempts
error = undefined;
try {
// POST method to src/routes/auth/register.js endpoint
const res = await fetch('/auth/register', {
method: 'POST',
body: JSON.stringify({
email,
password,
name
}),
headers: {
'Content-Type': 'application/json'
}
});
if (res.ok) {
dispatch('success');
} else {
error = 'An error occured';
}
} catch (err) {
console.log(err);
error = 'An error occured';
}
};
</script>
<h1>Register</h1>
<input type="text" name="name" placeholder="Enter your name" bind:value={name} />
<input type="email" name="email" placeholder="Enter your email" bind:value={email} />
<input type="password" name="password" placeholder="Enter your password" bind:value={password} />
{#if error}
<p>{error}</p>
{/if}
<button on:click={register}>Register</button>
src/lib/Login.svelte
<script>
import { createEventDispatcher } from 'svelte';
// Dispatcher for future usage in /index.svelte
const dispatch = createEventDispatcher();
// Variables bound to respective inputs via bind:value
let email;
let password;
let error;
const login = async () => {
// Reset error from previous failed attempts
error = undefined;
// POST method to src/routes/auth/login.js endpoint
try {
const res = await fetch('/auth/login', {
method: 'POST',
body: JSON.stringify({
email,
password
}),
headers: {
'Content-Type': 'application/json'
}
});
if (res.ok) {
dispatch('success');
} else {
error = 'An error occured';
}
} catch (err) {
console.log(err);
error = 'An error occured';
}
};
</script>
<h1>Login</h1>
<input type="email" name="email" placeholder="Enter your email" bind:value={email} />
<input type="password" name="password" placeholder="Enter your password" bind:value={password} />
{#if error}
<p>{error}</p>
{/if}
<button on:click={login}>Login</button>
#7 Update src/routes/index.svelte
src/routes/index.svelte
<script>
import Login from '$lib/Login.svelte';
import Register from '$lib/Register.svelte';
import { goto } from '$app/navigation';
// Redirection to /profile
function redirectToProfile() {
goto('/profile');
}
</script>
<main>
<h1>Auth with cookies</h1>
<!-- on:success listens for dispatched 'success' events -->
<Login on:success={redirectToProfile} />
<Register on:success={redirectToProfile} />
</main>
#8 Create index.svelte inside profile folder
src/routes/profile/index.svelte
<script context="module">
export async function load({ session }) {
if (!session.user.authenticated) {
return {
status: 302,
redirect: '/auth/unauthorized'
};
}
return {
props: {
email: session.user.email
}
};
}
</script>
<script>
import { onMount } from 'svelte';
export let email;
let name;
onMount(async () => {
const res = await fetch('/user');
const user = await res.json();
name = user.name;
});
</script>
<h1>Profile</h1>
<p>Hello {name} you are logged in with the email {email}</p>
Pay attention to session we set up in hooks.js. console.log() it to understand its structure better. I won't be implementing /auth/unauthorized route, so mind that.
#9 Create index.js endpoint inside user folder
src/routes/user/index.js
import { connectToDatabase } from '$lib/db';
export const get = async (context) => {
// Connecting to DB
// All database code can only run inside async functions as it uses await
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
// Checking for auth coming from hooks' handle({ request, resolve })
if (!context.locals.user.authenticated) {
return {
status: 401,
body: {
message: 'Unauthorized'
}
};
}
const user = await db.collection('testUsers').findOne({ email: context.locals.user.email });
if (!user) {
return {
status: 404,
body: {
message: 'User not found'
}
};
}
// Find a proper way in findOne(), I've run out of gas ;)
delete user.password;
return {
status: 200,
body: user
};
};
Final thoughts
There are almost none tutorials regarding SvelteKit and I'll surely find this guide useful in my future projects. If you find a bug or see an improvement, feel free to let me know so I can make this guide better ;)
Big thanks to Brayden Girard for a precedent for this guide!
https://www.youtube.com/channel/UCGl66MHcjMDJyIPZkuKULSQ
Happy coding!

Discord bot unable to get access to the Google Sheets getting error The request is missing a valid API key

I am having problem with my Discord bot trying to access the Google Sheets API v4.0. I want it to read, write, and update data in the sheet. My code work fine for Node.js application but when I put my code in a Discord bot, it gives me this error:
UnhandledPromiseRejectionWarning: Error: The request is missing a valid API key
I don't understand why I am getting this error. I even made a service account for my Discord bot.
Here is the Discord bot's code.
require('dotenv').config();
const sheet = require('./sheet');
const { Client } = require('discord.js');
const { sheets } = require('googleapis/build/src/apis/sheets');
const client = new Client();
Sheet file code:
const {google} = require('googleapis');
const keys = require('./keys.json');
const client = new google.auth.JWT(
keys.client_email,
keys.private_key_id,
keys.private_key,
['https://www.googleapis.com/auth/spreadsheets']
);
client.authorize(function(err,tokens){
if(err){
console.log(err);
return;
}
// else{
// console.log('Connected');
// gsrun(client);
// }
});
const getdata = async function gsrun(cl){
const gsapi = google.sheets({version:'v4',auth:cl});
const opt = {
spreadsheetId:'1LzdhD4rb2FdElTyQCAkgb5oyeGeu0d9rT2abS4n_4i8',
range: 'A2:B5'
};
let data = await gsapi.spreadsheets.values.get(opt);
console.log(data.data.values);
}
exports.getdata = getdata;
I have given access to bot email id to sheet which is discordbotapi#nodejs-api-testing-discordbot.iam.gserviceaccount.com. I have also enabled the Google Sheets API. Where am I making error?

When you tried it in Node.js without Discord.js, you didn't export anything and you simply called gsrun(client) after the authorisation was successful, so it worked fine. The problem is that now you try to use getData without any authorisation. Although you have client.authorize in your code, you never use it.
To solve this, I would make at least two different functions here; one for generating the client and one for the get request itself and export them both.
To generate a client I’d wrap this in a promise. This way I could use async/await later. This function will create a client with the JWT, perform the authorisation, and either resolve with the client or reject the promise depending on the results of client.authorize().
function connect() {
return new Promise((resolve, reject) => {
const scope = ['https://www.googleapis.com/auth/spreadsheets'];
const { client_email, private_key, private_key_id } = keys;
const client = new google.auth.JWT(
client_email,
private_key_id,
private_key,
scope,
);
client.authorize((err) => {
if (err) {
reject(err);
} else {
resolve(client);
}
});
});
}
Now you can simply connect and get the client by using const client = await connect().
The second part is to get some data from the spreadsheet. Again, I'd wrap it in a promise. This function will accept the client (we’ve just created above) and the options with the spreadsheetId and range. Inside the promise you just call the API endpoint with the options:
function getData(client, options) {
return new Promise((resolve, reject) => {
const endpoint = google.sheets({ version: 'v4', auth: client });
endpoint.spreadsheets.values.get(options, (err, data) => {
if (err) {
reject(err);
} else {
// or resolve with data.data.values if you only want the values
resolve(data.data);
}
});
});
}
You can export both of these in an object. Here is the full sheet.js file:
const { google } = require('googleapis');
const keys = require('./keys.json');
function connect() {
return new Promise((resolve, reject) => {
const scope = ['https://www.googleapis.com/auth/spreadsheets'];
const { client_email, private_key, private_key_id } = keys;
const client = new google.auth.JWT(
client_email,
private_key_id,
private_key,
scope,
);
client.authorize((err) => {
if (err) {
reject(err);
} else {
resolve(client);
}
});
});
}
function getData(client, options) {
return new Promise((resolve, reject) => {
const endpoint = google.sheets({ version: 'v4', auth: client });
endpoint.spreadsheets.values.get(options, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data.data);
}
});
});
}
module.exports = { connect, getData };
Then, you can import it in your bot's file and use it in there to connect first and then get the values:
const { Client } = require('discord.js');
const { connect, getData } = require('./sheet');
const client = new Client();
client.on('message', async (message) => {
if (message.author.bot) return;
const auth = await connect();
const options = {
spreadsheetId: '1LzdhD4rb2FdElTyQCAkgb5oyeGeu0d9rT2abS4n_4i8',
range: 'A2:B5',
};
const { values } = await getData(auth, options);
message.channel.send(values);
});

Cloudinary Signed Uploads with Widget

Documentation is extremely frustrating.
I'm using the upload widget to try to allow users to upload multiple pictures for their profile. I can't use unsigned uploads because of the potential for abuse.
I would much rather upload the file through the upload widget instead of through the server as it seems like it should be so simple
I've pieced together what I think should work but it is still saying: Upload preset must be whitelisted for unsigned uploads
Server:
// grab a current UNIX timestamp
const millisecondsToSeconds = 1000;
const timestamp = Math.round(Date.now() / millisecondsToSeconds);
// generate the signature using the current timestmap and any other desired Cloudinary params
const signature = cloudinaryV2.utils.api_sign_request({ timestamp }, CLOUDINARY_SECRET_KEY);
// craft a signature payload to send to the client (timestamp and signature required)
return signature;
also tried
return {
signature,
timestamp,
};
also tried
const signature = cloudinaryV2.utils.api_sign_request(
data.params_to_sign,
CLOUDINARY_SECRET_KEY,
);
Client:
const generateSignature = async (callback: Function, params_to_sign: object): Promise<void> => {
try {
const signature = await generateSignatureCF({ slug: 'xxxx' });
// also tried { slug: 'xxxx', params_to_sign }
callback(signature);
} catch (err) {
console.log(err);
}
};
cloudinary.openUploadWidget(
{
cloudName: 'xxx',
uploadPreset: 'xxxx',
sources: ['local', 'url', 'facebook', 'dropbox', 'google_photos'],
folder: 'xxxx',
apiKey: ENV.CLOUDINARY_PUBLIC_KEY,
uploadSignature: generateSignature,
},
function(error, result) {
console.log(error);
},
);

Let's all take a moment to point out how horrible Cloudinary's documentation is. It's easily the worst i've ever seen. Nightmare fuel.
Now that i've got that off my chest... I really needed to be able to do this and I spent way too long banging my head against walls for what should be extremely simple. Here it is...
Server (Node.js)
You'll need an endpoint that returns a signature-timestamp pair to the frontend:
import cloudinary from 'cloudinary'
export async function createImageUpload() {
const timestamp = new Date().getTime()
const signature = await cloudinary.utils.api_sign_request(
{
timestamp,
},
process.env.CLOUDINARY_SECRET
)
return { timestamp, signature }
}
Client (Browser)
The client makes a request to the server for a signature-timestamp pair and then uses that to upload a file. The file used in the example should come from an <input type='file'/> change event etc.
const CLOUD_NAME = process.env.CLOUDINARY_CLOUD_NAME
const API_KEY = process.env.CLOUDINARY_API_KEY
async function uploadImage(file) {
const { signature, timestamp } = await api.post('/image-upload')
const form = new FormData()
form.append('file', file)
const res = await fetch(
`https://api.cloudinary.com/v1_1/${CLOUD_NAME}/image/upload?api_key=${API_KEY}&timestamp=${timestamp}&signature=${signature}`,
{
method: 'POST',
body: form,
}
)
const data = await res.json()
return data.secure_url
}
That's it. That's all it takes. If only Cloudinary had this in their docs.

Man. I hate my life. I finally figured it out. It literally took me beautifying the upload widget js to understand that the return of the function should be a string instead of an object even though the docs make it seem otherwise.
Here is how to implement a signed upload with a Firebase Cloud Function
import * as functions from 'firebase-functions';
import cloudinary from 'cloudinary';
const CLOUDINARY_SECRET_KEY = functions.config().cloudinary.key;
const cloudinaryV2 = cloudinary.v2;
module.exports.main = functions.https.onCall(async (data, context: CallableContext) => {
// Checking that the user is authenticated.
if (!context.auth) {
// Throwing an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError(
'failed-precondition',
'The function must be called while authenticated.',
);
}
try {
return cloudinaryV2.utils.api_sign_request(data.params_to_sign, CLOUDINARY_SECRET_KEY);
} catch (error) {
throw new functions.https.HttpsError('failed-precondition', error.message);
}
});
// CLIENT
const uploadWidget = () => {
const generateSignature = async (callback: Function, params_to_sign: object): Promise<void> => {
try {
const signature = await generateImageUploadSignatureCF({ params_to_sign });
callback(signature.data);
} catch (err) {
console.log(err);
}
};
cloudinary.openUploadWidget(
{
cloudName: 'xxxxxx',
uploadSignature: generateSignature,
apiKey: ENV.CLOUDINARY_PUBLIC_KEY,
},
function(error, result) {
console.log(error);
},
);
};