How do I use HTTP urls with Node ESM module loader? - javascript

I have the following import mongo from "mongodb"; I would like to avoid using npm and instead use unpkg.com like this import mongo from "https://unpkg.com/mongodb";. However, when I run I get...
...#penguin:~/...$ node --harmony test.mjs node:internal/process/esm_loader:74
internalBinding('errors').triggerUncaughtException(
^
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader. Received protocol 'https:'
at new NodeError (node:internal/errors:328:5)
at Loader.defaultResolve [as _resolve] (node:internal/modules/esm/resolve:825:11)
at Loader.resolve (node:internal/modules/esm/loader:86:40)
at Loader.getModuleJob (node:internal/modules/esm/loader:230:28)
at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:56:40)
at link (node:internal/modules/esm/module_job:55:36) {
code: 'ERR_UNSUPPORTED_ESM_URL_SCHEME'
}
Because it uses the term default module loader I was wondering if there was an alternative ESM loader I could use.

Node will eventually support ESM URL imports. This feature is currently available under a CLI flag --experimental-network-imports. It doesn't ship with public releases yet (as of Node 18.0.0), so you will need to build Node from source for this as of current. See https://nodejs.org/api/esm.html#https-and-http-imports
In the meanwhile, Node has experimental support for module loaders and even shows a HTTPS loader as an example. See https://nodejs.org/api/esm.html#esm_https_loader
// https-loader.mjs
import { get } from 'https';
export function resolve(specifier, context, defaultResolve) {
const { parentURL = null } = context;
// Normally Node.js would error on specifiers starting with 'https://', so
// this hook intercepts them and converts them into absolute URLs to be
// passed along to the later hooks below.
if (specifier.startsWith('https://')) {
return {
url: specifier
};
} else if (parentURL && parentURL.startsWith('https://')) {
return {
url: new URL(specifier, parentURL).href
};
}
// Let Node.js handle all other specifiers.
return defaultResolve(specifier, context, defaultResolve);
}
export function load(url, context, defaultLoad) {
// For JavaScript to be loaded over the network, we need to fetch and
// return it.
if (url.startsWith('https://')) {
return new Promise((resolve, reject) => {
get(url, (res) => {
let data = '';
res.on('data', (chunk) => data += chunk);
res.on('end', () => resolve({
// This example assumes all network-provided JavaScript is ES module
// code.
format: 'module',
source: data,
}));
}).on('error', (err) => reject(err));
});
}
// Let Node.js handle all other URLs.
return defaultLoad(url, context, defaultLoad);
}
Use like so:
node --experimental-loader ./https-loader.mjs ./main.mjs

Related

Why are imported modules on serverless functions undefined?

TL;DR - why do imports return undefined? Especially for native Node modules like path?
I have a very small test application, built with Vite, that has a single endpoint in the /api directory. I created this just to play around with Vite + Vercel.
I have only 2 module imports for my endpoint - path and fs-extra. Both are returning as undefined. I was getting cannot read join of undefined errors with the path module, so I wrapped everything in a try/catch just to see if the endpoint responds. It does. See my code below.
import type {VercelRequest, VercelResponse} from '#vercel/node'
import path from 'node:path' // I've also tried 'path'
import fs from 'fs-extra'
export default function handler(req: VercelRequest, res: VercelResponse) {
// Both of these log 'undefined' on my Vercel dashboard for function logs.
console.log('path module:', path)
console.log('fs module:', fs)
try {
// https://vercel.com/guides/how-can-i-use-files-in-serverless-functions
const pathFromProjectRootToFile = '/api/usersData.json'
const usersDataFilePath = path.join( // THIS IS WHERE THE ERROR HAPPENS 🙃
process.cwd(),
pathFromProjectRootToFile
)
const usersData = fs.readJSONSync(usersDataFilePath)
res.json({users: usersData})
} catch (e) {
res.json({error: errorToObject(e as Error)})
}
}
function errorToObject(err?: Error): Record<string, string> | null {
if (!err || !(err instanceof Error)) return null
const obj: Record<string, string> = {}
Object.getOwnPropertyNames(err).forEach(prop => {
const value = err[prop as keyof Error]
if (typeof value !== 'string') return
obj[prop] = value
})
return obj
}
As an aside, instead of node:path I also tried just path, but same thing - undefined. And I do have fs-extra as a dependency in my package.json.
When deploying a Vite app on Vercel and utilizing the /api directory for a back end with Serverless Functions, Node modules need to be imported via require, not import. The import syntax works for types and local modules. See this answer (and conversation) in Vercel's community discussions.
import type {VercelRequest, VercelResponse} from '#vercel/node'
import localModule from '../myLocalModule'
const path = require('path')
const fsOriginal = require('fs')
const fs = require('fs-extra')
// Rest of the file here...

How to setup clipboardy in cypress 10?

I'm pretty new to cypress. I tried to install clipboardy to one of my project.
But the guide that I found online like this mostly setup on the older cypress which is using the plugins/index.js file.
I tried something like this and got error
const { defineConfig } = require("cypress");
const createBundler = require("#bahmutov/cypress-esbuild-preprocessor");
const addCucumberPreprocessorPlugin =
require("#badeball/cypress-cucumber-preprocessor").addCucumberPreprocessorPlugin;
const createEsbuildPlugin =
require("#badeball/cypress-cucumber-preprocessor/esbuild").createEsbuildPlugin;
const clipboardy = require("clipboardy");
module.exports = defineConfig({
e2e: {
async setupNodeEvents(on, config) {
const bundler = createBundler({
plugins: [createEsbuildPlugin(config)],
});
on("file:preprocessor", bundler);
await addCucumberPreprocessorPlugin(on, config);
on('task', {
getClipboard () {
return clipboardy.readSync();
}
});
return config;
},
specPattern: "cypress/e2e/features/*.feature",
baseUrl: "XXXXXXXXXXXXX",
chromeWebSecurity: false,
},
});
The error
Error screen
Stack Trace
Error [ERR_REQUIRE_ESM]: require() of ES Module C:\Users\XXXXXXXXXXXXXXXXXXXXXXX\node_modules\clipboardy\index.js from C:\Users\XXXXXXXXXXXXXXXXXXXXXXX\cypress.config.js not supported.
Instead change the require of index.js in C:\Users\XXXXXXXXXXXXXXXXXXXXXXX\cypress.config.js to a dynamic import() which is available in all CommonJS modules.
at Object. (C:\Users\XXXXXXXXXXXXXXXXXXXXXXX\cypress.config.js:8:20)
at async Promise.all (index 0)
at async loadFile (C:\Users\XXXX\AppData\Local\Cypress\Cache\10.4.0\Cypress\resources\app\node_modules\#packages\server\lib\plugins\child\run_require_async_child.js:106:14)
at async EventEmitter. (C:\Users\XXXX\AppData\Local\Cypress\Cache\10.4.0\Cypress\resources\app\node_modules\#packages\server\lib\plugins\child\run_require_async_child.js:116:32)
It's because the clipboardy package has type: "module" in it's 'package.json'.
This tells anything that uses the package that you should use import not require to load the package.
Use dynamic imports, as hinted in the error message
Instead change the require of index.js in ... to a dynamic import()
const clipboard = import('clipboardy')

React Module not found: Can't resolve '../utils/api'

I created a module in which I keep functions to call an api. While 'requiring' it, I get the following error:
./src/Components/Search/SearchPage.js
Module not found: Can't resolve '../utils/api' in 'C:\Users\riksch\Dropbox\projects\Current\greenmp\frontend\src\Components\Search'
My main question is: how do I correctly import the api module into SearchPage.js?
Here is the structure of my project:
I've highlighted the files that I use, 1 is the file that I import (require) from, and 2 is the module I try to import.
This worked before, but now that I changed the folder structure, even after adjusting the path, I can't get it too work.
I've tried different import paths, all with the same error.
SearchPage.js require statement
const api = require('../utils/api')
api.js
var axios = require('axios')
module.exports = {
retrievePlants: function(search_query, locale) {
console.log("api.retrievePlants executes")
console.log("url: " + 'http://127.0.0.1:8000/search/'+locale+'/'+search_query)
//FIXME: hardcoded URL HOST
// return axios.get('https://127.0.0.1/search/'+locale+'/'+search_query)
return axios.get('http://127.0.0.1:8000/search/'+locale+'/'+search_query)
.then(function(response) {
console.log("response.data:")
console.log(response.data)
return response.data
})
.catch(function(error) {
console.log("Error in Components.utils.api.retrievePlants:")
console.log(error)
console.log("console.log(error.response.data):")
console.log(error.response.data)
})
},
}
You have to go two directories up like below
const api = require('../../utils/api');
it will work.

ReferenceError: fetch is not defined

I have this error when I compile my code in node.js, how can I fix it?
RefernceError: fetch is not defined
This is the function I am doing, it is responsible for recovering information from a specific movie database.
function getMovieTitles(substr){
pageNumber=1;
let url = 'https://jsonmock.hackerrank.com/api/movies/search/?Title=' + substr + "&page=" + pageNumber;
fetch(url).then((resp) => resp.json()).then(function(data) {
let movies = data.data;
let totPages = data.total_pages;
let sortArray = [];
for(let i=0; i<movies.length;i++){
sortArray.push(data.data[i].Title);
}
for(let i=2; i<=totPages; i++){
let newPage = i;
let url1 = 'https://jsonmock.hackerrank.com/api/movies/search/?Title=' + substr + "&page=" + newPage;
fetch(url1).then(function(response) {
var contentType = response.headers.get("content-type");
if(contentType && contentType.indexOf("application/json") !== -1) {
return response.json().then(function(json) {
//console.log(json); //uncomment this console.log to see the JSON data.
for(let i=0; i<json.data.length;i++){
sortArray.push(json.data[i].Title);
}
if(i==totPages)console.log(sortArray.sort());
});
} else {
console.log("Oops, we haven't got JSON!");
}
});
}
})
.catch(function(error) {
console.log(error);
});
}
If you're using a version of Node prior to 18, the fetch API is not implemented out-of-the-box and you'll need to use an external module for that, like node-fetch.
Install it in your Node application like this
npm install node-fetch
then put the line below at the top of the files where you are using the fetch API:
import fetch from "node-fetch";
This is a quick dirty fix, please try to eliminate this usage in production code.
If fetch has to be accessible with a global scope
import fetch from 'node-fetch'
globalThis.fetch = fetch
You can use cross-fetch from #lquixada
Platform agnostic: browsers, node or react native
Install
npm install --save cross-fetch
Usage
With promises:
import fetch from 'cross-fetch';
// Or just: import 'cross-fetch/polyfill';
fetch('//api.github.com/users/lquixada')
.then(res => {
if (res.status >= 400) {
throw new Error("Bad response from server");
}
return res.json();
})
.then(user => {
console.log(user);
})
.catch(err => {
console.error(err);
});
With async/await:
import fetch from 'cross-fetch';
// Or just: import 'cross-fetch/polyfill';
(async () => {
try {
const res = await fetch('//api.github.com/users/lquixada');
if (res.status >= 400) {
throw new Error("Bad response from server");
}
const user = await res.json();
console.log(user);
} catch (err) {
console.error(err);
}
})();
If you want to avoid npm install and not running in browser, you can also use nodejs https module;
const https = require('https')
const url = "https://jsonmock.hackerrank.com/api/movies";
https.get(url, res => {
let data = '';
res.on('data', chunk => {
data += chunk;
});
res.on('end', () => {
data = JSON.parse(data);
console.log(data);
})
}).on('error', err => {
console.log(err.message);
})
fetch came to Node v17 under experimental flag --experimental-fetch
It will be available in Node v18 without the flag.
https://github.com/nodejs/node/pull/41749#issue-1118239565
You no longer need any additional package to be installed
EDITED - New Solution
To use the latest version (3.0.0) you must do the import like this:
const fetch = (url) => import('node-fetch').then(({default: fetch}) => fetch(url));
Old Anwser:
This may not be the best solution, but if you install this version :
npm install node-fetch#1.7.3
you can now use the line below without error's.
const fetch = require("node-fetch");
Node.js hasn't implemented the fetch() method, but you can use one of the external modules of this fantastic execution environment for JavaScript.
In one of the other answers, "node-fetch" is cited and that's a good choice.
In your project folder (the directory where you have the .js scripts) install that module with the command:
npm i node-fetch --save
Then use it as a constant in the script you want to execute with Node.js, something like this:
const fetch = require("node-fetch");
You should add this import in your file:
import * as fetch from 'node-fetch';
And then, run this code to add the node-fetch:
$ yarn add node-fetch
If you're working with typescript, then install node-fetch types:
$ yarn add #types/node-fetch
Best one is Axios library for fetching.
use npm i --save axios for installng and use it like fetch, just write axios instead of fetch and then get response in then().
You have to use the isomorphic-fetch module to your Node project because Node does not contain Fetch API yet. For fixing this problem run below command:
npm install --save isomorphic-fetch es6-promise
After installation use below code in your project:
import "isomorphic-fetch"
For those also using typescript on node-js and are getting a ReferenceError: fetch is not defined error
npm install these packages:
"amazon-cognito-identity-js": "3.0.11"
"node-fetch": "^2.3.0"
Then include:
import Global = NodeJS.Global;
export interface GlobalWithCognitoFix extends Global {
fetch: any
}
declare const global: GlobalWithCognitoFix;
global.fetch = require('node-fetch');
It seems fetch support URL scheme with "http" or "https" for CORS request.
Install node fetch library npm install node-fetch, read the file and parse to json.
const fs = require('fs')
const readJson = filename => {
return new Promise((resolve, reject) => {
if (filename.toLowerCase().endsWith(".json")) {
fs.readFile(filename, (err, data) => {
if (err) {
reject(err)
return
}
resolve(JSON.parse(data))
})
}
else {
reject(new Error("Invalid filetype, <*.json> required."))
return
}
})
}
// usage
const filename = "../data.json"
readJson(filename).then(data => console.log(data)).catch(err => console.log(err.message))
In node.js you can use : node-fetch package
npm i node-fetch
then :
import fetch from 'node-fetch';
here is a full sample in (nodejs) :
import fetch from "node-fetch";
const fetchData = async () => {
const res = await fetch("https://restcountries.eu/rest/v2/alpha/col"); // fetch() returns a promise, so we need to wait for it
const country = await res.json(); // res is now only an HTTP response, so we need to call res.json()
console.log(country); // Columbia's data will be logged to the dev console
};
fetchData();
In HackerRank, some libraries are installed by default and some are not.
Because it is running Node.js, the fetch API is not installed by default.
The best thing for you to do is to check whether the libraries are or not installed.
on the top of the exercise, there is the following:
const https = require('https');
Please try to add this to the top as well:
const axios = require('axios');
and then run the code.
If there is a compilation error, then it's not available, otherwise you can use axios, which is a good alternative to fetch
To use it with then, you can:
function getMovieTitles(substr){
axios.get(url)
.then(function(response){
console.log(response.data);
})
}
or taking advantage of the async/await
async function getMovieTitles(substr){
let response = await axios.get(url)
console.log(response.data);
}
This is the related github issue
This bug is related to the 2.0.0 version, you can solve it by simply upgrading to version 2.1.0.
You can run
npm i graphql-request#2.1.0-next.1
The following works for me in Node.js 12.x:
npm i node-fetch;
to initialize the Dropbox instance:
var Dropbox = require("dropbox").Dropbox;
var dbx = new Dropbox({
accessToken: <your access token>,
fetch: require("node-fetch")
});
to e.g. upload a content (an asynchronous method used in this case):
await dbx.filesUpload({
contents: <your content>,
path: <file path>
});
This worked for me:
const nodeFetch = require('node-fetch') as typeof fetch;
For me these are looking more simple.
npm install node-fetch
import fetch from "node-fetch";
There are actually a lot of different libraries for making fetch available in the browser.
The main ones I'm aware of are:
node-fetch
cross-fetch
whatwg-fetch
isomorphic-fetch
I currently use node-fetch, and it has worked fine, but I don't really know which one is "the best". (though the openbase.com pages I linked to provide some metadata on usage [eg. Github stars, npm downloads], which can help)
npm i node-fetch
Once installed, in your JavaScript file:
import fetch from "node-fetch";
Lastly make this change package.json file:
"type": "module"
Just make your app.js file Extension as app.mjs and the problem will be solved!!!:)
Solution without installations
Method 1
import { PLATFORM_ID } from '#angular/core';
import { isPlatformBrowser, isPlatformServer } from '#angular/common';
constructor(#Inject(PLATFORM_ID) private platformId: Object) {
// constructor code
}
ngOnInit() {
if (isPlatformBrowser(this.platformId)) {
// Client only code. Any javascript base functions
}
if (isPlatformServer(this.platformId)) {
// Server only code. Any javascript base functions
}
}
Method 2
import { PLATFORM_ID} from '#angular/core';
import { isPlatformBrowser } from '#angular/common';
#Component({
selector: 'app-navigation',
templateUrl: './navigation.component.html',
styleUrls: ['./navigation.component.scss'],
changeDetection: ChangeDetectionStrategy.OnPush,
})
export class NavigationComponent implements OnInit {
private isBrowser: boolean = false;
constructor(
#Inject(PLATFORM_ID) private platformId: Object
) {
this.isBrowser = isPlatformBrowser(platformId);
}
ngOnInit(): void {
if (this.isBrowser) {
fetch('https://jsonplaceholder.typicode.com/posts/1')
.then((response) => response.json())
.then((json) => console.log(json));
}
}
}
DEMO - JSFIDDLE - Open console to view the fetch api service working
This answer does not directly answer this question. Instead it suggests for an alternative.
Why? Because the using 'node-fetch' is getting complicated since you cannot import the updated versions using const fetch = require('node-fetch') . You will have to do more things to just make it work.
Try using axios package:
Simple installation npm i axios
code for fetching goes like
const response = await axios.get(url).then(res => res.data)
Might sound silly but I simply called npm i node-fetch --save in the wrong project. Make sure you are in the correct directory.
If need install:
npm install --save global-fetch
then
var fetch = require("node-fetch");

Calling child_process with Node.js vs calling child process from C and creating a C++ bind to call from node.js

I would like to call pdftotext to extract the content of 100.000 files (and i need to be fast), so, which of these two implementations would be the fastest?
Implementation 1:
Create a child_process from node.js, for every extraction:
export default (file) => new Promise((resove, reject) => {
const command = 'pdftotext'
const args = ['-layout', '-enc', 'UTF-8', file, '-']
const process = spawn(command, args)
const stdout = []
const stderr = []
process.stdout.on('data', (data) => {
stdout.push(data)
})
process.stderr.on('data', (data) => {
stderr.push(data)
})
process.on('error', (error) => {
if (error.code === 'ENOENT')
error.message = 'pdftotext is not installed, so will be unable to extract the file content'
reject(error)
})
process.on('close', () => {
if (stderr.length)
return reject(stderr.map(Error))
resolve(stdout.join())
})
}
Implementation 2:
Create a child_process from C, and create a C++ binding to call from node.js
-- Without code because I'm still learning how to do it --
Most likely process invocation code will have nonessential impact on performance and the speed of document processing depends on pdftotext implementation and disk io. So I guess there is no point to bother with writing custom process launcher.

Categories

Resources