Some Context
Im running a nodejs aplication that works with the Twit API.
Im using it to periodically tweet some images on tweeter, to generate the images im using processing (java API)
So im using the 'fs' lib of nodejs to execute the command:
processing-java --sketch="%cd%"\\processing\\sketch --run
Then im using a eventListener to know when processing is closed to load the generated image, to do this im using this JS promise:
File: generate-sketch-images.js
const {exec} = require('child_process');
const fs = require('fs');
module.exports = new Promise((resolve,reject) =>{
let cmd = `processing-java --sketch="%cd%"\\processing\\sketch --run`;
console.log('running processing');
exec(cmd).addListener('close',() => {
resolve(fs.readFileSync(
'processing/paints/paint.png',
{encoding:'base64'}
));
console.log('image received from processing');
});
})
Then i made this function:
File: bot.js
const generate_image = require('./utils/generate-sketch-images');
async function gerar_imagem(){
console.log('generating the image');
let img = await generate_image;
}
The problem
The 'gerar_imagens' function only works at first time
When i do something like setInterval(gerar_imagens,1000); it works just one time
I have already made a lot of console logs, and seems like the function call happens but the promise is ignored
someone knows whats happening??
Related
I am trying to understand how stream work. I have a readableStream which i am pipeing to a writableStream.
However, i am getting this error in the console when i do so. Failed to execute 'pipeTo' on 'ReadableStream': Illegal invocation
I am making a call to fetch and pass the ReadableStream using PipeTo to a WritableStream
below is the relevant part of my code.
let writeStream = fs.createWriteStream(`exports/${exportParams.key}`, {flags:'a'});
writeStream.on('end', function(){
console.log('file downloaded');
});
const fetchData = await exportPopup.evaluate(async (fetchUrl, writeStream) => {
const stream = await fetch(fetchUrl);
stream.body.pipeTo(writeStream);
}, fetchUrl, writeStream);
Any help to fix this would be really great, Thanks.
Are you using node-fetch or are you on electron? I faced the same problem trying to download files from electron. It looks like streams from the browser aren't compatible with the streams used by fs. What I finally do is.
async function downloadFile(filename, url) {
const file = fs.createWriteStream(filename);
res = await fetch(url);
buffer = await res.arrayBuffer()
file.write(Buffer.from(buffer));
file.close();
}
I don't know if this is the most efficient way of doing this but it works.
This is my first phone app. I am using Ionic for the cross-platform work which uses Angular as you know I'm sure. I have a separate program which scrapes a webpage using puppeteer and cheerio and creates an array of values from the web page. This works.
I'm not sure how I get the array in my web scraping program read by my ionic/angular program.
I have a basic ionic setup and am just trying a most basic activity of being able to see the array from the ionic/angular side but after trying to put it in several places I realized I really didnt know where to import the code to ionic/angular which returns the array or where to put the webscraper code directly in one of the .ts files or ???
This is my web scraping program:
const puppeteer = require('puppeteer'); // live webscraping
let scrape = async () => {
const browser = await puppeteer.launch({
headless: true
});
const page = await browser.newPage();
await page.goto('--page url here --'); // link to page
const result = await page.evaluate(() => {
let data = []; // Create an empty array that will store our data
let elements = document.querySelectorAll('.list-myinfo-block'); // Select all Products
let photo_elements = document.getElementsByTagName('img'); //
var photo_count = 0;
for (var element of elements) { // Loop through each product getting photos
let picture_link = photo_elements[photo_count].src;
let name = element.childNodes[1].innerText;
let itype = element.childNodes[9].innerText
data.push({
picture_link,
name,
itype
}); // Push an object with the data onto our array
photo_count = photo_count + 1;
}
return data;
});
browser.close();
return result; // Return the data
};
scrape().then((value) => {
console.log(value); // Success!
});
When I run the webscraping program I see the array with the correct values in it. Its getting it into the ionic part of it. Sometimes the ionic phone page will show up with nothing in it, sometimes it says it cannot find "/" ... I've tried so many different places and looked all over the web that I have quite a combination of errors. I know I'm putting it in the wrong places - or maybe not everywhere I should. Thank you!
You need a server which will run the scraper on demand.
Any scraper that uses a real browser (ie: chromium) will have to run in a OS that supports it. There is no other way.
Think about this,
Does your mobile support chromium and nodeJS? It does not. There are no chromium build for mobile which supports automation with nodeJS (yet).
Can you run a browser inside another browser? You cannot.
Way 1: Remote wsEndpoint
There are some services which offers wsEndpoint but I will not mention them here. I will describe how you can create your own wsEndPoint and use it.
Run browser and Get wsEndpoint
The following code will launch a puppeteer instance whenever you connect to it. You have to run it inside a server.
const http = require('http');
const httpProxy = require('http-proxy');
const proxy = new httpProxy.createProxyServer();
http
.createServer()
.on('upgrade', async(req, socket, head) => {
const browser = await puppeteer.launch();
const target = browser.wsEndpoint();
proxyy.ws(req, socket, head, { target })
})
.listen(8080);
When you run this on the server/terminal, you can use the ip of the server to connect. In my case it's ws://127.0.0.1:8080.
Use puppeteer-web
Now you will need to install puppeteer-web on your mobile/web app. To bundle Puppeteer using Browserify follow the instruction below.
Clone Puppeteer repository:
git clone https://github.com/GoogleChrome/puppeteer && cd puppeteer
npm install
npm run bundle
This will create ./utils/browser/puppeteer-web.js file that contains Puppeteer bundle.
You can use it later on in your web page to drive another browser instance through its WS Endpoint:
<script src='./puppeteer-web.js'></script>
<script>
const puppeteer = require('puppeteer');
const browser = await puppeteer.connect({
browserWSEndpoint: '<another-browser-ws-endpont>'
});
// ... drive automation ...
</script>
Way 2: Use an API
I will use express for a minimal setup. Consider your scrape function is exported to a file called scrape.js and you have the following index.js file.
const express = require('express')
const scrape= require('./scrape')
const app = express()
app.get('/', function (req, res) {
scrape().then(data=>res.send({data}))
})
app.listen(8080)
This will launch a express API on the port 8080.
Now if you run it with node index.js on a server, you can call it from any mobile/web app.
Helpful Resources
I had some fun with puppeteer and webpack,
playground-react-puppeteer
playground-electron-react-puppeteer-example
To keep the api running, you will need to learn a bit about backend and how to keep the server alive etc. See these links for full understanding of creating the server and more,
Official link to puppeteer-web
Puppeteer with docker
Docker with XVFB and Puppeteer
Puppeteer with chrome extension
Puppeteer with local wsEndpoint
Avoid memory leak on server
I've set the remote-debugging-port option for Chrome in my Electron main process:
app.commandLine.appendSwitch('remote-debugging-port', '8315')
Now, how can I get the ws:// URL that I can use to connect to Chrome?
I see that the output while I'm running Electron shows
DevTools listening on ws://127.0.0.1:8315/devtools/browser/52ba17be-0c0d-4db6-b6f9-a30dc10df13c
but I would like to get this URL from inside the main process. The URL is different every time. How can I get it from inside the Electron main process?
Can I somehow read my Electron's main process output, from within my main process JavaScript code?
Here's how to connect Puppeteer to your Electron window from your Electron main process code:
app.commandLine.appendSwitch('remote-debugging-port', '8315')
async function test() {
const response = await fetch(`http://localhost:8315/json/list?t=${Math.random()}`)
const debugEndpoints = await response.json()
let webSocketDebuggerUrl = ''
for (const debugEndpoint of debugEndpoints) {
if (debugEndpoint.title === 'Saffron') {
webSocketDebuggerUrl = debugEndpoint.webSocketDebuggerUrl
break
}
}
const browser = await puppeteer.connect({
browserWSEndpoint: webSocketDebuggerUrl
})
// use puppeteer APIs now!
}
// ... make your window, etc, the usual, and then: ...
// wait for the window to open/load, then connect Puppeteer to it:
mainWindow.webContents.on("did-finish-load", () => {
test()
})
I have been struggling to get the following code to work properly. I am using the serverless framework and it works beautifully when invoked locally. I have a whole workflow where I take the downloaded file and process it before uploading back to S3. Ideally I would like to process line-by-line, but small steps first!
I can download a small file from S3 with no problems, but the file I am handling is much larger and shouldn't be in memory.
const AWS = require("aws-sdk");
const S3 = new AWS.S3();
const fs = require("fs");
module.exports = async function (event, context, callback) {
let destWriteStream = fs.createWriteStream("/tmp/a.csv");
let s3Input = { Bucket : "somebucket", Key : "somekey.csv" };
await new Promise((resolve, reject) => {
let s3Stream = S3.getObject(s3Input)
.createReadStream()
.on("close", function() {
resolve("/tmp/a.csv");
})
.pipe( destWriteStream );
}).then(d => {
console.log(d);
});
callback( null, "good" );
};
I have tried many different ways of doing this, including Promise's. I am running Node 8.10. It is just not working when run as a Lambda. It simply times out.
My issue is that I see very little complete examples. Any help would be appreciated.
I managed to get this working. Lambda, really does not do a good job with some of its error reporting.
The issue was that the bucket I was downloading from was from a different region than the Lambda was hosted in. Apparently this does not make a difference when running it locally.
So ... for others that may tread here ... check your bucket locations relative to your Lambda region.
I'm using mongoose and node.js (express), and I wish to insert seed data using script. Like when I do node scripts/createNotifications.js I can insert a data into my db.
My code
//createNotifications.js
const mongoose = require('mongoose')
const Notification = require('../api/models/notificationModel')
mongoose.Promise = global.Promise
module.exports = (async () => {
try {
const new_notification = await new Notification({
"userId" : mongoose.Types.ObjectId("5a3e76ce914e1d1bd854451d"),
"msg" : "Something"
}).save()
} catch(e) {
console.log('Error creating notifications. ', e)
}
})()
When I run the code I don't see any data been inserted. I have my server started in port 3000, do I have to connect to mongodb too in this file? since this file has nothing to do with my express app, it's just a separated file.
If you want to see this module running make sure the following
Make sure you've made connection with the database like mongoose.connect('mongodb://IP/DBName')
What you've posted above is just a module definition. It won't execute on its own. You'll have to require this module in your mail file, the file you're running with node for example node server.js and call the method. Something like
var notification = require(path/to/createNotifications);
notification();