I'm using Apify web scraper to scrape h1,h2 and a span from a website. I have read all of their docs but I can't seem to find an answer for this.
async function pageFunction(context) {
const $ = context.jQuery;
const h1 = $('main h1').first().text();
const first_h2 = $('h2').first().text();
const random_text_from_the_page = $('main span').text();
// Print some information to actor log
context.log.info(`URL: ${context.request.url}, What is going on?: ${h1}`);
// Manually add a new page to the queue for scraping.
// await context.enqueueRequest({ url: 'http://www.example.com' });
// Return an object with the data extracted from the page.
// It will be stored to the resulting dataset.
return {
url: context.request.url,
h1,
first_h2,
random_text_from_the_page
};
}
How can I use that json data I get back and display it in a simple React UI for people to see?
Is it even possible? If not in React maybe just a simple vanilla JS UI?
Thank you.
Related
I have a Laravel 8 backend API and a completely separate NuxtJS frontend. I have integrated Algolia into the backend API. I have a nice adapter and service and I can search my index.
Now, I am not interested in using scout because I don't like what it is doing and how it works and that's not the problem so I would like to leave it out of the discussion.
So I've made search work on the frontend with Vuetify autocomplete but I decided to use VueInstant search as this is supposed to save me some work when it comes to integrating query suggestions.
Before I can even get query suggestion I need to get the basic search working with Vue Instant Search.
GOAL
I want to have a VueInstant Search with the backend search client.
WHAT I HAVE SO FAR
THAT IS WITHOUT QUERY SUGGESTIONS JUST THE BASIC SEARCH WITH VUEINSTANT SEARCH
I have backend code that searches my index. I have the frontend code that creates a new connection to my backend (don't worry about how it looks like I just need to get this to work first and then I will invest the time to refactor it):
customSearchClient () {
const that = this
return {
search(requests) {
return that.fetchContainers({ criteria: { query: 'super' }, updateStore: false }).then(response => {
// console.log({ response }, typeof response)
// return response.data.hits
return { results: response.data }
// return response
// return response.data.hits
})
}
}
}
And this is my code for the form:
<ais-instant-search index-name="containers-index" :search-client="customSearchClient()" >
<ais-search-box />
<ais-hits>
<template slot="item" slot-scope="{ item }">
<h1><ais-highlight :hit="item" attribute="name" /></h1>
<p><ais-highlight :hit="item" attribute="description" /></p>
</template>
</ais-hits>
</ais-instant-search>
PROBLEMS
I can get the searchbox to show and query if I remove ais-hits tags. As soon as I add them I get weird errors depending on how I format my response from the backend. I just try to pass it as it is.
I went through some debugging and tried to wrap this into various wrappers as they seem to be missing but eventually it always breaks, for example:
algoliasearch.helper.js?ea40:1334 Uncaught (in promise) TypeError: content.results.slice is not a function at AlgoliaSearchHelper._dispatchAlgoliaResponse (algoliasearch.helper.js?ea40:1334:1)
And that is the Algolia code that breaks.
this._currentNbQueries -= (queryId - this._lastQueryIdReceived);
this._lastQueryIdReceived = queryId;
if (this._currentNbQueries === 0) this.emit('searchQueueEmpty');
var results = content.results.slice();
states.forEach(function(s) {
var state = s.state;
var queriesCount = s.queriesCount;
var helper = s.helper;
var specificResults = results.splice(0, queriesCount);
var formattedResponse = helper.lastResults = new SearchResults(state, specificResults);
SUMAMRY
The ideal solution would be to not to use this InstantSearch thing but I have no clue how to manage more than one index on the server side.
Or am I completely wrong about all of that? Anyone can advise?
So I'm working on a CMS project that allows users to create their own websites just like wordpress or other CMS platforms...
The users can implement different modals into their websites (text modal, image modal, search modal and other stuff), and then we create an object with the created page infos.
The Object contains all the page infos like the example bellow:
{
pageName: "Home page",
pageLink: "home",
pageSlug: "home-page",
pageMetaDescription: "home meta description",
pageMetaTitle : "home meta title",
pageModals : [
modal1: {
//modal infos here.
}
modal2: {
//modal infos here.
}
]
}
What I'm doing now is stocking these Objects on a database and when the user requests a page, I fetch the object and then generate a react JS file. But this approach isn't the best for performance or SEO.
So I would like to actually generate an HTML file from these Objects and store them in the database and when the user requests a page, it just loads the generated HTML file instead of fetching the Object and populating a react JS page.
If you have an Idea or approach to do this, I would like your help.
Yes you can use Next.js to handle your case
Next allow you to fetch some external data and render html based on api result
Here an example from documentation adapted for your cases
// pagesInfos will be populated at build time by getStaticProps()
function Blog({ pagesInfos }) {
const { pageSlug, pageMetaDescription , pageMetaTitle ...} = pagesInfos
return (
<div>
<h1>{postMetaTitle}</h1>
<p>{pageMetaDescription}</p>
</div>
)
}
// This function gets called at build time on server-side.
// It won't be called on client-side, so you can even do
// direct database queries.
export async function getStaticProps() {
// Call an external API endpoint to get posts.
// You can use any data fetching library
const res = await fetch('https://.../getPages')
const pagesInfos = await res.json()
// By returning { props: { pagesInfos } }, the Blog component
// will receive `pagesInfos` as a prop at build time
return {
props: {
pagesInfos,
},
}
}
export default Blog
Here is full docs : https://nextjs.org/docs/basic-features/data-fetching/get-static-props
I have looked at multiple resources for this, however, none seem to be able to answer my question. I have a local JSON file in my React app called items.json. In that file, is a list of objects, which I want to be able to update. I have tried using fs however this apparently doesn't work in React, as I received this error:
Unhandled Rejection (TypeError): fs.readFileSync is not a function
What I am trying to do, is that when the code gets a new item, it looks through the JSON file to see if there is an existing object with a matching values in its name property. If there is, it increments that objects count property by 1, otherwise it creates a new object, and appends it to the list in the JSON file. This is the code that I have written to do that. The logic seems sound (although its not tested) but I can't figure out how to read/write the data.
let raw = fs.readFileSync("../database/items.json");
let itemList = JSON.parse(raw);
let found = false;
for (let item of itemList.averages) {
if (item.name === this.state.data.item_name) {
found = true;
item.count += 1;
}
}
if (!found) {
let newItem = {
name: this.state.data.item_name,
count: 1,
}
itemList.averages.push(newItem);
}
let newRaw = JSON.stringify(itemList);
fs.writeFileSync("../database/items.json", newRaw);
The JSON file:
{
"averages": [
{
"name": "Example",
"count": 1,
}
]
}
First of all, the browser itself doesn't have access to the filesystem, so you won't be able to achieve that using your react app. However, this can be achieved if you use Node.js(or any other FW) at the backend and create an API endpoint which can help you to write to the filesystem.
Secondly, if you wanted to only do things on the frontend side without creating an extra API just for saving the data in a JSON file which I think is not necessary in your case. You can use localstorage to save the data and ask the user to download a text file using this :
TextFile = () => {
const element = document.createElement("a");
const textFile = new Blob([[JSON.stringify('pass data from localStorage')], {type: 'text/plain'}); //pass data from localStorage API to blob
element.href = URL.createObjectURL(textFile);
element.download = "userFile.txt";
document.body.appendChild(element);
element.click();
}
Now, To use local storage API you can check here - https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage
reading and writing JSON file to local storage is quite simple with NodeJs, which means a tiny piece of backend API in express would help get this job done.
few piece of code that might help you. Assuming you JSON structure would be such as below;
{
"name":"arif",
"surname":"shariati"
}
Read JSON file;
// import * as fs from 'fs';
const fs = require('fs')
fs.readFile('./myFile.json', 'utf8', (err, jsonString) => {
if (err) {
return;
}
try {
const customer = JSON.parse(jsonString);
} catch(err) {
console.log('Error parsing JSON string:', err);
}
})
customer contains your JSON, and values can be accessed by customer.name;
Write to JSON File
Let's say you have an update on your JSON object such as below;
const updatedJSON = {
"name":"arif updated",
"surname":"shariati updated"
}
Now you can write to your file. If file does not exist, it will create one. If already exists, it will overwrite.
fs.writeFile('./myFile.json', JSON.stringify(updatedJSON), (err) => {
if (err) console.log('Error writing file:', err);
})
Importing and reading from json can be like:
import data from ‘./data/data.json’;
then use .map() to iterate data.
for writing locally you can use some libraries like https://www.npmjs.com/package/write-json-file
I'm using the shopify-buy SDK to try and fetch the articles off of my Shopify store just using JavaScript on the frontend, following the "Expanding the SDK" directions here: https://shopify.github.io/js-buy-sdk/#expanding-the-sdk.
Using the code below, I am able to retrieve my articles and some of the fields that I need.
// Build a custom query using the unoptimized version of the SDK
const articlesQuery = client.graphQLClient.query((root) => {
root.addConnection('articles', {args: {first: 10}}, (article) => {
article.add('title')
article.add('handle')
article.add('url')
article.add('contentHtml')
})
})
// Call the send method with the custom query
client.graphQLClient.send(articlesQuery).then(({model, data}) => {
console.log('articles data')
console.log(data)
})
However, I really need to pull the featured image for each article as well, and unfortunately, when I add the line article.add('image') in my articlesQuery, the resulting articles data logs null. I tried building a custom productsQuery, and that has a similiar problem - I can retrieve some of the product fields, but when I try add the line product.add('images'), I just get null back from the storefront API.
Does anyone have experience building custom/expanded queries and successfully retrieving images?
Try following:
// Fetch all products in your shop
client.graphQLClient.fetchAll().then((acticles) => {
console.log(acticles);
});
And then check in console what sort of available property names your articles have. If SDK allows you get any image data, there should be for sure anything like imageSrc || imageUrl || img......
Thanks to Rebecca Friedman on the js-buy-sdk repo's github issues section for providing this working solution:
const articlesQuery = client.graphQLClient.query((root) => {
root.addConnection('articles', {args: {first: 10}}, (article) => {
article.add('title')
article.add('handle')
article.add('url')
article.add('contentHtml')
article.addField('image', {}, (image) => {
image.add('id')
image.add('originalSrc')
})
})
})
// Call the send method with the custom query
client.graphQLClient.send(articlesQuery).then(({model, data}) => {
console.log('articles data')
console.log(data) // works!
})
Because the image field is its own object, you have to add a callback function to specify the fields you need.
So I am trying to scrape some content with node.js x-ray scraping framework. While I can get the content from a single page but for exemple only for one employee I can't get my head around on how to get for all the employees.
Working Exemple but return me the first employee:
const request =require('request');
const Xray=require('x-ray');
var x = Xray();
x('http://www.viadeo.com/fr/company/unicef',
'.pan',[{
name:'.pan-emp-name',
job:'.pan-emp-pos',
since:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Thank you so much
x('http://www.viadeo.com/fr/company/unicef',
'#pan-emp .pan-employees .pan-empployee',[{
company:'#company-info .company-logo-picture',
nom:'.pan-emp-name',
job:'.pan-emp-pos',
depuis:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Working like a charm,
So now my problem is to get the company info