I'm looking for a simple way to cache HTML that I pull using the request-promise library.
The way I've done this in the past is specify a time-to-live say one day. Then I take the parameters passed into request and I hash them. Then whenever a request is made I save the HTML contents on the file-system in a specific folder and name the file the name of the hash and the unix timestamp. Then when a request is made for the using the same parameters I check if the cache is still relevant via timestamp and pull it or make a new request.
Is there any library that can help with this that can wrap around request? Does request have a method of doing this natively?
I went with the recco in the comments and used Redis. Note this only works for get requests.
/* cached requests */
async function cacheRequest(options){
let stringOptions = JSON.stringify(options)
let optionsHashed = crypto.createHash('md5').update(stringOptions).digest('hex')
let get = await client.getAsync(optionsHashed)
if (get) return get
let HTML = await request.get(options)
await client.setAsync(optionsHashed, HTML)
return HTML
}
Related
I'm making a Discord bot in JavaScript and implementing a feature where when you ask a coding question it gives you a snippet. I'm using Grepper and returning the url with the search results. For example:
Hello World in JavaScript Search Results. I would like to access the div containing the snippet. Is this possible? And how would I do it?
Here's my code:
if (message.startsWith('programming')) {
// Command = programming
message = message.replace('programming ', ''); // Remove programming from the string
message = encodeURIComponent(message) // Encode the string for a url
msg.channel.send(`https://www.codegrepper.com/search.php?answer_removed=1&q=${message}`); // Place formatted string into url and send it to the discord server
// Here the program should access the element containing the snippet instead of sending the url:
}
I'm new to JavaScript so sorry if this is a stupid question.
As far as I know the API you are using returns HTML/Text data, not JSON, Grepper has a lot more APIs if you just look into them, you can instead use this API that returns JSON data. If you need more information you can check this Unofficial List of Grepper APIs
https://www.codegrepper.com/api/get_answers_1.php?v=2&s=${SearchQuery}&u=98467
How Do I Access the div containing the snippet?
To access the div you might need to use python web scraping to scrape the innerHTML of the div but I think it's easier to use the other API.
Or
You can put /api/ in the url like:
https://www.codegrepper.com/api/search.php?answer_removed=1&q=js%20loop
The easiest way for this is to send a GET request to the underlying API
https://www.codegrepper.com/api/search.php?q=hello%20world%20javascript&search_options=search_titles
This will return the answers in JSON format. Obviously you'd have to adjust the parameters.
How did I find out about this?
Simply look at the network tab of your browser's dev tools while loading the page. You'll see a GET request being sent out to the endpoint, returning mentioned answers as JSON.
The best way is to use the grepper api.
Install node-fetch
npm i node-fetch, You need this package for making requestes to the api.
To import It in the code just type:
const fetch = require('node-fetch');
Write this code
Modify your code like this:
if (message.startsWith('programming')) {
message = message.replace('programming ', '');
message = encodeURIComponent(message)
// Making the request
fetch(`https://www.codegrepper.com/api/search.php?answer_removed=1&q=${message}`)
.then(res => res.json())
.then(response => {
// response is a json object containing all the data You need
// now You need to parse this data
const answers = response.answers; // this is an array of objects
const answers_limit = 3; // set a limit for the answers
// cheking if there is any answer
if(answers.length == 0) {
return msg.channel.send("No answers were found!");
}
// creating the embed
const embed = new Discord.MessageEmbed()
.setTitle("Here the answers to your question!")
.setDescription("")
// parsing
for(let i = 0; i < answers_limit; i++) {
if(answers[i]) {
embed.description += `**${i+1}° answer**:\n\`\`\`js\n${answers[i].answer}\`\`\`\n`;
}
}
console.log(embed)
msg.channel.send(embed);
});
}
I have a test that makes an API call, and a required part of the body that I'm passing in is a timestamp. The whole test is working except for this last piece (if I manually set the date/time before each test execution the test succeeds).
I have been trying to set a variable like this:
const todaysDate = Cypress.moment().format('YYYY-MM-DDTHH:MM:SS-07:00');
but I cannot figure out how to use this in the API call. The API parameter is formatted as such:
"offDateTime": "YYYY-MM-DDTHH:MM:SS-07:00"
Can anyone help me out?
Well I would say just adapt the given parameter according to the api's needs.
const todaysDate = Cypress.moment().format('YYYY-MM-DDTHH:MM:SS-07:00');
How to use in the API Call depends on how you create your call and what the concrete requirements in your case are. As your API use some kind of JSON to run, it usually might look something like this.
const date = new Date();
const timestamp = date.getTime();
const publicKey = '<publickey>';
const privateKey = '<privateKey>'
cy.request(`/v1/public/orders?ts=${timestamp}&apikey=${publicKey}&hash=${hash}`)
.then((response) => {
expect(response.body).to.have.property('offDateTime', todaysDate )
})
This snipplet would state a request to an existing File on your baseUrl or an explicit given url, using -cv.visit(url). If file referres to baseUrl from your cypress.json configuration file, that url would be [baseUrl]/v1/public/orders. The rest is like you would have set an AJAX request to this file, you'll get the response form the file and can try to catch if the expected values are in.
To retrieve a future date, you can easily relay on moment.js cypress includes automatically.
Quick example how to add future dates:
var twoDaysForward = new Cypress.moment().add(2, 'day');
document.write(twoDaysForward.format('YYYY-MM-DDTHH:MM:SS-07:00');
Same works with minutes
twoDaysForward.add(minutes, 'minutes');
My question is how retrieve data using many jsonp requests? it is practical? Currently, I'm using in my CRA this fragment of code (below)(pseudocode).
import * as fetch from 'fetch-jsonp';
import * as BlueBird from 'bluebird'; // bluebird is promise library
const getData = async () => {
const urls = ['https://...', 'https://...'] // containt about 20000 urls
const response = await BlueBird.map(urls, url => fetch(url), { concurrency: 10 })
return response
}
Working version of code creates script tags in my DOM, so as you can guess it prolongs DOM rendering, my laptop starts really heating and I end up getting error "render process gone" (on small data everything works). So what I should do? move my code to the server side and use json? or it's possible to create separate react dom and use it for jsonp? (cannot use json on client side cause cors)
Use jsoup official dependency if you are working for Android or library for java.
Then use
String html="here html";
Document doc=Jsoup.parse(html);
In the VSTS Rest API, there's a piece of documentation showing me how to create a folder. Specifically, I would like to create a folder within the Shared Queries folder. It seems like I can do this with the REST API.
I would like to do the same thing with the VSTS Node API (vso-node-api). The closest analogous function I can seem to find would be WorkItemTrackingApi.createQuery. Is this the correct function to use?
When I try to use this function, I'm getting an error:
Failed request: (405)
That seems strange, since a "Method Not Allowed" error doesn't seem like the right error here. In other words, I'm not the person deciding what method (GET/POST/...etc) to use, I'm just calling the VSTS Node API's function which should be using the correct HTTP Request Method.
I think the error code would/should be different if something about my request is wrong (like providing bad parameters/data).
But, I would not be surprised if VSTS didn't like the data I provided with the request. I wrote the following test function:
async function createQueryFolder (QueryHeirarchyItem, projectId, query) {
let result = await (WorkItemTrackingApi.createQuery(QueryHeirarchyItem, projectId, query))
return result
}
I set some variables and called the function:
let projectID = properties.project // A previously set project ID that works in other API calls
let QueryHeirarchyItem = {
isFolder: true,
name: 'Test Shared Query Folder 1'
}
try {
let result = await createQueryFolder(QueryHeirarchyFunction, projectID, '')
Notice that I provided a blank string for the query - I have no idea what to provide there when all I want to create is a folder.
So, I think a lot of things could be wrong with my approach here, but also if my request parameters are wrong maybe I should be getting a 400 error? 405 leads me to believe that the VSTS Node API is making a REST call that the underlying VSTS REST API doesn't understand.
For the third parameter of the createQueryFolder, you should specify the folder path where you want to create the new folder.
Such as if you want to create a folder Test Shared Query Folder 1 under Shared Queries, you should specify parameters for createQueryFolder as:
let result = await createQueryFolder(QueryHeirarchyFunction, projectID, 'Shared Queries')
I'm wanting to invalidate a previously cached GET from my service worker when a POST, PUT, or DELETE to the same url or any url of a resource or collection 'under' it happens, for example:
let's say I cache /subscriptions and later on I do a POST to /subscriptions to add a new subscription, or say I PUT to /subscriptions/243 to update an existing subscription.
This means that my cached subscriptions collection is now stale data and I want to delete it from my cache so the next request will go to the server.
I've thought of two options, where I'm not sure either are possible:
Can I use a Regexp in the caches.match() call?
This way I could just match the parent collection piece of the requested url with keys found in the cache.
Can I get the keys of each cached data response?
If so, I could just loop through each response and see if the key meets my criteria for deleting it.
Any ideas?
Thanks!
You can't use a RegExp or anything other than string matching (optionally ignoring query parameters) when doing a lookup via caches.match().
I'd recommend the second approach, in which you open a named cache, get its keys, and then filter for the ones you care about. It's not that much code, and looks fairly nice with await/async:
async function deleteCacheEntriesMatching(cacheName, regexp) {
const cache = await caches.open(cacheName);
const cachedRequests = await cache.keys();
// request.url is a full URL, not just a path, so use an appropriate RegExp!
const requestsToDelete = cachedRequests.filter(request => request.url.match(regexp));
return Promise.all(requestsToDelete.map(request => cache.delete(request)));
}
// Call it like:
await deleteCacheEntriesMatching('my-cache', new RegExp('/subscriptions'));