Develop Reference

API call to youtube.videos.list failed with error

When I run the following JavaScript through Google Apps script with more then 100 keywords.
function youTubeSearchResults() {
// 1. Retrieve values from column "A".
const sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
const values = sheet.getRange("A2:A" + sheet.getLastRow()).getDisplayValues().filter(([a]) => a);
// 2. Retrieve your current values.
const modifyResults = values.flatMap(([keywords]) => {
const searchResults = YouTube.Search.list("id, snippet", { q: keywords, maxResults: 10, type: "video", order: "viewCount", videoDuration: "short", order: "date" });
const fSearchResults = searchResults.items.filter(function (sr) { return sr.id.kind === "youtube#video" });
return fSearchResults.map(function (sr) { return [keywords, sr.id.videoId, `https://www.youtube.com/watch?v=${sr.id.videoId}`, sr.snippet.title, sr.snippet.publishedAt, sr.snippet.channelTitle, sr.snippet.channelId, `https://www.youtube.com/channel/${sr.snippet.channelId}`, sr.snippet.thumbnails.high.url] });
});
// 3. Retrieve viewCounts and subscriberCounts.
const { videoIds, channelIds } = modifyResults.reduce((o, r) => {
o.videoIds.push(r[1]);
o.channelIds.push(r[6]);
return o;
}, { videoIds: [], channelIds: [] });
const limit = 50;
const { viewCounts, subscriberCounts } = [...Array(Math.ceil(videoIds.length / limit))].reduce((obj, _) => {
const vIds = videoIds.splice(0, limit);
const cIds = channelIds.splice(0, limit);
const res1 = YouTube.Videos.list(["statistics"], { id: vIds, maxResults: limit }).items.map(({ statistics: { viewCount } }) => viewCount);
const obj2 = YouTube.Channels.list(["statistics"], { id: cIds, maxResults: limit }).items.reduce((o, { id, statistics: { subscriberCount } }) => (o[id] = subscriberCount, o), {});
const res2 = cIds.map(e => obj2[e] || null);
obj.viewCounts = [...obj.viewCounts, ...res1];
obj.subscriberCounts = [...obj.subscriberCounts, ...res2];
return obj;
}, { viewCounts: [], subscriberCounts: [] });
const ar = [viewCounts, subscriberCounts];
const rr = ar[0].map((_, c) => ar.map(r => r[c]));
// 4. Merge data.
const res = modifyResults.map((r, i) => [...r, ...rr[i]]);
// 5. Put values on Spreadsheet.
sheet.getRange(2, 2, res.length, res[0].length).setValues(res);
}
it gives me that error
GoogleJsonResponseException: API call to youtube.videos.list failed with error:
The request cannot be completed because you have exceeded your quota.
reduce.viewCounts #code.gs:23
youTubeSearchResults #code.gs:20
I know YouTube have data call limits for example you can call the results of not more then 50 video ids at one time but if you have 1000 video ids in your sheet you can run then loop for first 50 then next so on. Is it anything like that I can do with search results too.
Please help me understand how can I fix this issue.

Note that the endpoint the most expensive in your script is the Search: list one which costs 100 of your 10,000 quota (you can have a look to other endpoint costs here).
You may be interested in the standalone quota-free solution that consists in reverse-engineering the YouTube UI search feature.
Otherwise a temporary solution to Google audit consists in using my no-key service.
With my no-key service:
const searchResults = YouTube.Search.list("id, snippet", { q: keywords, maxResults: 10, type: "video", order: "viewCount", videoDuration: "short", order: "date" });
would become:
const searchResults = JSON.parse(UrlFetchApp.fetch(`https://yt.lemnoslife.com/noKey/search?part=snippet&q=${keywords}&maxResults=10&type=video&order=viewCount&videoDuration=short`).getContentText())
As part=id doesn't add more data to the response and AFAIK using two order isn't supported by YouTube Data API v3.

axios: how to merge transformRequest

axios.defaults.transformRequest = [
(data, headers) => {
console.log('default config')
// ...some code
return qs.stringify(data)
}
]
function postData(value) {
axios.post('/xxx', value, {
transformRequest: [data => {
console.log('postData config')
const { aProp, ...rest } = data
if (isX(aProp)) {
return { ...rest, x: aProp }
}
return { ...rest, y: aProp }
}]
})
}
Expected output:
postData config
default config
But only output postData config.
The transformRequest seems like merged by index, not using concat. So how can I make both functions called?

Seems that's not possible like the way you written, as Axios is just overriding the transformRequest property of global config by request specific config.
Please take a look at the code that is in mergeConfig.js
in below code snippets
config2 is request specific config
config1 is global config
var valueFromConfig2Keys = ['url', 'method', 'data'];
var mergeDeepPropertiesKeys = ['headers', 'auth', 'proxy', 'params'];
var defaultToConfig2Keys = [
'baseURL', 'transformRequest', 'transformResponse', 'paramsSerializer',
'timeout', 'timeoutMessage', 'withCredentials', 'adapter', 'responseType', 'xsrfCookieName',
'xsrfHeaderName', 'onUploadProgress', 'onDownloadProgress', 'decompress',
'maxContentLength', 'maxBodyLength', 'maxRedirects', 'transport', 'httpAgent',
'httpsAgent', 'cancelToken', 'socketPath', 'responseEncoding'
];
var directMergeKeys = ['validateStatus'];
Only mergeDeepPropertiesKeys are deep merged like here
function mergeDeepProperties(prop) {
if (!utils.isUndefined(config2[prop])) {
config[prop] = getMergedValue(config1[prop], config2[prop]);
} else if (!utils.isUndefined(config1[prop])) {
config[prop] = getMergedValue(undefined, config1[prop]);
}
}
utils.forEach(mergeDeepPropertiesKeys, mergeDeepProperties);
defaultToConfig2Keys are just overriden by request specific config like below and transformRequest is one of those properties.
utils.forEach(defaultToConfig2Keys, function defaultToConfig2(prop) {
if (!utils.isUndefined(config2[prop])) {
config[prop] = getMergedValue(undefined, config2[prop]);
} else if (!utils.isUndefined(config1[prop])) {
config[prop] = getMergedValue(undefined, config1[prop]);
}
});
PS: I would suggest you to write your own wrapper around axios or you can prepare your data ahead before making the request.

How do I display Inline images and Attachments from Gmail API in HTML -- Last part of the puzzle

I have a 'message' object, the response from the Gmail API, in my frontend and adapting this very useful Javascript gist I've parsed the message as well.
function indexHeaders(headers) {
if (!headers) {
return {};
} else {
return headers.reduce(function (result, header) {
result[header.name.toLowerCase()] = header.value;
return result;
}, {});
}
}
function urlB64Decode(string) {
encodedBody = string
encodedBody = encodedBody.replace(/-/g, '+').replace(/_/g, '/').replace(/\s/g, '');
return decodeURIComponent(escape(window.atob(encodedBody)));
}
function parseMessage(response) {
var result = {
id: response.id,
threadId: response.threadId,
labelIds: response.labelIds,
snippet: response.snippet,
historyId: response.historyId
};
if (response.internalDate) {
result.internalDate = parseInt(response.internalDate);
}
var payload = response.payload;
if (!payload) {
return result;
}
var headers = indexHeaders(payload.headers);
result.headers = headers;
var parts = [payload];
var firstPartProcessed = false;
while (parts.length !== 0) {
var part = parts.shift();
if (part.parts) {
parts = parts.concat(part.parts);
}
if (firstPartProcessed) {
headers = indexHeaders(part.headers);
}
if (!part.body) {
continue;
}
var isHtml = part.mimeType && part.mimeType.indexOf('text/html') !== -1;
var isPlain = part.mimeType && part.mimeType.indexOf('text/plain') !== -1;
var isAttachment = headers['content-disposition'] && headers['content-disposition'].indexOf('attachment') !== -1;
var isInline = headers['content-disposition'] && headers['content-disposition'].indexOf('inline') !== -1;
if (isHtml && !isAttachment) {
result.textHtml = urlB64Decode(part.body.data);
} else if (isPlain && !isAttachment) {
result.textPlain = urlB64Decode(part.body.data);
} else if (isAttachment) {
var body = part.body;
if(!result.attachments) {
result.attachments = [];
}
result.attachments.push({
filename: part.filename,
mimeType: part.mimeType,
size: body.size,
attachmentId: body.attachmentId,
headers: indexHeaders(part.headers)
});
} else if (isInline) {
var body = part.body;
if(!result.inline) {
result.inline = [];
}
result.inline.push({
filename: part.filename,
mimeType: part.mimeType,
size: body.size,
attachmentId: body.attachmentId,
headers: indexHeaders(part.headers)
});
}
firstPartProcessed = true;
}
return result;
};
So, after parsing the response, it's now in this format:
{
// id: '{MESSAGE_ID}',
// threadId: '{THREAD_ID}',
// labelIds: [ 'SENT', 'INBOX', 'UNREAD' ],
// snippet: 'This is one cool message, buddy.',
// historyId: '701725',
// internalDate: 1451995756000,
// attachments: [{
// filename: 'example.jpg',
// mimeType: 'image/jpeg',
// size: 100446,
// attachmentId: '{ATTACHMENT_ID}',
// headers: {
// 'content-type': 'image/jpeg; name="example.jpg"',
// 'content-description': 'example.jpg',
// 'content-transfer-encoding': 'base64',
// 'content-id': '...',
// ...
// }
// }],
// inline: [{
// filename: 'example.png',
// mimeType: 'image/png',
// size: 5551,
// attachmentId: '{ATTACHMENT_ID}',
// headers: {
// 'content-type': 'image/jpeg; name="example.png"',
// 'content-description': 'example.png',
// 'content-transfer-encoding': 'base64',
// 'content-id': '...',
// ...
// }
// }],
// headers: {
// subject: 'Example subject',
// from: 'Example Name <example#gmail.com>',
// to: '<foo#gmail.com>, Foo Bar <fooBar#gmail.com>',
// ...
// },
// textPlain: '<div dir="ltr">hey! 😅<div><br></div><div><div><img src="cid:ii_k0l2i10d0" alt="Image_Name 2019-09-11 at 20.47.16.jpeg" width="452" height="339"><br></div></div></div>',
// textHtml: '<div dir="ltr">hey! 😅<div><br></div><div><div><img src="cid:ii_k0l2i10d0" alt="Image_Name 2019-09-11 at 20.47.16.jpeg" width="452" height="339"><br></div></div></div>'
// }
I'm having difficulty rendering the actual inline image / attachment in html though. If I use the 'textHtml' that comes through in the parsedMessage, it renders as something like <img src='cid:ii_k0l2i10d0'> in the html, which doesn't display the image.
This Stackoverflow answer was helpful but I need help with the last part. I have the attachmentID now. It says "Get the attachment from the Gmail API and replace the cid with the base64-data" which is the part I'm struggling with. How do I do that? -- Do I have to make another call to the Gmail API for this? Or am I going wrong somewhere in the decoding?
Thanks so much for any help!

I finally got it working -- so the answer is, yes, I did have to make another call to the API to get the attachments data. That returns an attachment resource that looks like this:
{
"attachmentId": string,
"size": integer,
"data": bytes
}
Once I get the response:
// construct the new src with the data from the response
newSrc = "data:" + "image/jpeg" + ";" + "base64" + "," + response.data.replace(/-/g, `+`).replace(/_/g, `/`)
// replace the src of the image with the new src
thisImg.attr({"src":newSrc});

Object properties are undefined, the object itself shows all the data though

I'm building an Single Page application for a minor project. I've been trying to save data from API call's to the Movie Database in an object. If i console.log the object, I can see all its properties and values. If I console.log the object.property, it returns 'undefined'. This is the code:
(() => {
"use strict"
/* Saving sections to variables
--------------------------------------------------------------*/
const movieList = document.getElementsByClassName('movie_list')[0];
const movieSingle = document.getElementsByClassName('movie_single')[0];
/* All standard filters for displaying movies
--------------------------------------------------------------*/
const allFilters = {
trending: 'movie/popular',
toplist: 'movie/top_rated',
latest: 'movie/now_playing',
upcoming: 'movie/upcoming'
};
const allData = {};
/* Initialize app - Get al standard data and save it in object
--------------------------------------------------------------*/
const app = {
init() {
getData(allFilters.trending, 'popular');
getData(allFilters.toplist, 'toplist');
getData(allFilters.latest, 'latest');
getData(allFilters.upcoming, 'upcoming');
this.startPage();
},
startPage() {
window.location.hash = "trending";
}
}
/* Function for getting data from the API
--------------------------------------------------------------*/
const getData = (filter, key) => {
const request = new XMLHttpRequest();
const apiKey = '?api_key=xxx';
const getUrl = `https://api.themoviedb.org/3/${filter}${apiKey}`;
request.open('GET', getUrl, true);
request.onload = () => {
if (request.status >= 200 && request.status < 400) {
let data = JSON.parse(request.responseText);
data.filter = key;
cleanData.init(data);
} else {
window.location.hash = 'random';
}
};
request.onerror = () => {
console.error('Error');
};
request.send();
};
/* Check if the data is list or single, and clean up
--------------------------------------------------------------*/
const cleanData = {
init(originalData) {
if (!originalData.results) {
this.single(originalData);
} else {
allData[originalData.filter] = originalData;
}
},
list(data) {
data.results.map(function(el) {
el.backdrop_path = `https://image.tmdb.org/t/p/w500/${el.backdrop_path}`;
});
let attributes = {
movie_image: {
src: function() {
return this.backdrop_path;
},
alt: function() {
return this.title;
}
},
title_url: {
href: function() {
return `#movie/${this.id}/${this.title}`;
}
}
}
showList(data.results, attributes);
},
single(data) {
data.poster_path = `https://image.tmdb.org/t/p/w500/${data.poster_path}`;
data.budget = formatCurrency(data.budget);
data.revenue = formatCurrency(data.revenue);
data.runtime = `${(data.runtime / 60).toFixed(1)} uur`;
data.imdb_id = `http://www.imdb.com/title/${data.imdb_id}`;
let attributes = {
movie_image: {
src: function() {
return this.poster_path;
},
alt: function() {
return this.title;
}
},
imdb_url: {
href: function() {
return this.imdb_id
}
},
similar_url: {
href: function() {
return `#movie/${this.id}/${this.title}/similar`
}
}
};
showSingle(data, attributes);
}
};
const showList = (cleanedData, attributes) => {
movieList.classList.remove('hidden');
movieSingle.classList.add('hidden');
Transparency.render(movieList, cleanedData, attributes);
};
const showSingle = (cleanedData, attributes) => {
movieSingle.classList.remove('hidden');
movieList.classList.add('hidden');
Transparency.render(movieSingle, cleanedData, attributes);
}
const formatCurrency = amount => {
amount = amount.toFixed(0).replace(/./g, function(c, i, a) {
return i && c !== "." && ((a.length - i) % 3 === 0) ? '.' + c : c;
});
return `€${amount},-`;
};
app.init();
console.log(allData); // Returns object with 4 properties: trending, toplist, latest & upcoming. Each property is filled with 20 results (movies with data) from the API.
console.log(allData.trending) // Returns 'undefined' (each property I've tried).
console.log(allData['trending']) // Returns 'undefined'
Object.keys(allData); // Returns an empty Array []
})();
When I use console.log(allData) I see 4 properties, all filled with the results from the API. But when i do console.log(allData.trending) or console.log(allData['trending']) it returns 'Undefined' in the console. Has anyone an idea how to fix this?

When you call app.init() it fires the init and sends the api call(s) to fetch data.
The call to fetch data is Asynchronous, which means it doesn't wait for the response to continue execution. So it goes ahead and executes next lines of code, which are your console.logs. At this time the API calls haven't responded with the data, so when you try to access data.property it fails as the data is not yet here.
When you do log(data) it creates a log of the reference to data, which gets updated in the log when the reference is filled with a value later. To get the value of data at that instance and prevent updating later you can try log(JSON.stringify(data)). When you do that you get a consistent result, none of your logs work, which is the actual behaviour.
To get your logs work, look into the success/load callback of you A(synchronous)JAX request, log it from there. Or if you constructed allData later in cleanData then call the logs after cleanData function.
So to answer your question, none of your logs should work as it is an asynchronous call. You are getting the allData logged due to the way console.log works with references that are updated later, use console.log(JSON.stringify(allData)) to get the actual snapshot of Object

normalizr v3 and JSON api

I want to normalise the responses I receive from an API. A typical response could look something like this:
// Get all projects
{data:[
{
id: 1
...
team:{
data: {
id:15
...
}
}
},
{
id:2,
....
},
{
id:3,
...
}
]}
How do I write my schemas so that it removes the 'data' container?
Currently, my schema looks like:
export const project = new schema.Entity('projects', {
team: team, // team omitted
},
{
processStrategy: (value, parent, key) => parent.data
}
)
export const arrayOfProjects = new schema.Array(project)
And I am using it like:
const normalizedProjects = normalize(jsonResponse, arrayOfProjects)
normalizedProjects then looks like this:
{
entities:{
projects:{
undefined:{
0:{
team:{
data:{
id:15,
...
}
}
},
1:{...},
2:{...}.
...
50:{...},
}
}
},
result:[] // length is 0
}
I'm not sure why the list of projects is contained in 'undefined', either?

I also use json_api schema.
How about like this?
const projectsSchema = new schema.Entity('projects', {}, {
processStrategy: processStrategy
});
export const processStrategy = (value, parent, key) => {
const attr = value.attributes;
delete value.attributes;
return { ...value, ...attr };
};
export const fetchProjectsSchema = {
data: [projectsSchema]
}

Each of your entity schema that you want to have the data omitted (or anything else fundamentalyl changed) needs to include a processStrategy that you write to remove or change any data. (see more examples in the tests)