I have a lot of files that I need to upload to my database, I am doing this through the browser and the server is having sizing issues with the data that is being sent to it and it rejects it. The server is not under my control in this context so I need to break up this object(s) into smaller objects before sending them to the server to get inserted into the database (rethinkdb if it matters). There is no order they need to be inserted in I just need to break up the file into smaller pieces.
I know I need a loop, but I do not understand how I can loop through 100 objects, save them as another object send them to the server, then start from 101 and so on....
Looping through all objects I get but this I cannot figure out or find examples for.
The code below is working for its purpose however it only goes to 250 then stops, it doesn't go past that.....
var lineData = jsonData.theData;
var newData = [];
var newLine;
//Shows on the browser console how many objects there are in the data
console.log(jsonData.theData.length);
//Goes through the Data and grabs the first 1000 lines
for(var i = 0; i< 250; i++){
newLine = lineData[i]
newData.push(newLine);
console.log(lineData[i]);
}
All you need to do is loop through your objects and for every 100th object, push out your new array and start over. So something like:
var lineData = jsonData.theData;
var newData = [];
for (var i=0; i < lineData.length; i++) {
newLine = lineData[i];
newData.push(newLine);
if (i && !(i % 100)) {
// upload newData
newData = [];
}
}
// check if there's any data left
if (newData.length) {
// upload the remaining data
}
Related
So I have this script that connects to a video service api and I am trying to push the video titles that are served up in the ajax request in the order of the video_codes array. So these video codes are for seperate videos that give out information such as titles from the api (3t7l1ok4ba gets alicia, 5cj3ro2ujq gets drew, etc.)
// these codes connect to the api and get data
var video_codes = [
'3t7l1ok4ba',
'5cj3ro2ujq',
'ztodsfeifx',
'u4g78mwyee'
]
// this is to hold the video titles provided in the api
var names = [];
// for loop makes the different requests
for (i = 0; i < video_codes.length; i++){
(function(i) {
var video_request = new XMLHttpRequest()
video_request.open('GET', 'https://api.wistia.com/v1/medias/'+video_codes[i]+'.json?api_password=', true)
video_request.onload = function() {
var video = JSON.parse(this.response)
var video_name = video.name;
// add to names array
names.push(video_name);
// console log names array 4 times
console.log(names)
}
video_request.send()
})(i);
}
the script works in the sense that it pushes the names to the name array and console logs but they dont correspond to the order of the video_codes array.
So for example 3t7l1ok4ba pushes out alicia but sometimes alicia will be the second, third, or fourth entry. It is completely random i get different results everytime I run the script. Any help would be greatly appreciated!
The problem is the order the console logs is the order that the api responds. You are making all requests at the same time, so the second request may return before the first. One way to maintain the order is doing the next request only after the previous returned:
function loadVideo(i) {
if (i < video_codes.length) {
var video_request = new XMLHttpRequest()
video_request.open('GET', 'https://api.wistia.com/v1/medias/'+video_codes[i]+'.json?api_password=', true)
video_request.onload = function() {
var video = JSON.parse(this.response)
var video_name = video.name;
// add to names array
names.push(video_name);
// console log names array 4 times
console.log(names);
loadVideo(i + 1); // Load next only after response received
}
video_request.send()
}
}
loadVideo(0); // Load first video
I am new to JS and I have a program (written in the Common Workflow Language) that takes in two arrays, one of files one of strings. The arrays are initially the same length:
var headers = ["Header1", "Header2", "Header3"];
var files = [file1, file2, file3];
Each header corresponds to a particular file so Header1 -> file1, Header2 -> file2, Header3 -> file3. I have a step in my workflow that splits the files by a certain number of lines into an array of split files, so now I have an array of arrays of files like so:
var files = [[01.file1, 02.file1, 03.file1], [01.file2, 02.file2],
[01.file3, 02.file3, 03.file3, 04.file3, 05.file3]];
Since the files are split by a certain number of lines, I have no way of knowing beforehand how many times the file is going to be split (I am processing a lot of files at a time).
Basically, what I am trying to do now, is duplicate each header equal to the length of each subarray, so Header1 should be duplicated 3 times, Header2 should be duplicated twice, and Header3 should be duplicated 5 times.
I have seen a few posts on how to duplicate a single string into an array of known length that use the var arrayNew = new Array(int).fill(x) method, but this seems to statically assign a length to an array and fill it with a single value.
I am trying to duplicate the headers strings by the length of the files array of arrays. Here is what I have so far:
var dupStrings = [];
for (var i = 0; i < files.length; i++) {
var arrLen = files[i].length;
dupStrings.push(headers[i].repeat(arrLen));
}
But this is currently giving me:
var headers = ["Header1Header1Header1", "Header2Header2",
"Header3Header3Header3Header3Header3"]
I now understand why that is happening, but I don't know how to fix it. I would like it to be:
var headers = [["Header1", "Header1", "Header1"], ["Header2", "Header2"],
["Header3", "Header3", "Header3", "Header3", "Header3"]]
Your current issue occurs, because headers[i].repeat(arrLen) returns string, whereas you need an array, you may do dupStrings.push(Array(arrLen).fill(headers[i])) instead.
However, there's the other way around:
const headers = ["Header1", "Header2", "Header3"],
files = [[01.file1, 02.file1, 03.file1], [01.file2, 02.file2], [01.file3, 02.file3, 03.file3, 04.file3, 05.file3]],
result = headers.map((h,i) => Array.from({length: files[i].length}, () => h))
console.log(result)
Try map and fill:
var headers = ["Header1", "Header2", "Header3"];
var files = [["01.file1", "02.file1", "03.file1"], ["01.file2", "02.file2"],
["01.file3", "02.file3", "03.file3", "04.file3", "05.file3"]];
var dupStrings = headers.map(function(element, index) {
return Array(files[index].length).fill(element);
});
console.log(dupStrings)
Your for-loop is not correct, you have to iterate on files[i] in order to create an array because repeat is creating a string. The answer without using map of fill (it may be difficult for you, a newbie to understand how map and fill work) is :
for (let i = 0; i < files.length; i++) {
for (let j = 0; j < files[i].length; j++) {
dupStrings.push(headers[i]);
}
headers[i] = dupStrings;
dupStrings = [];
}
I wrote a script that compares rows between two sheets and deletes all matching rows on the second sheet (named 'temp'). I set the loop to start at the end of temp and decrement, working toward the top. The script works but it ignores the bottom two rows on 'temp'...how can I fix this? I want to ensure it will delete the bottom two rows on temp when they match the data set from the other sheet.
I have confirmed that the bottom two rows are in fact duplicates and should be caught by the script and deleted.
Script:
function trimTempSheet() {
var ss, s, s1, dt;
var dirname='X DIR'
var fs, f, fls, fl, name;
var ncols=1,i, newRows, rw;
ss=SpreadsheetApp.getActiveSpreadsheet();
s=ss.getSheetByName('Report Results');
name = 'temp';
//Load current sheet to compare
var currentDataSet = s.getRange("A:S").getValues(); //Ignore final columns
var newSheet = ss.getSheetByName(name);
//Load imported data to compare
var newData = newSheet.getRange("A:S").getValues();
var headers = newData.shift();
//Create empty array to store data to be written [to add later]
newRows=[];
//Compare data from newData with current data
for(var i = newData.length-1; i > 0; --i)
{
for(var j in currentDataSet)
{
if(newData[i].join() == currentDataSet[j].join() )
{
newSheet.deleteRow(i);
}
}
}
How about this modification?
From :
for(var i = newData.length-1; i > 0; --i)
To :
for(var i = newData.length-1; i >= 0; --i)
Note :
About "the bottom two rows on 'temp'",
By above modification, one loop is added to the for loop.
In your script, the first element of newData is removed by var headers = newData.shift();. By this, newData decreases one element.
For example, how about also removing this line?
If this modification was not useful for you, I'm sorry. At that time, can you show us your sample spreadsheet?
I do not know the optimal way to force the script to look at and delete the last row first, without skipping it. So I changed the delete line to "newSheet.deleteRow(i+1);"
Will this produce any unintended row deletes based on my iteration loops? I am not an expert at how google scripts handles arrays. I assume the loops will examine rows in newData (the sheet named 'temp) sequentially, from the last row to the first. In which case my solution would be acceptable. But I am not certain of this.
Using JavaScript I would like to offset the elements in a typed array so that if the original array had values of 0 to 99, the new array would start at 10 and go to 99 leaving 10 empty elements for new data.
So if the original array can be viewed as a 10 x 10 grid, I would like to move all data up one row and then enter new data in the bottom row.
I know this can be done using a loop but this method would be too slow for my project which has a much larger array (990 x 1920).
I've tried ArrayBuffers and got nowhere.
The problem with the following test method (using subarray) is that although data1 size is specified as 100 it appears to reduce down to the subarray size when applied. I can then find no way to add further data at the end.
function initialize() {
data = new Uint32Array(100);
data1 = new Uint32Array(100);
for (var i = 0; i < data.length; i++) {
data[i] = i;
}
data1 = data.subarray(10);
console.log(data1);
}
Is there any way other than a loop to offset data in a typed array and add data at the end.
OK, I found a solution at Typed Arrays in Gecko 2: Float32Array concatenation and expansion using Set().
In JavaScript, a typed array is a fixed-length data structure which is based on ArrayBuffer i.e. a pre-allocated piece of memory anyway. Because of that, typed arrays do not have variable-length methods like push, pop etc.
So in order to offset a typed array you only have two options.
Preallocate a lot of memory in advance and shift the 'typed array', which is a actually a view over a memory block
var SIZE=100;
var SHIFT=10;
var buffer = new ArrayBuffer(100000); // preallocate a lot of memory
var data = new Uint32Array(buffer, 0, SIZE);
for (var i = 0; i < SIZE; i++) {
data[i] = i;
}
var data1 = new Uint32Array(buffer, Uint32Array.BYTES_PER_ELEMENT*SHIFT, SIZE)
data1[90]=100; //set the 101st (91st) element
console.log('data1', data1);//10,11,...98,99,100,0,0,0,0,0,0,0,0,0
Copy the slice of the old data into a new memory area.
var SIZE=100;
var SHIFT=10;
var data = new Uint32Array(SIZE);
for (var i = 0; i < SIZE; i++) {
data[i] = i;
}
var data1 = new Uint32Array(SIZE)
data1.set(data.subarray(SHIFT));
data1[90]=100; //set the 101st (91st) element
console.log('data1', data1); //10,11,...98,99,100,0,0,0,0,0,0,0,0,0
This is a classic space-time tradeoff.
The first option takes more memory but less cpu cycles, the second option is the other way round.
I'm new to javascript, and I'm having trouble figuring out how to loop through some code so that it will basically create an array that I can then pass on to my plot variable.
I'm not really sure where to start. Right now I have a chunk of code that takes my first dataset (dataOne) and formats it so that it can go into my plot variable. I basically need to do that three more times for the other data sets - hoping to include the example.getDataSets function to loop through somehow.
Is there a good way to do this?
Here is my code:
script.js
var example = {};
example.data = {
dataOne: {data: [{"date":1333238400000,"data":23},{"date":1333324800000,"data":37},{"date":1333411200000,"data":49},{"date":1333497600000,"data":54},{"date":1333584000000,"data":30},{"date":1333670400000,"data":19},{"date":1333756800000,"data":15},{"date":1333843200000,"data":19},{"date":1333929600000,"data":145}]},
dataTwo: {data: [{"date":1335830400000,"data":63},{"date":1335916800000,"data":77},{"date":1336003200000,"data":66}]},
dataThree: {data: [{"date":1341100800000,"data":24},{"date":1341187200000,"data":50},{"date":1341273600000,"data":43},{"date":1341360000000,"data":39},{"date":1341446400000,"data":56},{"date":1341532800000,"data":66}]},
dataFour: {data: [{"date":1333238400000,"data":71},{"date":1333324800000,"data":46},{"date":1333411200000,"data":66},{"date":1333497600000,"data":73},{"date":1333584000000,"data":105},{"date":1333670400000,"data":84}]}
}
example.getDataSets = function(){
return ['dataOne', 'dataTwo', 'dataThree', 'dataFour']
}
example.getSeries = function(month){
return example.data[month]
}
example.processData = function(data){
var newData = []
for(var i = 0; i < data.length; i++){
newData.push([data[i].date, data[i].data])
};
return newData;
}
My script in the HTML page:
$.getScript("script.js")
.done(function() {
var b = example.getSeries('dataOne');
var d = example.processData(b.data);
// first correct the timestamps - they are recorded as the daily
// midnights in UTC+0100, but Flot always displays dates in UTC
// so we have to add one hour to hit the midnights in the plot
for (var i = 0; i < d.length; ++i)
d[i][0] += 60 * 60 * 1000;
var plot = $.plot($("#placeholder"), [d] , options);
Any suggestions are much appreciated!
Loop over the value returned from .getDataSets, too
var sets = example.getDataSets(), set_i = 0, // get list of sets
b = {}, d = [], i = 0, plot; // set-up vars
for(; set_i < sets.length; set_i++){ // loop over each set
b = example.getSeries( sets[ set_i ] ); // get your set
d = example.processData(b.data);
for (i = 0; i < d.length; ++i)
d[i][0] += 60 * 60 * 1000;
plot = $.plot($("#placeholder"), [d] , options);
// ... etc with plot
}
If you want them all in one array before plotting, use concat on another array (call it d_total or something) and then in the set loop, d_total = d_total.concat( d );, then plot outside loop
I'm assuming you want to process each of
var b = example.getSeries(X);
where X is, in turn, 'dataOne', 'dataTwo', 'dataThree', 'dataFour'?
If so, you'll be looking for something like this:
// see below for where example.ordering is suggested
for (var i in example.ordered) {
var month = example.ordered[i];
var b = example.getSeries(month);
var d = example.processData(b.data);
... further processing with month, b, d.
}
The getDataSets() is fine for yielding your 'months' in what must be proper order for you. My only concern is that you have the list of months in two places. The array you have is necessary for ordering the months, because you cannot extract the 'month names' from example.data and expect them to be in the same order (keys are essentially stored randomly due to hashing).
The qualification would be if your keys are sortable, but the keys you have here are not. Of course, if you're building the pair of structures from the same source, then storing the data in the associated array (example.data object) along with a separate array to indicate ordering is both common and acceptable. Ideally you'd pair these two into yet another object so they can be handled as a team.
Since your array of ordered months is literally 'lost' in a function, I would suggest you add something like
example.ordering = [ ..... ]
and even better, push the months onto the end of that list in the same order they are put into the hash. I would be using a helper function to store the data:
example.ordering = []; // initially empty
example.storeSet = function (month, dataList) {
example.ordering.push(month);
example.data[month] = { data : dataList };
}
....
// now store the data
example.storeSet('dataOne', [{"date":1333238400000,"data":23}, ....]);
example.storeSet('dataTwo', [{"date":1335830400000,"data":63}, ....]);
example.storeSet('dataThree', [{"date":1341100800000,"data":24}, ....]);
example.storeSet('dataFour', [{"date":1333238400000,"data":71}, ....]);
Of course my code is untested (of course!) but it looks right, and I'm sure you can fix any stupid mistakes I might have made.
I am new and understand your frustration when you have sat and looked and still cannot find the answer. However, what I think you are looking for is something like this:
for (example.data=dataOne;example.data<dataFour;example.data=example.data+increment)
{
code to be executed
}
or maybe even recursively but I am not the best at that but below is an standard example
function recurs(p1) {
p1--;
if (p1 < 0) return // exit condition
setTimeout("recurs("+p1+")",1);
}