Develop Reference

DynamoDB: Query only every 10th value

I am querying data between two specific unixtime values. for example:
all data between 1516338730 (today, 6:12) and 1516358930 (today, 11:48)
my database receives a new record every minute. Now, when i want to query the data of last 24h, its way too dense. every 10th minute would be perfect.
my question now is: how can i read only every 10th database record, using DynamoDB?
As far as i know, theres no posibility to use modulo or something similar that pleases my needs.
This is my AWS Lambda Code so far:
var read = {
TableName: "user",
ProjectionExpression:"#time, #val",
KeyConditionExpression: "Id = :id and TIME between :time_1 and :time_2",
ExpressionAttributeNames:{
"#time": "TIME",
"#val": "user_data"
},
ExpressionAttributeValues: {
":id": event, // primary key
":time_1": 1516338730,
":time_2": 1516358930
},
ScanIndexForward: true
};
docClient.query(read, function(err, data) {
if(err) {
callback(err, null);
}
else {
callback(null, data.Items);
}
});
};

You say that you insert 1 record every minute?
The following might be an option:
At the time of insertion, set another field on the record, let's call it MinuteBucket, which is calculated as the timestamp's minute value mod 10.
If you do this via a stream function, you can handle new records, and then write something to touch old records to force a calculation.
Your query would change to this:
/*...snip...*/
KeyConditionExpression: "Id = :id and TIME between :time_1 and :time_2 and MinuteBucket = :bucket_id",
/*...snip...*/
ExpressionAttributeValues: {
":id": event, // primary key
":time_1": 1516338730,
":time_2": 1516358930,
":bucket_id": 0 //can be 0-9, if you want the first record to be closer to time_1, then set this to :time_1 minute value mod 10
},
/*...snip...*/
Just as a follow-up thought: if you want to speed up your queries, perhaps investigate using the MinuteBucket in an index, though that might come at a higher price.

I don't think that it is possible with dynamoDB API.
There are FilterExpression that contains conditions that DynamoDB applies after the Query operation, but before the data is returned to you.
But AFAIK it isn't possible to use a custom function. And build-in functions are poor.
As a workaround, you could mark each 10th item on the client side. And then query with checking attribute_exists (or attribute value) to filter them.
BTW, it would be nice to create the index for 'Id' attribute with sort key 'TIME' for improving query performance.

Fetch list of 50,000 most subscribed channels

I'm trying to figure out a way to grab the top 50,000 most subscribed youtube channels using javascript. These only need to be grabbed once and will be stored in a file to be used for an autocomplete input in a webpage.
I've gotten pretty close to getting the first top 50 by using search:list (/youtube/v3/search) by searching with parameters maxResults=50, order=viewCount, part=snippet, type=channel, fields=nextPageToken,items(snippet(channelId,title))
Returning:
{
"nextPageToken": "CDIQAA",
"items": [{
"snippet": {
"channelId": "UC-9-kyTW8ZkZNDHQJ6FgpwQ",
"title": "Music"
}
},{
"snippet": {
"channelId": "UC-lHJZR3Gqxm24_Vd_AJ5Yw",
"title": "PewDiePie"
}
},{
"snippet": {
"channelId": "UCVPYbobPRzz0SjinWekjUBw",
"title": "Анатолий Шарий"
}
},{
"snippet": {
"channelId": "UCam8T03EOFBsNdR0thrFHdQ",
"title": "VEGETTA777"
}
},...
Then all I'd have to do is fetch that 1000 more times using the nextPageToken to get a list of the top 50,000.
Unfortunately, sorting by relevance, rating, viewCount, or nothing is not yielding the 50 most subscribed channels, and there doesn't seem to be any sort of way to order them by subscriber count according to the documentation; so it seems like i am stuck.

Just before you writing your 50 results in file (or database), you can make one more API call, using channelId field from your result, and merge all of them with comma delimited and make another API call Channels: list.
On that page for example you can use following parameters:
(these are IDs from your example above)
part=statistics
id=UC-9-kyTW8ZkZNDHQJ6FgpwQ,UC-lHJZR3Gqxm24_Vd_AJ5Yw,UCVPYbobPRzz0SjinWekjUBw,UCam8T03EOFBsNdR0thrFHdQ`
And result will look something like this:
{
"kind": "youtube#channel",
"etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/MG6zgnd09mqb3nAdyRnPDgFwfkE\"",
"id": "UC-lHJZR3Gqxm24_Vd_AJ5Yw",
"statistics": {
"viewCount": "15194203723",
"commentCount": "289181",
"subscriberCount": "54913094",
"hiddenSubscriberCount": false,
"videoCount": "3175"
}
}
And you can take subscriberCount from result for each channel.
I know, this is not the way to sort your 50 results while writing into the file,
but with this you can sort later your results by "subscriber count" while fetching from file for your autocomplete input.
I didn't find any other way to sort results by subscriber count, so maybe this can be helpful.

The idea to do is to run a server side script, that makes RESTful api calls in a loop, and writes the results to .JSON file, to save results. For that you can create PHP script, that makes REST API call to google, and fetch first 50 results, and then use file write operations to write your results. Run that PHP script as corn job to update results at regular intervals. Executing corn job at every specific time interval you set keeps results fresh.
Hit CURL command with loop for next, to fetches 50 results every time and create temp file with all the results saved in .JSON file. Once your results are fetched, replace your old JSON file with newly created temporary file. This will generate fresh JSON file are regular, with new results if any changes are made to data.
However, the idea to use temporary file is to avoid script avoid wait/slow of AJAX down due to consistent read and write operations on same file. Once temporary file is written, simply use move command to replace the actual file.
Make sure, you use cache control headers in AJAX results to keep its freshness of data.

Querying a parse table and eagerly fetching Relations for matching

Currently, I have a table named Appointments- on appointments, I have a Relation of Clients.
In searching the parse documentation, I haven't found a ton of help on how to eagerly fetch all of the child collection of Clients when retrieving the Appointments. I have attempted a standard query, which looked like this:
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User",Parse.User.current());
query.include('Rate'); // a pointer object
query.find().then(function(appointments){
let appointmentItems =[];
for(var i=0; i < appointments.length;i++){
var appt = appointments[i];
var clientRelation = appt.relation('Client');
clientRelation.query().find().then(function(clients){
appointmentItems.push(
{
objectId: appt.id,
startDate : appt.get("Start"),
endDate: appt.get("End"),
clients: clients, //should be a Parse object collection
rate : appt.get("Rate"),
type: appt.get("Type"),
notes : appt.get("Notes"),
scheduledDate: appt.get("ScheduledDate"),
confirmed:appt.get("Confirmed"),
parseAppointment:appt
}
);//add to appointmentitems
}); //query.find
}
});
This does not return a correct Clients collection-
I then switched over to attempt to do this in cloud code- as I was assuming the issue was on my side for whatever reason, I thought I'd create a function that did the same thing, only on their server to reduce the amount of network calls.
Here is what that function was defined as:
Parse.Cloud.define("GetAllAppointmentsWithClients",function(request,response){
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User", request.user);
query.include('Rate');
query.find().then(function(appointments){
//for each appointment, get all client items
var apptItems = appointments.map(function(appointment){
var ClientRelation = appointment.get("Clients");
console.log(ClientRelation);
return {
objectId: appointment.id,
startDate : appointment.get("Start"),
endDate: appointment.get("End"),
clients: ClientRelation.query().find(),
rate : appointment.get("Rate"),
type: appointment.get("Type"),
notes : appointment.get("Notes"),
scheduledDate: appointment.get("ScheduledDate"),
confirmed:appointment.get("Confirmed"),
parseAppointment:appointment
};
});
console.log('apptItems Count is ' + apptItems.length);
response.success(apptItems);
})
});
and the resulting "Clients" returned look nothing like the actual object class:
clients: {_rejected: false, _rejectedCallbacks: [], _resolved: false, _resolvedCallbacks: []}
When I browse the data, I see the related objects just fine. The fact that Parse cannot eagerly fetch relational queries within the same call seems a bit odd coming from other data providers, but at this point I'd take the overhead of additional calls if the data was retrieved properly.
Any help would be beneficial, thank you.

Well, in your Cloud code example - ClientRelation.query().find() will return a Parse.Promise. So the output clients: {_rejected: false, _rejectedCallbacks: [], _resolved: false, _resolvedCallbacks: []} makes sense - that's what a promise looks like in console. The ClientRelation.query().find() will be an async call so your response.success(apptItems) is going to be happen before you're done anyway.
Your first example as far as I can see looks good though. What do you see as your clients response if you just output it like the following? Are you sure you're getting an array of Parse.Objects? Are you getting an empty []? (Meaning, do the objects with client relations you're querying actually have clients added?)
clientRelation.query().find().then(function(clients){
console.log(clients); // Check what you're actually getting here.
});
Also, one more helpful thing. Are you going to have more than 100 clients in any given appointment object? Parse.Relation is really meant for very large related collection of other objects. If you know that your appointments aren't going to have more than 100 (rule of thumb) related objects - a much easier way of doing this is to store your client objects in an Array within your Appointment objects.
With a Parse.Relation, you can't get around having to make that second query to get that related collection (client or cloud). But with a datatype Array you could do the following.
var query = new Parse.Query(Appointment);
query.equalTo("User", request.user);
query.include('Rate');
query.include('Clients'); // Assumes Client column is now an Array of Client Parse.Objects
query.find().then(function(appointments){
// You'll find Client Parse.Objects already nested and provided for you in the appointments.
console.log(appointments[0].get('Clients'));
});

I ended up solving this using "Promises in Series"
the final code looked something like this:
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User",Parse.User.current());
query.include('Rate');
var appointmentItems = [];
query.find().then(function(appointments){
var promise = Parse.Promise.as();
_.each(appointments,function(appointment){
promise = promise.then(function(){
var clientRelation = appointment.relation('Clients');
return clientRelation.query().find().then(function(clients){
appointmentItems.push(
{
//...object details
}
);
})
});
});
return promise;
}).then(function(result){
// return/use appointmentItems with the sub-collection of clients that were fetched within the subquery.
});
You can apparently do this in parallel, but that was really not needed for me, as the query I'm using seems to return instantaniously. I got rid of the cloud code- as it didnt seem to provide any performance boost. I will say, the fact that you cannot debug cloud code seems truly limiting and I wasted a bit of time waiting for console.log statements to show themselves on the log of the cloud code panel- overall the Parse.Promise object was the key to getting this to work properly.

How to access Xively datapoints history with javascript?

I am new to Xively. Now I am trying to access the datapoints history from the feed that I get.
From this documentation: http://xively.github.io/xively-js/docs/ It seems that I can use the method xively.datapoint.history(feedID, datastreamID, options{}, callback(data)) but I don't know how to use it.
I know the parameter feedID, datastreamID, but I am not sure about the options...
from Xively site https://xively.com/dev/docs/api/quick_reference/historical_data/, I think I should put start and end parameter. I used feed id:40053 and datastream id:airpressure. You can try to input the feed id here to get more info about it:http://xively.github.io/xively-js/demo/
I tried the code below but its not working. Am I doing something wrong, or the datapoints history itself is restricted and cant be accessed?
// Make sure the document is ready to be handled
$(document).ready(function($) {
// Set the Xively API key (https://xively.com/users/YOUR_USERNAME/keys)
xively.setKey("yWYxyi3HpdqFCBtKHueTvOGoGROSAKxGRFAyQWk5d3JNdz0g" );
// Replace with your own values
var feedID = 40053;
var datastreamID = "airpressure"; // Datastream ID
// Get datastream data from Xively
xively.datapoint.history(feedID, datastreamID,
{
start:"2013-09-10T00:00:00.703576Z",
end:"2013-10-10T00:00:00.703576Z"
},
function(data){
//data.forEach(function(datapoints){document.write(JSON.stringify(datapoints["value"], null, 4));});
document.write(JSON.stringify(data, null, 4));
});
});

I didn't read the documentation right...
The maximal duration for each query is 6 hrs, so changing the end time to "2013-09-10T06:00:00.703576Z solved my problem.

You can use parameters: duration, interval
xively.datapoint.history (feedID, datastreamID1, **{ duration: "14days", interval: "1000"}**,
function(data){
document.write(JSON.stringify(data, null, 4));
}
);

Alvinadi,
That's correct. The other thing you could do is set the interval parameter to something greater than 0. This will reduce the density of datapoints and only return one datapoint for every number of seconds specified in the interval. However, this can be useful when trying to retrieve the average of large amounts of data.
Here is the API documentation explaining the available intervals: https://xively.com/dev/docs/api/quick_reference/historical_data/
Pro tip: Set the parameter limit=1000 to return the maximum number of results and not have to paginate through data.

Sorting, updating and presenting async response data in a HTML table

I've got an app that issues multiple async calls to a backend; each async response will be an array of {date, string} values.
I'd like to present the responses of each of these async calls as they arrive, in a sorted table in my web browser (ordered by date). As subsequent responses to the other async calls arrive, I want the sorted table to be updated in real time.
For example, the first async response might be [{'2012-02-10', 'ABC'}, {'2012-01-19', 'DEF'}, {'2012-03-01', 'GHI'}], and I'd want to see something like <table><tr><td>2012-01-19</td><td>DEF</td></tr><tr><td>2012-02-10</td><td>ABC</td></tr><tr><td>2012-03-01</td><td>GHI</td></tr></table> rendered in my browser immediately.
If a second async response of [{'2011-12-09', 'JKL'}] appears some time later, I want the HTML to then be changed to <table><tr><td>2011-12-09</td><td>JKL</td></tr><tr><td>2012-01-19</td><td>DEF</td></tr><tr><td>2012-02-10</td><td>ABC</td></tr><tr><td>2012-03-01</td><td>GHI</td></tr></table>
The effect I'm after is to have a table of results in my browser, being progressively updated with sorted data as the various async responses turn up. The table would grow in size as each new async response is received, as the new {date, string} elements appear in the correct date sequence with the data that's already being displayed.
This probably isn't a unique scenario, so I'm guessing there's a way of doing this that would be considered 'best practice' - possibly using jQuery UI. If so, could someone please point me towards a 'howto' or example? - I haven't been able to find anything suitable using Google.
Thanks in advance

You can start with this
<script>
function my_func()
{
var myarray = ["2012-02-10, ABC", "2012-01-19, DEF", "2012-03-01, GHI"];
var table_content = '';
var temp_arr = '';
for(var i in myarray)
{
myarray.sort();
temp_arr = myarray[i].split(',');
table_content = table_content + '<tr><td>'+temp_arr[1]+'</td><td>'+temp_arr[0]+'</td></tr>'
}
table_content = '<table>'+table_content+'</table>'
document.getElementById('mytable').innerHTML = table_content;
}
</script>
<input type="button" value="clickme" onclick="my_func();"/>
<div id="mytable"></div>
In place of calling that function on button click you can run that function on sucessful call of your function and then fill up the html.
I hope this helps