Develop Reference

Javascript: sync with server

I want to sync with my server and found some code (which may be a starting point for my own):
var serverDateTimeOffset
function getServerDateTime() {
if (serverDateTimeOffset === undefined || serverDateTimeOffset === null) {
serverDateTimeOffset = 0; //while syncing with the server, do not start a new sync, but return localtime
var clientTimestamp = (new Date()).valueOf();
$.getJSON('/getDateTime.ashx?ct=' + clientTimestamp, function (data) {
var nowTimeStamp = (new Date()).valueOf();
var serverTimestamp = data.serverTimestamp;
var serverClientRequestDiffTime = data.diff;
var serverClientResponseDiffTime = nowTimeStamp - serverTimestamp;
var responseTime = (serverClientRequestDiffTime - nowTimeStamp + clientTimestamp - serverClientResponseDiffTime) / 2
var syncedServerTime = new Date((new Date()).valueOf() - (serverClientResponseDiffTime - responseTime) / 2);
serverDateTimeOffset = (serverClientResponseDiffTime - responseTime) / 2;
console.log("synced time with server:", syncedServerTime, serverDateTimeOffset);
})
}
return new Date((new Date()).valueOf() - serverDateTimeOffset);
}
But I do not understand what is the meaning of the variable responseTime (and the logical reasoning behind it). What does it represent? Anyone has suggestions? It appears that it combines some relative with absolute times.
Example in my reasoning:
10 start at client ==+5 (req in transit) ==> 15 (arrival) ==> (server processing +5) ==> before sending (20) ==+7 (resp in transit) ==> 27 (back at client). Calculation for responseTime would result in -19/2 = -9.5.
When receive time is same as sent time at server (server processing would be 0) then 5 -22 + 10 -7 responseTime resulsts in -14/2 = -7.
And what about the variable serverDateTimeOffset?
Do I miss something? I found a description of NTP which may be helpful:
Synchronizing a client to a network server consists of several packet exchanges where each exchange is a pair of request and reply. When sending out a request, the client stores its own time ( ) into the packet being sent. When a server receives such a packet, it will in turn store its own time ( ) into the packet, and the packet will be returned after putting a transmit timestamp into the packet. When receiving the reply, the receiver will once more log its own receipt time to estimate the travelling time of the packet. The travelling time (delay) is estimated to be half of "the total delay minus remote processing time", assuming symmetrical delays.
Edit:
The function is polled regularly (it is part of the update() function, so the first time it may indeed be 0)
$(document).ready(function () {
...
setTimeout(function () { update() }, 2000); }
..
});
Some tests resulted in the following examples:
responseTime= (serverClientRequestDiffTime( -578 )-nowTimeStamp( 1621632913398 )+clientTimestamp( 1621632913081 )-serverClientResponseDiffTime( 895 ))/2= -895
serverDateTimeOffset=(serverClientResponseDiffTime( 895 )-responseTime( -895 ))/2= 895
added req delay via proxy:
responseTime= (serverClientRequestDiffTime( 2456 )-nowTimeStamp( 1621632992161 )+clientTimestamp( 1621632988800 )-serverClientResponseDiffTime( 905 ))/2= -905
serverDateTimeOffset=(serverClientResponseDiffTime( 905 )-responseTime( -905 ))/2= 905

Query Instagram posts by hashtag and time range

I'm trying to query posts from Instagram by providing the hashtag and the time range (since and until dates).
I use the recent tags endpoint.
https://api.instagram.com/v1/tags/{tag-name}/media/recent?access_token=ACCESS-TOKEN
My code is written in Node.js using the instagram-node library (see the inline comments):
// Require the config file
var config = require('../config.js');
// Require and intialize the instagram instance
var ig = require('instagram-node').instagram();
// Set the access token
ig.use({ access_token: config.instagram.access_token });
// We export this function for public use
// hashtag: the hashtag to search for
// minDate: the since date
// maxDate: the until date
// callback: the callback function (err, posts)
module.exports = function (hashtag, minDate, maxDate, callback) {
// Create the posts array (will be concated with new posts from pagination responses)
var posts = [];
// Convert the date objects into timestamps (seconds)
var sinceTime = Math.floor(minDate.getTime() / 1000);
var untilTime = Math.floor(maxDate.getTime() / 1000);
// Fetch the IG posts page by page
ig.tag_media_recent(hashtag, { count: 50 }, function fetchPosts(err, medias, pagination, remaining, limit) {
// Handle error
if (err) {
return callback(err);
}
// Manually filter by time
var filteredByTime = medias.filter(function (currentPost) {
// Convert the created_time string into number (seconds timestamp)
var createdTime = +currentPost.created_time;
// Check if it's after since date and before until date
return createdTime >= sinceTime && createdTime <= untilTime;
});
// Get the last post on this page
var lastPost = medias[medias.length - 1] || {};
// ...and its timestamp
var lastPostTimeStamp = +(lastPost.created_time || -1);
// ...and its timestamp date object
var lastPostDate = new Date(lastPostTimeStamp * 1000);
// Concat the new [filtered] posts to the big array
posts = posts.concat(filteredByTime);
// Show some output
console.log('found ' + filteredByTime.length + ' new items total: ' + posts.length, lastPostDate);
// Check if the last post is BEFORE until date and there are no new posts in the provided range
if (filteredByTime.length === 0 && lastPostTimeStamp <= untilTime) {
// ...if so, we can callback!
return callback(null, posts);
}
// Navigate to the next page
pagination.next(fetchPosts);
});
};
This will start fetching the posts with the most recent to least recent ones, and manually filter the created_time.
This works, but it's very very inefficient because if we want, for example, to get the posts from one year ago, we have to iterate the pages until that time, and this will use a lot of requests (probably more than 5k / hour which is the rate limit).
Is there a better way to make this query? How to get the Instagram posts by providing the hashtag and the time range?

I think this is the basic idea you're looking for. I'm not overly familiar with Node.js, so this is all in plain javascript. You'll have to modify it to suit your needs and probably make a function out of it.
The idea is to convert an instagram id (1116307519311125603 in this example) to a date and visa versa to enable you to quickly grab a specific point in time rather then backtrack through all results until finding your desired timestamp. The portion of the id after the underscore '_' should be trimmed off as that refers, in some way, to the user IIRC. There are 4 functions in the example that I hope will help you out.
Happy hacking!
//static
var epoch_hour = 3600,
epoch_day = 86400,
epoch_month = 2592000,
epoch_year = 31557600;
//you'll need to set this part up/integrate it with your code
var dataId = 1116307519311125603,
range = 2 * epoch_hour,
count = 1,
tagName = 'cars',
access = prompt('Enter access token:'),
baseUrl = 'https://api.instagram.com/v1/tags/' +
tagName + '/media/recent?access_token=' + access;
//date && id utilities
function idToEpoch(n){
return Math.round((n / 1000000000000 + 11024476.5839159095) / 0.008388608);
}
function epochToId(n){
return Math.round((n * 0.008388608 - 11024476.5839159095) * 1000000000000);
}
function newDateFromEpoch(n){
var d = new Date(0);
d.setUTCSeconds(n);
return d;
}
function dateToEpoch(d){
return (d.getTime()-d.getMilliseconds())/1000;
}
//start with your id and range; do the figuring
var epoch_time = idToEpoch(dataId),
minumumId = epochToId(epoch_time),
maximumId = epochToId(epoch_time + range),
minDate = newDateFromEpoch(epoch_time),
maxDate = newDateFromEpoch(epoch_time + range);
var newUrl = baseUrl +
'&count=' + count +
'&min_tag_id=' + minumumId +
'&max_tag_id=' + maximumId;
//used for testing
/*alert('Start: ' + minDate + ' (' + epoch_time +
')\nEnd: ' + maxDate + ' (' + (epoch_time +
range) + ')');
window.location = newUrl;*/

To support this excellent answer, an instagram ID is generated via the plpgSQL function:
CREATE OR REPLACE FUNCTION insta5.next_id(OUT result bigint) AS $$
DECLARE
our_epoch bigint := 1314220021721;
seq_id bigint;
now_millis bigint;
shard_id int := 5;
BEGIN
SELECT nextval('insta5.table_id_seq') %% 1024 INTO seq_id;
SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
result := (now_millis - our_epoch) << 23;
result := result | (shard_id << 10);
result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;
from Instagram's blog

Despite a similar getting posts process, Data365.co Instagram API, I currently working at, seems to be more suitable and efficient. It does not have a limit of 5,000 posts per hour, and you can specify the period of time for which your need posts in the request itself. Also, the billing will be taken into account only posts from the indicated period. You won't have to pay for data you don't need.
You can see below a task example to download posts by the hashtag bitcoins for the period from January 1, 2021, to January 10, 2021.
POST request: https://api.data365.co/v1.1/instagram/tag/bitcoins/update?max_posts_count=1000&from_date=2021-01-01&to_date=2021-01-10&access_token=TOKEN
A GET request example to get the corresponding list of posts:
https://api.data365.co/v1.1/instagram/tag/bitcoins/posts?from_date=2021-01-01&to_date=2021-01-10&max_page_size=100&order_by=date_desc&access_token=TOKEN
More detailed info view in API documentation at https://api.data365.co/v1.1/instagram/docs#tag/Instagram-hashtag-search

Reading socket connection responses in Node.js

I'm in the process of converting a PHP function to JS for use with Node.
The PHP function:
takes in a partially formed packet as an arg
finalizes the packet
creates a socket
sends the packet over the socket
Reads the response code from the server
Looks up the error code in an array
Sends the error code back to the caller
I have most of the function converted except for the bit that reads the server response.
Original PHP Function
(Comments are my understanding of what the code does. May be incorrect)
function serverInteractive($buf) { // $buf = partially formed packet
$fp = fsockopen($ip, $port , $errno, $errstr, 5);
$rs = '';
if (!$fp) return $this -> fsockerror;
$packet = pack("s", (strlen($buf)+2)).$buf; // Finalizes the packet
fwrite($fp, $packet); // Sends the packet to the server.
// ----- Read Server Response START -----//
$len = unpack("v", fread($fp, 2));
$rid = unpack("c", fread($fp, 1));
for ($i = 0; $i < (($len[1] - 4) / 4); $i++) {
$read = unpack("i", fread($fp, 4));
$rs .= $read[1];
}
// ----- Read Server Response FINISH -----//
fclose($fp); // Closes the socket
$result = $this -> socketerrors[$rs];
// $socketerrors is an array of error messages.
return($result);
}
My JavaScript Version
var net = require('net');
var submitPacket = function(packet) {
// Generate Final Packet
var p = pack('s', packet.length + 2) + packet;
// Create socket to Server
var serverSocket = net.createConnection(config.serverConfig.port,
config.serverConfig.host,
function() {
// Executes of connection is OK.
console.log("Connected to Server");
if (serverSocket.write(p, function() {
console.log("Buffer Flushed!");
})) {
console.log("Packet sent ok!");
} else {
console.log("There was a problem sending the packet!")
}
});
serverSocket.on('error', function(error) {
if (error) {
console.log(error);
}
});
serverSocket.on('data', function(data) {
if (data) {
console.log("Response: ", data);
// Need to put the error code generation
// here and fire a callback.
serverSocket.end();
}
});
}
The response i get from the server looks something like this when everything is ok:
<Buffer 07 00 06 01 00 00 00>
When not ok, looks something like this:
<Buffer 0b 00 06 00 00 00 00 03 00 00 00>
Any advice would be greatly appreciated.
UPDATE 1: This is what i've come up with so far however it the resulting code is undefined.
serverdSocket.on('data', function(data) {
if (data) {
var response = toArrayBuffer(data);
var len = response.slice(0,2);
var rid = response.slice(0,1);
var rs = '';
for (var i = 0; i < ((len[1]-4) / 4); i++) {
var read = response.slice(0,4);
rs += read[1];
}
console.log("Code: ",rs);
}
UPDATE 2: The PHP unpack function does indeed convert a buffer of binary data into an array. It looks like i can do the same thing by JSON.stringify then JSON.parse() to get it into an array. I now have an array object of the correct data, but the rest of the function doesnt seem to replicate the original.

I'll give it a try, although you haven't actually said what you want the "Code:" output to look like for the two input strings. We'll start with the differences between the PHP code and the JavaScript code.
Let's talk about these lines from the PHP code:
$len = unpack("v", fread($fp, 2));
$rid = unpack("c", fread($fp, 1));
Now those little fread() function calls are actually reading data from an input stream/file/whatever. The bottom line is that $len gets its value from the first two bytes of the stream and $rid gets its value from the third byte.
Now compare with the JavaScript code:
var len = response.slice(0,2);
var rid = response.slice(0,1);
I have a few observations about this code. First, the calls to slice() are both operating on the same array-like object, starting from the same location. So the value of rid will be wrong. Second, the value for rid is never used in the sample code, so if you never really need it you can eliminate that line entirely (which you could not do in the PHP code). The third observation is that calling slice() seems like overkill. Why not just use the square brackets on response?
One final puzzle about the original PHP code. Looking at the code that builds up $rs:
$rs .= $read[1];
It looks like this is a string concatenation of the successive integer data values. It's been a few years since I worked in PHP, so maybe I'm missing some subtlety, but this seems like a kind of odd thing to do. If you could tell me what the expected output codes are supposed to be it would help.
That brings me to the data itself. Just guessing from the data examples in the PHP code, it looks like the first two bytes are a little-endian encoded 16-bit buffer length. The next byte is the rid value, which is 6 in both examples. Then it looks like the rest of the buffer is composed of 32-bit little-endian values. Of course this is a WAG, but I'm just working with what has been provided.
So here's some code that processes the data values from the two sample buffers. For the first buffer it prints Code: 1 and for the second Code: 03. If these are not the correct values, then let me know what the expected output is and I'll see if I can make it work.
function toArrayBuffer(buffer) {
var ab = new ArrayBuffer(buffer.length);
var view = new Uint8Array(ab);
for (var i = 0; i < buffer.length; ++i) {
view[i] = buffer[i];
}
return ab;
}
function serverSocketOnData(data)
{
if (data) {
var response = toArrayBuffer(data);
var len = response[0] + 256*response[1];
var rid = response[2]; // get rid of this line?
var rs = '';
for (var i = 3; i < len; i+=4) {
rs += response[i];
// if the 32-bit value could ever exceed 255, then use this instead:
// rs += response[i] + 256*response[i+1] +
// 256*256*response[i+2] + 256*256*256*response[i+3];
}
console.log("Code: ",rs);
}
}
function testArray(sourceArray)
{
var sourceBuffer = new Buffer(sourceArray);
console.log("source buffer");
console.log(sourceBuffer);
serverSocketOnData(sourceBuffer);
}
function main()
{
var sourceArray1 = [0x07,0x00,0x06,0x01,0x00,0x00,0x00];
var sourceArray2 = [0x0b,0x00,0x06,0x00,0x00,0x00,0x00,0x03,0x00,0x00,0x00];
testArray(sourceArray1);
testArray(sourceArray2);
}
main();

Are you using slice appropriately?
When you call slice on a string, it returns a string.
var hello = "hello"
var hel = hello.slice(0, 3)
var h = hel[1]
var ello = hello.slice(1)

Twitter OAuth Request_Token with Javascript, possibly wrong time?

So I'm trying and failing to get a token from twitter.
I get the following error: "Failed to validate oauth signature and token".
I have read it can be due to your system clock being wrong.
In javascript I tested my date with the following code
var minutes=1000*60;
var hours=minutes*60;
var days=hours*24;
var years=days*365;
var d=new Date();
var t=d.getTime();
var y=t/years;
console.log((y+1970) + " year and " + (t%years)/days)
This gave me the year as 2012 and 17 days..
1972
1976
1980
1984
1988
1992
1996
2000
2004
2008
__=10 leap days. Today is the 8th, so taking away leap days it appears my system clock is on the 7th? Or have I made a mistake here? If this is the problem how do I fix correct my clock?
In cmd when I do the date cmd it gives me todays date, i.e the 8th.
Here is my post request and code in case the problem lies within the code and not the clock.
My Post request is:
POST http://api.twitter.com/oauth/request_token?oauth_callback=127.0.0.1&oauth_consumer_key=FFZJrBaPLsiwTDg5159tTQ&oauth_nonce=tWHEEIW8vLS6tMggo3IXe6e449qv1GpE8LunKRsbRF&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1326039495&oauth_version=1.0&oauth_signature=d%2BQqgTzJCjYIp9vKwm%2BCWzVLPvA
which gets 401 (Unauthorized)
Here is my javascript code.
var url = "http://api.twitter.com/oauth/request_token";
var params={
oauth_callback : "127.0.0.1"
,oauth_consumer_key : "FFZJrBaPLsiwTDg5159tTQ"
,oauth_nonce : OAuth.nonce(42)
,oauth_signature_method : "HMAC-SHA1"
,oauth_timestamp : OAuth.timestamp()
,oauth_version: "1.0"}
//temp is to be the signature base string
var temp = toSignParams("POST",url,params);
console.log(temp);
//This logs the signature base string as "POST&http%3A%2F%2Fapi.twitter.com%2Foauth%2Frequest_token&oauth_callback%3D127.0.0.1%26oauth_consumer_key%3DFFZJrBaPLsiwTDg5159tTQ%26oauth_nonce%3D5gQVIa3WmwD6ARGGQTITl1Ozgxe2t8em5HC7g8wvMi%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1326038871%26oauth_version%3D1.0"
//which is correct I think.
//When I use this with the base signature from twitters oauth example page I get the result they got.
//it hashes the twitter signing key with base signature.
params.oauth_signature = b64_hmac_sha1("MCD8BKwGdgPHvAuvgvz4EQpqDAtx89grbuNMRd7Eh98&",temp);
var req = new XMLHttpRequest();
req.open("POST",toURIParams(url,params),true);
req.send();
console.log(params)
req.onreadystatechange=function(){
if (req.readyState==4)
{
console.log(req.responseText); //this is saying "Failed to validate oauth signature and token"
}
}
//function to convert to Signature paramaters, as indicated on twitter page.
function toSignParams(method,base,params){
tail=[];
for (var p in params) {
if (params.hasOwnProperty(p)) {
tail.push(p + "%3D" + encodeURIComponent(params[p]));
}
}
return method + "&" + encodeURIComponent(base) + "&" + tail.join("%26")
}
//function to convert to uri encoded parameters.
function toURIParams(base, params) {
tail = [];
for (var p in params) {
if (params.hasOwnProperty(p)) {
tail.push(p + "=" + encodeURIComponent(params[p]));
}
}
return base + "?" + tail.join("&")
}
Any ideas?

I used a library called "jsOAuth-1.3.3" as was found on the twitter list of libraries compatible with JavaScript but is no longer there.
The 2 javascript options they list now are:
user-stream by #AivisSilins — a simple Node.js User streams client
and
codebird-js by #mynetx — a Twitter Library in JScript.
as found on https://dev.twitter.com/docs/twitter-libraries
If anybody has a problem where they can't get either of those solutions to work, I can try find online or upload the library I used somewhere.

websocket handshake problem

I'm using python to implement a simple websocket server.
The handshake I'm using comes from http://en.wikipedia.org/w/index.php?title=WebSockets&oldid=372387414.
The handshake itself seems to work, but when I hit send, I get a javascript error:
Uncaught Error: INVALID_STATE_ERR: DOM Exception 11
Here's the html:
<!doctype html>
<html>
<head>
<title>ws_json</title>
</head>
<body onload="handleLoad();" onunload="handleUnload();">
<input type="text" id='input' />
<input type="button" value="submit" onclick="handleSubmit()" />
<div id="display"></div>
<script type="text/javascript">
function showmsg(str){
display = document.getElementById("display");
display.innerHTML += "<p>" + str + "</p>";
}
function send(str){
ws.send(str.length);
ws.send(str);
}
function handleSubmit(){
input = document.getElementById('input');
send(input.value);
input.focus();
input.value = '';
}
function handleLoad(){
ws = new WebSocket("ws://localhost:8888/");
ws.onopen = function(){
showmsg("websocket opened.");
}
ws.onclose = function(){
showmsg("websocket closed.");
}
}
function handleUnload(){
ws.close();
}
</script>
</body>
</html>
And here's the python code:
import socket
import threading
import json
PORT = 8888
LOCATION = "localhost:8888"
def handler(s):
print " in handler "
ip, _ = s.getpeername()
print "New connection from %s" % ip
request = s.recv(1024)
print "\n%s\n" % request
print s.getpeername()
# send response
response = "HTTP/1.1 101 Web Socket Protocol Handshake\r\n"
response += "Upgrade: WebSocket\r\n"
response += "Connection: Upgrade\r\n"
try:
peername = s.getpeername()
response += "Sec-WebSocket-Origin: http://%s\r\n" % peername[0] # % request[request.index("Origin: ")+8:-4]
except ValueError:
print "Bad Request"
raise socket.error
response += "Sec-WebSocket-Location: ws://%s\r\n" % LOCATION
response += "Sec-WebSocket-Protocol: sample"
response = response.strip() + "\r\n\r\n"
print response
s.send(response)
while True:
length = s.recv(1)
print length
if not length:
break
length = int(length)
print "Length: %i" % length
data = s.recv(length)
print "Received: %s" % data
print ""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('localhost', PORT))
s.listen(5)
print "server is running..."
while True:
sock, addr = s.accept()
threading.Thread(target=handler, args=(sock, )).start()
Does anyone know what I'm doing wrong here?

I tested your code on Firefox 4 and got the same error upon hitting send, however before that I got
Firefox can't establish a connection
to the server at ws://localhost:8888/.
which is probably why the WebSocket object was destroyed. I suspect your handshake response is missing something, so Firefox is closing the socket.
From the Wikipedia article on Websockets:
The Sec-WebSocket-Key1 and
Sec-WebSocket-Key2 fields and the
eight bytes after the fields are
random tokens which the server uses to
construct a 16 byte token at the end
of its handshake to prove that it has
read the client's handshake.
Your server's response does not have this special number at the bottom, So I think we need to figure out how to generate it, and include it.
EDIT: How to generate that number
Lets start with key1, key2, and the 8 bytes at the end of the handshake
key1 = "18x 6]8vM;54 *(5: { U1]8 z [ 8"
key2 = "1_ tx7X d < nw 334J702) 7]o}` 0"
end8 = "Tm[K T2u"
We make a number for each key by ignoring every character that is not a digit 0-9. In Python:
def numFromKey(key):
return int(filter(lambda c: c in map(str,range(10)),key))
next we divide that number by the number of spaces in the original key string, so here is a is a function that counts the spaces in a string.
def spacesIn(key):
return len(filter(lambda c: c==' ',key))
The two numbers resulting from the keys are:
pkey1 = numFromKey(key1)/spacesIn(key1)
pkey2 = numFromKey(key2)/spacesIn(key2)
Now we need to concatenate the bytes of pkey1, pkey2, and end8. The processed keys need to be represented as 32 bit Big-Endian numbers.
from struct import pack
catstring = pack('>L',pkey1) + pack('>L',pkey2) + end8
Then we take the md5 hash of those bytes to get the magic number that we tack on the end of the handshake
import md5
magic = md5.new(catstring).digest()
Thats how I think it works at least

As of Version 8, this protocol is deprecated please refer to:
http://tools.ietf.org/id/draft-ietf-hybi-thewebsocketprotocol-12.txt
for the new version of the protocol.