So I'm trying and failing to get a token from twitter.
I get the following error: "Failed to validate oauth signature and token".
I have read it can be due to your system clock being wrong.
In javascript I tested my date with the following code
var minutes=1000*60;
var hours=minutes*60;
var days=hours*24;
var years=days*365;
var d=new Date();
var t=d.getTime();
var y=t/years;
console.log((y+1970) + " year and " + (t%years)/days)
This gave me the year as 2012 and 17 days..
1972
1976
1980
1984
1988
1992
1996
2000
2004
2008
__=10 leap days. Today is the 8th, so taking away leap days it appears my system clock is on the 7th? Or have I made a mistake here? If this is the problem how do I fix correct my clock?
In cmd when I do the date cmd it gives me todays date, i.e the 8th.
Here is my post request and code in case the problem lies within the code and not the clock.
My Post request is:
POST http://api.twitter.com/oauth/request_token?oauth_callback=127.0.0.1&oauth_consumer_key=FFZJrBaPLsiwTDg5159tTQ&oauth_nonce=tWHEEIW8vLS6tMggo3IXe6e449qv1GpE8LunKRsbRF&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1326039495&oauth_version=1.0&oauth_signature=d%2BQqgTzJCjYIp9vKwm%2BCWzVLPvA
which gets 401 (Unauthorized)
Here is my javascript code.
var url = "http://api.twitter.com/oauth/request_token";
var params={
oauth_callback : "127.0.0.1"
,oauth_consumer_key : "FFZJrBaPLsiwTDg5159tTQ"
,oauth_nonce : OAuth.nonce(42)
,oauth_signature_method : "HMAC-SHA1"
,oauth_timestamp : OAuth.timestamp()
,oauth_version: "1.0"}
//temp is to be the signature base string
var temp = toSignParams("POST",url,params);
console.log(temp);
//This logs the signature base string as "POST&http%3A%2F%2Fapi.twitter.com%2Foauth%2Frequest_token&oauth_callback%3D127.0.0.1%26oauth_consumer_key%3DFFZJrBaPLsiwTDg5159tTQ%26oauth_nonce%3D5gQVIa3WmwD6ARGGQTITl1Ozgxe2t8em5HC7g8wvMi%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1326038871%26oauth_version%3D1.0"
//which is correct I think.
//When I use this with the base signature from twitters oauth example page I get the result they got.
//it hashes the twitter signing key with base signature.
params.oauth_signature = b64_hmac_sha1("MCD8BKwGdgPHvAuvgvz4EQpqDAtx89grbuNMRd7Eh98&",temp);
var req = new XMLHttpRequest();
req.open("POST",toURIParams(url,params),true);
req.send();
console.log(params)
req.onreadystatechange=function(){
if (req.readyState==4)
{
console.log(req.responseText); //this is saying "Failed to validate oauth signature and token"
}
}
//function to convert to Signature paramaters, as indicated on twitter page.
function toSignParams(method,base,params){
tail=[];
for (var p in params) {
if (params.hasOwnProperty(p)) {
tail.push(p + "%3D" + encodeURIComponent(params[p]));
}
}
return method + "&" + encodeURIComponent(base) + "&" + tail.join("%26")
}
//function to convert to uri encoded parameters.
function toURIParams(base, params) {
tail = [];
for (var p in params) {
if (params.hasOwnProperty(p)) {
tail.push(p + "=" + encodeURIComponent(params[p]));
}
}
return base + "?" + tail.join("&")
}
Any ideas?
I used a library called "jsOAuth-1.3.3" as was found on the twitter list of libraries compatible with JavaScript but is no longer there.
The 2 javascript options they list now are:
user-stream by #AivisSilins — a simple Node.js User streams client
and
codebird-js by #mynetx — a Twitter Library in JScript.
as found on https://dev.twitter.com/docs/twitter-libraries
If anybody has a problem where they can't get either of those solutions to work, I can try find online or upload the library I used somewhere.
Related
I am trying to make a post request through google scripts to amazon to collect information.
We are trying to get our orders to MWS and transfer them to sheets automatically.
I got to the last step which is signing the request.
A few things I wasnt sure about:
They say we use the secret key for hashing,I only see a client secret
and an access key id, which do I use?
Do I add the URL as part of whats getting signed? On the MWS Scratch pad they add this as shown:
POST
mws.amazonservices.com
/Orders/2013-09-01
Does it have to be on separate lines does it need post and the rest of the stuff. Its a little unclear.?
I read online that the sha256 byte code gets base64 encoded, not the string literal, is that true?
I tried to hash the string that amazon gave to me with an online tool and compare it to the hash they provided and the base64 encode, thing matched. I tried decoding as well, nothing matched
Can someone please send me an example that works, so I can understand what happens and how it works?
Thank you!
Below is what I have so far:
function POSTRequest() {
var url = 'https:mws.amazonservices.com/Orders/2013-09-01?';
var today = new Date();
var todayTime = ISODateString(today);
var yesterday = new Date();
yesterday.setDate(today.getDate() - 1);
yesterday.setHours(0,0,0,0);
var yesterdayTime = ISODateString(yesterday);
var dayBeforeYesterday = new Date();
dayBeforeYesterday.setDate(today.getDate() - 2);
dayBeforeYesterday.setHours(0,0,0,0);
var dayBeforeYesterdayTime = ISODateString(dayBeforeYesterday);
var unsignedURL =
'POST\r\nhttps:mws.amazonservices.com\r\n/Orders/2013-09-01\r\n'+
'AWSAccessKeyId=xxxxxxxxxxx' +
'&Action=ListOrders'+
'&CreatedAfter=' + dayBeforeYesterdayTime +
'&CreatedBefore' + yesterdayTime +
'&FulfillmentChannel.Channel.1=AFN' +
'&MWSAuthToken=xxxxxxxxxxxx'+
'&MarketplaceId.Id.1=ATVPDKIKX0DER' +
'&SellerId=xxxxxxxxxxx'+
'&SignatureMethod=HmacSHA256'+
'&SignatureVersion=2'+
'&Timestamp='+ ISODateString(new Date) +
'&Version=2013-09-0';
var formData = {
'AWSAccessKeyId' : 'xxxxxxxxx',
'Action' : "ListOrders",
'CreatedAfter' : dayBeforeYesterdayTime,
'CreatedBefore' : yesterdayTime,
'FulfillmentChannel.Channel.1' : 'AFN',
'MWSAuthToken' : 'xxxxxxxxxxxx',
'MarketplaceId.Id.1' : 'ATVPDKIKX0DER',
'SellerId' : 'xxxxxxxxxx',
'SignatureMethod' : 'HmacSHA256',
'SignatureVersion' : '2',
'Timestamp' : ISODateString(new Date),
'Version' : '2013-09-01',
'Signature' : calculatedSignature(unsignedURL)
};
var options = {
"method" : "post",
"muteHttpExceptions" : true,
"payload" : formData
};
var result = UrlFetchApp.fetch(url, options);
writeDataToXML(result);
Logger.log(result);
if (result.getResponseCode() == 200) {
writeDataToXML(result);
}
}
function calculatedSignature(url) {
var urlToSign = url;
var secret = "xxxxxxxxxxxxxxxxxxx";
var accesskeyid = 'xxxxxxxxxxxxxxx';
var byteSignature = Utilities.computeHmacSha256Signature(urlToSign, secret);
// convert byte array to hex string
var signature = byteSignature.reduce(function(str,chr){
chr = (chr < 0 ? chr + 256 : chr).toString(16);
return str + (chr.length==1?'0':'') + chr;
},'');
Logger.log("URL to sign: " + urlToSign);
Logger.log("");
Logger.log("byte " + byteSignature);
Logger.log("");
Logger.log("reg " + signature);
var byte64 = Utilities.base64Encode(byteSignature)
Logger.log("base64 byte " + Utilities.base64Encode(byteSignature));
Logger.log("");
Logger.log("base64 reg " + Utilities.base64Encode(signature));
return byte64;
}
Step 1, creating the string to be signed
The string_to_sign is the combination of the following:
The string POST followed by a NEWLINE character
The name of the host, mws.amazonservices.com, followed by a NEWLINE
The API URL, often just /, or somthing like /Orders/2013-09-01, followed by a NEWLINE
An alphabetical list of all parameters except Signature in URL encoding, like a=1&b=2, not followed by anything
The minimum parameters seem to be the following:
AWSAccessKeyId is a 20-character code provided by Amazon
Action is the name of your API call, like GetReport
SellerId is your 14-character seller ID
SignatureMethod is HmacSHA256
SignatureVersion is 2
Timestamp is a date like 20181231T23:59:59Z for one second before new year UTC
Version is the API version like 2013-09-01
additional parameters may be needed for your call, depending on the value of Action
Please note:
The NEWLINE character is just "\n", not "\r\n"
The name of the host should not include "https://" or "http://"
The Signature parameter is necessary for the actual call later (see step 3), but is not part of the string_to_sign.
Your string_to_sign should now look somewhat like this:
POST
mws.amazonservices.com
/Orders/2013-09-01
AWSAccessKeyId=12345678901234567890&Action=ListOrders&CreatedAfter .... &Version=2013-09-01
Step 2, signing that string
Calculate a SHA256 hash of above string using the 40-character Secret Key
Encode this hash using Base64
In Pseudocode: signature = Base64encode( SHA256( string_to_sign, secret_key ))
Step 3, send the call
Send a HTTPS POST request, using the full alphabetical list of parameters, now including above signature as Signature somewhere in the middle, because you need to keep ascending alphabetical order.
https://mws.amazonservices.com/Orders/2013-09-01?AWSAccessKeyId....Version=2013-09-01
Step 4, processing the result
You should be getting two things back: a response header and a XML document. Be sure to evaluate the HTTP status in the header as well as all contents of the XML document. Some error messages are hidden deeply in XML while HTTP returns "200 OK".
Step 5, Advanced Stuff
If you use calls that require you to send a document, like SendFeed, you need to do the following additional steps:
Calculate the MD5 hash of your document
Encode this hash using Base64
In Pseudocode: contentmd5= Base64encode( MD5( document ))
Add Content-Type: text/xml (or whatever fits your document) as HTTP header
Add Content-MD5: plus the base64 encoded hash as HTTP header
This code was a great help for building a similar application. I corrected some small issues to bring it up to work:
before signing the ISODate need to be worked out to replace the ":" chars
var todayTime = ISODateString(today);
var todayISO = todayTime;
var todayTime_ = todayTime.replace(":", "%3A");
todayTime = todayTime_.replace(":","%3A");
The same for the calculated signature (quick & dirty solution, replace only 3 appearences and need to updated for more chars)
Logger.log(unsignedURL);
var tmpsignature = calculatedSignature(unsignedURL);
var orsignature = tmpsignature;
// encode special chars
tmpsignature = encodeURIComponent(orsignature);
}
I added a header and dropped the form, put all parameters in the url
var header = {
"x-amazon-user-agent": "GoogleSheets/1.0 (Language=Javascript)",
"Content-Type": "application/x-www-form-urlencoded"
// "Content-Type": "text/xml"
};
var options = {
"method" : "post",
"muteHttpExceptions" : true,
// "payload" : formData,
"header":header
};
And changed the call to url encoded (Form was not working, do not know why)
var url = 'https:mws-eu.amazonservices.com/Orders/2013-09-01?'+
'AWSAccessKeyId=<your stuff>'+
'&Action=GetOrder'+
'&SellerId=<your stuff>+
'&MWSAuthToken=<your token>'+
'&SignatureVersion=2'+
'&Timestamp='+todayTime'+ // remember to replace the ":" thru hex
'&Version=2013-09-01'+
'&Signature='+ tmpsignature+
'&SignatureMethod=HmacSHA256'+
'&AmazonOrderId.Id.1='+<your order;
parsed the response:
var result = UrlFetchApp.fetch(url, options);
//writeDataToXML(result);
Logger.log(result);
var xml = result.getContentText();
var document = XmlService.parse(xml);
This worked! :-)
I checked the signature also with
https://mws-eu.amazonservices.com/scratchpad/index.html
I'm looking to write a JavaScript function that will return the current BTC/USD exchange rate. I've done some research, but I just want something simple. It won't be used server-side for calculating values (obvious security implications), but just as a convenience for my users. I have 2 text fields, and when the user changes one of the values, it will update the other field.
Here is my code so far:
var getBTCRate = function(){ /* code here */ };
var btcprice = getBTCRate();
// update the BTC value as the USD value is updated
$("#usdvalue").keyup(function(ev){
var usdvalue = $("#usdvalue").val();
$("#btcvalue").val(usdvalue / btcprice);
});
// update the USD value as the BTC value is updated
$("#btcvalue").keyup(function(ev){
var btcvalue = $("#btcvalue").val();
$("#usdvalue").val(btcvalue * btcprice);
});
Plain and simple. In my research I haven't been able to find something that will do this, only a bunch of confusing APIs. Any help is much appreciated.
EDITED to fix a mistake in the code.
EDITED AGAIN to fix the position of the function declaration. Thanks to #RobG for pointing this out.
My first idea was to use JQuery load like this
$.get('https://www.google.com/search?q=btc+value', function(p) {
console.log(p);
});
but cross-origin rules stopped me.
Now, you can pay for a service that has an API, but I wanted to do it without having to pay. What I ended up doing is a server based solution. I use PowerBasic for my back end, with the SocketTools Library.
#COMPILE EXE
#DIM ALL
#Include "pbcgi.inc"
#Include "C:\bas_src\socketTools\v9.5\inc\cstools9.inc"
Function PBMain () As Long
Local btc As String
Local html As String
html= httpGet("https://www.google.com/search?q=btc+value")
' filter out just the current BTC value
' by looking for what is between the words "Bitcoin =" and "United States Dollar"
btc=Remain$(html,"Bitcoin =")
btc=Extract$(btc,"United States Dollar")
btc=Trim$(btc)
writeCGI "{"+jsonPad("btc")+":"+jsonPad(btc)+","+jsonPad("error")+":"+jsonPad("0")+"}"
END FUNCTION
'================================================================
' retrieve the page and return it as a string
Function httpGet(ByVal URL As String) As String
If IsFalse( HttpInitialize($CSTOOLS9_LICENSE_KEY) ) Then
Function="unable to init socket library"
Exit Function
End If
Local hClient As Dword
Local lpszURL As STRINGZ * 4096
Local lpszBuffer As STRINGZ * 100000
Local httpContents As String
Local httpPort As Long
Local httpOptions As Dword
Local nResult As Long
If LCase$(Trim$(Left$(URL, 8))) = "https://" Then
httpPort = 443
httpOptions = %HTTP_OPTION_SECURE Or %HTTP_OPTION_REDIRECT
Else
httpPort = 80
httpOptions = %HTTP_OPTION_REDIRECT
End If
lpszURL = URL & Chr$(0)
hClient = HttpConnect(lpszURL, _
httpPort, _
%HTTP_TIMEOUT, _
httpOptions, _
%HTTP_VERSION_10)
If hClient = %INVALID_CLIENT Then
Function = "Could not connect to: " & URL
Exit Function
End If
HttpSetEncodingType(hClient, %HTTP_ENCODING_NONE)
nResult = HttpGetText(hClient, lpszURL, lpszBuffer, 100000)
If nResult = %HTTP_ERROR Then
Function = "Error retrieving file."+Str$(nResult)
Else
' success
httpContents = lpszBuffer
Function =httpContents
End If
Call HttpDisconnect(hClient)
HttpUninitialize()
End Function
'================================================================
' pad string for json return
Function jsonPad(jstr As String) As String
Local i As Long
Replace Chr$(10) With " " In jstr
Replace Chr$(13) With " " In jstr
Replace "\" With "\\" In jstr
Replace $Dq With "\"+$Dq In jstr
For i = 0 To 9
Replace Chr$(i) With " " In jstr
Next
Function=$Dq+Trim$(jstr)+$Dq
End Function
On my web page, I call it using AJAX
function showBTC(){
$.ajax({
type: "POST",
url: "../cgi/btcGet.exe",
dataType: "json",
success: function(json){
if(json.error !== "0" ){
console.log( json.error );
return;
}
$("#btcValue").html( "current BTC value $"+json.btc+"<br><br>" );
}
});
}
I know that seems like a "too specific" answer, but it's how I do it.
EDIT 5/11/2020:
there is a Bitcoin API found at coindesk.com which greatly simplifies this process.
const api = 'https://apiv2.bitcoinaverage.com/indices/local/ticker/short?crypto=BTC&fiat=USD'
$.get(api, p => {
document.querySelector('pre').textContent = JSON.stringify(p, null, 2)
});
Result
{
"BTCUSD": {
"ask": 3594.5649555077953,
"timestamp": 1550284932,
"bid": 3591.715961836563,
"last": 3592.745617344171,
"averages": {
"day": 3583.13243402
}
}
}
So take your pick p.BTCUSD.ask // or bid or last
demo
I have a SharePoint calendar list, and I'm looking for a script to retrieve the number of events on a day. For example, for today, 10 August, I want to just retrieve the number of events stored in my calendar.
Any suggestion will be very helpful.
The Problem with Querying Calendars in JavaScript: Recurrence
Ordinarily retrieving data from SharePoint with JavaScript is really straightforward (at least for versions beyond SharePoint 2007) using either REST or the JavaScript Object Model. However, calendars have functionality for creating recurring events which can complicate things.
For example, a recurring event may have a start date of two years ago and an end date many years in the future, but maybe the event itself only actually occurs on the third Tuesday of every month. If you just query the list and try to compare today's date against the start date and end date to see if they overlap, that recurring event will show up in your results (even though today is not the third Tuesday of the month).
In server-side code you can get around this by setting the ExpandRecurrence property to true on the SPQuery object used to query the list. However, as of SP2010 and SP2013, that property is not exposed on the equivalent JavaScript Object Model.
Workaround: Using the Lists.GetListItems web service**
An alternative is to use one of the old web services that are still floating around... specifically the Lists web service accessible at /_vti_bin/Lists.asmx. This web service has a GetListItems method that accepts a SOAP message in which you can specify a query option to expand recurrence as you would on the server side.
Here's an example demonstrating how you can query the Lists web service using plain JavaScript:
// Set webUrl and listGuid to values specific to your site and list
var webUrl = "http://server/sitewhereyourlistexists";
var listGuid = "{000000000-0000-0000-0000-000000000000}"
// An XMLHttpRequest object is used to access the web service
var xhr = new XMLHttpRequest();
var url = webUrl + "/_vti_bin/Lists.asmx";
xhr.open("POST", url, true);
xhr.setRequestHeader("Content-Type","text/xml; charset=utf-8");
xhr.setRequestHeader("SOAPAction","http://schemas.microsoft.com/sharepoint/soap/GetListItems");
// The message body consists of an XML document
// with SOAP elements corresponding to the GetListItems method parameters
// i.e. listName, query, and queryOptions
var data = "<?xml version=\"1.0\" encoding=\"utf-8\"?>"+
"<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">" +
"<soap:Body>" +
"<GetListItems xmlns=\"http://schemas.microsoft.com/sharepoint/soap/\">" +
"<listName>"+listGuid+"</listName>" +
"<query>" +
"<Query><Where>" +
"<DateRangesOverlap>" +
"<FieldRef Name=\"EventDate\"/>"+
"<FieldRef Name=\"EndDate\"/>"+
"<FieldRef Name=\"RecurrenceID\"/>"+
"<Value Type=\"DateTime\"><Today/></Value>"+
"</DateRangesOverlap>"+
"</Where></Query>"+
"</query>" +
"<queryOptions>"+
"<QueryOptions>"+
"<ExpandRecurrence>TRUE</ExpandRecurrence>"+
"</QueryOptions>"+
"</queryOptions>" +
"</GetListItems>" +
"</soap:Body>" +
"</soap:Envelope>";
// Here we define what code we want to run upon successfully getting the results
xhr.onreadystatechange = function(){
if(xhr.readyState == 4){
if(xhr.status == 200){
var doc = xhr.responseXML;
// grab all the "row" elements from the XML results
var rows = doc.getElementsByTagName("z:row");
var results = "Today's Schedule ("+rows.length+"):\n\n";
var events = {};
for(var i = 0, len = rows.length; i < len; i++){
var id = rows[i].getAttribute("ows_FSObjType"); // prevent duplicates from appearing in results
if(!events[id]){
events[id] = true;
var allDay = rows[i].getAttribute("ows_fAllDayEvent"),
title = rows[i].getAttribute("ows_Title"),
start = rows[i].getAttribute("ows_EventDate");
var index = start.indexOf(" ");
var date = start.substring(5,index)+"-"+start.substring(2,4); // get the date in MM-dd-yyyy format
start = start.substring(index, index+6); // get the start time in hh:mm format
var end = rows[i].getAttribute("ows_EndDate");
index = end.indexOf(" "); end = end.substring(index,index+6); // get the end time in hh:mm format
results += date + " " + (allDay == "1" ? "All Day\t" : start + " to " + end ) + " \t " + title + "\n";
}
}
alert(results);
}else{
alert("Error "+xhr.status);
}
}
};
// Finally, we actually kick off the query
xhr.send(data);
Checking ranges other than today's date
In the <Value> child node of the <DateRangesOverlap> node, you can specify <Now />, <Today />, <Week />, <Month />, or <Year />.
Week, Month, and Year will check for events within the same week, month, or year of the current date.
To check a date range relative to some date other than today's, you can add a <CalendarDate> node to the <QueryOptions> node of the CAML query, as seen below.
"<query>" +
"<Query><Where>" +
"<DateRangesOverlap>" +
"<FieldRef Name=\"EventDate\"/>"+
"<FieldRef Name=\"EndDate\"/>"+
"<FieldRef Name=\"RecurrenceID\"/>"+
"<Value Type=\"DateTime\"><Week /></Value>"+
"</DateRangesOverlap>"+
"</Where></Query>"+
"</query>" +
"<queryOptions>"+
"<QueryOptions>"+
"<ExpandRecurrence>TRUE</ExpandRecurrence>"+
"<CalendarDate>2017-03-10</CalendarDate>" +
"</QueryOptions>"+
"</queryOptions>" +
Note that values of <Now /> and <Year /> do not seem to respect the CalendarDate value.
I am trying to scrape the data from the below link, in a c# console app:
https://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability
Using the developer tools in chrome I can see that its possible to get a json response, the url to get this is:
https://www.eex-transparency.com/dsp/tem-12?country=de&expires=1454345128&md5=TRhtJei_go4ueLeekBc8yw
the website uses this js file (https://www.eex-transparency.com/assets/js/tpe-website.js) to generate the expires and md5 hash key. I think I've figured out that the expires value is a unix datetime. I have never used javascript before so finding it hard to figure out how they construct the md5.
The Javascript that generates these code is:
generateCryptedParams=function(url,clientIP)
{
var cryptedParams,md5,md5Encoded,md5WithoutSpeciaChars,parser,timePoint,urlPath;
return timePoint=moment().tz("Europe/Berlin").add(1,"minute").unix(),
parser=document.createElement("a"),
parser.href=url,
urlPath=parser.pathname,
"/"!==urlPath[0]&&(urlPath="/"+urlPath),
md5=CryptoJS.MD5(urlPath+timePoint+clientIP+" zYeHzBomGdgV"),
md5Encoded=md5.toString(CryptoJS.enc.Base64),
md5WithoutSpeciaChars=replaceSpecialChars(md5Encoded),
cryptedParams={"expires":timePoint,"md5":md5WithoutSpeciaChars}
}
replaceSpecialChars=function(str)
{
var key,specialChars,value;
specialChars={"=":"","\\+":"-","/":"_","%":"_"};
for(key in specialChars)
value=specialChars[key],
str=str.replace(new RegExp(key,"g"),value);
return str
}
As i said I think I'm comfortable with the timepoint part but the md5 is confusing me. Below is my C# code to replicate their but when I pass the md5 hash their site returns a 403 Forbidden error.
public Tuple<string, Int32> GenerateCrypto(string url, string ipAddress)
{
string cetId = "Central European Standard Time";
TimeZoneInfo cetZone = TimeZoneInfo.FindSystemTimeZoneById(cetId);
var CETDateTime = TimeZoneInfo.ConvertTimeFromUtc(DateTime.UtcNow, cetZone);
//Int32 unixTimestamp = (Int32)(CETDateTime.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
Int32 unixTimestamp = (Int32)(DateTime.UtcNow.AddMinutes(1).Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
url = url.Split('/')[3];
var md5 = CipherUtility.GenerateMd5(url + unixTimestamp + ipAddress + " zYeHzBomGdgV");
var md5Encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(md5));
var md5withoutSpecialCharts = replaceSpecialChars(md5Encoded);
md5withoutSpecialCharts = md5withoutSpecialCharts.Substring(0, 22);
return new Tuple<string, Int32>(md5withoutSpecialCharts, unixTimestamp);
}
The solution was that I needed to concatenate a const string to all the elements before hashing it.
I'm trying to query posts from Instagram by providing the hashtag and the time range (since and until dates).
I use the recent tags endpoint.
https://api.instagram.com/v1/tags/{tag-name}/media/recent?access_token=ACCESS-TOKEN
My code is written in Node.js using the instagram-node library (see the inline comments):
// Require the config file
var config = require('../config.js');
// Require and intialize the instagram instance
var ig = require('instagram-node').instagram();
// Set the access token
ig.use({ access_token: config.instagram.access_token });
// We export this function for public use
// hashtag: the hashtag to search for
// minDate: the since date
// maxDate: the until date
// callback: the callback function (err, posts)
module.exports = function (hashtag, minDate, maxDate, callback) {
// Create the posts array (will be concated with new posts from pagination responses)
var posts = [];
// Convert the date objects into timestamps (seconds)
var sinceTime = Math.floor(minDate.getTime() / 1000);
var untilTime = Math.floor(maxDate.getTime() / 1000);
// Fetch the IG posts page by page
ig.tag_media_recent(hashtag, { count: 50 }, function fetchPosts(err, medias, pagination, remaining, limit) {
// Handle error
if (err) {
return callback(err);
}
// Manually filter by time
var filteredByTime = medias.filter(function (currentPost) {
// Convert the created_time string into number (seconds timestamp)
var createdTime = +currentPost.created_time;
// Check if it's after since date and before until date
return createdTime >= sinceTime && createdTime <= untilTime;
});
// Get the last post on this page
var lastPost = medias[medias.length - 1] || {};
// ...and its timestamp
var lastPostTimeStamp = +(lastPost.created_time || -1);
// ...and its timestamp date object
var lastPostDate = new Date(lastPostTimeStamp * 1000);
// Concat the new [filtered] posts to the big array
posts = posts.concat(filteredByTime);
// Show some output
console.log('found ' + filteredByTime.length + ' new items total: ' + posts.length, lastPostDate);
// Check if the last post is BEFORE until date and there are no new posts in the provided range
if (filteredByTime.length === 0 && lastPostTimeStamp <= untilTime) {
// ...if so, we can callback!
return callback(null, posts);
}
// Navigate to the next page
pagination.next(fetchPosts);
});
};
This will start fetching the posts with the most recent to least recent ones, and manually filter the created_time.
This works, but it's very very inefficient because if we want, for example, to get the posts from one year ago, we have to iterate the pages until that time, and this will use a lot of requests (probably more than 5k / hour which is the rate limit).
Is there a better way to make this query? How to get the Instagram posts by providing the hashtag and the time range?
I think this is the basic idea you're looking for. I'm not overly familiar with Node.js, so this is all in plain javascript. You'll have to modify it to suit your needs and probably make a function out of it.
The idea is to convert an instagram id (1116307519311125603 in this example) to a date and visa versa to enable you to quickly grab a specific point in time rather then backtrack through all results until finding your desired timestamp. The portion of the id after the underscore '_' should be trimmed off as that refers, in some way, to the user IIRC. There are 4 functions in the example that I hope will help you out.
Happy hacking!
//static
var epoch_hour = 3600,
epoch_day = 86400,
epoch_month = 2592000,
epoch_year = 31557600;
//you'll need to set this part up/integrate it with your code
var dataId = 1116307519311125603,
range = 2 * epoch_hour,
count = 1,
tagName = 'cars',
access = prompt('Enter access token:'),
baseUrl = 'https://api.instagram.com/v1/tags/' +
tagName + '/media/recent?access_token=' + access;
//date && id utilities
function idToEpoch(n){
return Math.round((n / 1000000000000 + 11024476.5839159095) / 0.008388608);
}
function epochToId(n){
return Math.round((n * 0.008388608 - 11024476.5839159095) * 1000000000000);
}
function newDateFromEpoch(n){
var d = new Date(0);
d.setUTCSeconds(n);
return d;
}
function dateToEpoch(d){
return (d.getTime()-d.getMilliseconds())/1000;
}
//start with your id and range; do the figuring
var epoch_time = idToEpoch(dataId),
minumumId = epochToId(epoch_time),
maximumId = epochToId(epoch_time + range),
minDate = newDateFromEpoch(epoch_time),
maxDate = newDateFromEpoch(epoch_time + range);
var newUrl = baseUrl +
'&count=' + count +
'&min_tag_id=' + minumumId +
'&max_tag_id=' + maximumId;
//used for testing
/*alert('Start: ' + minDate + ' (' + epoch_time +
')\nEnd: ' + maxDate + ' (' + (epoch_time +
range) + ')');
window.location = newUrl;*/
To support this excellent answer, an instagram ID is generated via the plpgSQL function:
CREATE OR REPLACE FUNCTION insta5.next_id(OUT result bigint) AS $$
DECLARE
our_epoch bigint := 1314220021721;
seq_id bigint;
now_millis bigint;
shard_id int := 5;
BEGIN
SELECT nextval('insta5.table_id_seq') %% 1024 INTO seq_id;
SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
result := (now_millis - our_epoch) << 23;
result := result | (shard_id << 10);
result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;
from Instagram's blog
Despite a similar getting posts process, Data365.co Instagram API, I currently working at, seems to be more suitable and efficient. It does not have a limit of 5,000 posts per hour, and you can specify the period of time for which your need posts in the request itself. Also, the billing will be taken into account only posts from the indicated period. You won't have to pay for data you don't need.
You can see below a task example to download posts by the hashtag bitcoins for the period from January 1, 2021, to January 10, 2021.
POST request: https://api.data365.co/v1.1/instagram/tag/bitcoins/update?max_posts_count=1000&from_date=2021-01-01&to_date=2021-01-10&access_token=TOKEN
A GET request example to get the corresponding list of posts:
https://api.data365.co/v1.1/instagram/tag/bitcoins/posts?from_date=2021-01-01&to_date=2021-01-10&max_page_size=100&order_by=date_desc&access_token=TOKEN
More detailed info view in API documentation at https://api.data365.co/v1.1/instagram/docs#tag/Instagram-hashtag-search