I want to use YQL to retrieve all 10-Q & 10-K files from SEC EDGAR database.
After ref to the discussions [1] & [2], I bump into some problem.
It seems that YQL cannot get search results from the search engine.
However, I can directly access the filing detail page.
Here is a jsfiddle shows the problem. Although both queries return success message, the query to the search engine returns a result of empty array.
Is there any other way to get all the html addresses of the detail filing pages without querying EDGAR search engine? Thanks.
Example code by using YQL shows below:
// results page from EDGAR search engine:
// fail to get data
var queryURL = "http://www.sec.gov/cgi-bin/browse-edgar?" +
"action=getcompany&CIK=0001326801&type=10-K&dateb=&owner=exclude&count=100";
// EDGAR 10-K detail filing page:
// success to fetch by YQL
var filingURL = "http://www.sec.gov/Archives/edgar/data/1326801/" +
"000132680114000007/0001326801-14-000007-index.htm";
$.get(queryURL).then(function() {
// get successful message, but get results of empty array
})
.then(function() {
$.get(filingURL).then(function() {
// get successful message, and get results of empty array
})
} )
The /cgi-bin URL is restricted by robots.txt, so YQL will honour that and not crawl the page.
You can see this happening by enabling diagnostics for the YQL query.
Add diagnostics=true to the YQL URL, like /v1/public/yql?diagnostics=true&callback=?
Look for the diagnostics field in the results. This contains information about the query and any URLs it visited.
Related
I have A search rest API, when I run it through Share point Designer, i don't Get the Same Number if results as Returned by the Search page on the Share point site, I have tried Using different Source ids , also tried to use the default source id from results source but I always get the same results so i am not sure what I am doing wrong.
My Other Thought is, IS there a way to Get all the results From the Default search function Built in to Share-point?
var ct = new SP.ClientContext.get_current();
var keywordQuery = new Microsoft.SharePoint.Client.Search.Query.KeywordQuery(ct);
var queryStr = ctx.DataProvider.get_currentQueryState().k;
keywordQuery.set_queryText(queryStr);
keywordQuery.set_trimDuplicates(false);
keywordQuery.set_enableSorting(true);
keywordQuery.set_sourceId=("xxxxxx-xxxx-xxxx-xxx-xxxxxxx");
keywordQuery.set_rowLimit(500);
keywordQuery.set_trimDuplicates(false);
var searchExecutor = new Microsoft.SharePoint.Client.Search.Query.SearchExecutor(ct);
var results = searchExecutor.executeQuery(keywordQuery);
ct.executeQueryAsync(onQuerySuccess, onQueryFail);
function onQuerySuccess()
{
results.m_value.ResultTables[1].ResultRows.forEach(function (row)
{
var Aname1 = row.name;
console.log(row);
if (!$isNull(Aname1))
{
var name= Aname1;
console.log(name);
}
});
}
function onQueryFail()
{
}
Usually, the results are paginated. What it means is that, instead of returning all the results at once, they are divided into parts and each part (page) is sent once.
For example, when you search in google.com, instead of returning all 1,50,00,000.... results, Google returns only 10 results or so. To get the next 10 results, you click the next button in the pagination menu at the bottom of the page.
This is done so that the API and network don't get overloaded. Imagine how large a response with 1,50,00,000 records would be.
This is what's happening with you. In the response you recieved, see if there's a record with a URL for the next page, Microsoft usually does things this way. If you call that URL, you'll get the next page. If that's not there, see if the URL you called has a parameter somewhere, where you can select the page.
Is it possible to get the query parameters from google searches?
I.e. if someone googled bicycles the url becomes:
https://www.google.es/search?q=bicycles......
If you then come in the search results and someone clicks to your page you cannot see the query parameters with document.referrer it will only show
https://www.google.es/
Is there a way to know what a visitor searched before coming to your site?
Late response, but I have recently been doing some research and thought that this information may be interesting to other stackoverflow visitors who end up on this page.
Prior to 2016 you could get the url parameters using PHP from the referral data when visitors came to your site via organic search by using server variables:
function get_search_query(){
$query = $_SERVER['QUERY_STRING'];
return (strlen($query)? $query: 'none');
}
or
function get_search_query() {
$query = false;
$referrer = $_SERVER['HTTP_REFERER'];
if (!empty($referrer)) {
//Parse the referrer URL
$parts_url = parse_url($referrer);
// Check if a query string exists
$query = isset($parts_url['query']) ? $parts_url['query'] : '';
return (strlen($query)? $query: false);
}
return $query;
}
However, Google and other search engines have since made it impossible to view the query parameters from organic search and any script like that above returns a blank query string. This is true whether you are logged in to Google or not.
This is unfortunate because this removes valuable information like the the keywords used to find your site. The information still can be found in Google Search Console, but this provides Googles interpretation of what you should see and it is not as direct as getting the information immediately when a visitor hits a page after an organic search.
I am not aware of there being any way to still get organic query parameters.
On a search result page I would like to get the count of Google+ shares for the URL of each search result item via Ajax. I already managed to set up counting functions for Facebook and Twitter:
$.getJSON('http://urls.api.twitter.com/1/urls/count.json?url=' + url + '&callback=?', function(data){
tweets = data.count;
});
$.getJSON('https://api.facebook.com/method/links.getStats?urls='+ url+'&format=json', function(data){
fblikes = data[url].shares;
});
For Google+ I also already found a solution, but this requires the API Key for each URL. Is there any way to retrieve the Google+ count without such a key? Using it for tons of dynamically loaded search results, of course I cannot create an API key for each search result URL.
Yes, it's possible to get Google +1 counts. They are retrieved via a JSON-RPC POST call.
POST URL:
https://clients6.google.com/rpc?key=AIzaSyCKSbrvQasunBoV16zDH9R33D88CeLr9gQ
POST Body:
[{"method":"pos.plusones.get","id":"p","params":{"nolog":true,"id":"%%URL%%","source":"widget","userId":"#viewer","groupId":"#self"},"jsonrpc":"2.0","key":"p","apiVersion":"v1"}]
where %%URL%% is the desired URL.
Have a look at the following:
Getting google +1 Page shares via AJAX (hidden Api)
http://www.sharedcount.com/documentation.php
Recently was enabled Google + Domains API For Apps Script, I have explored some options and it seems is going to work, but in the specific case of PlusdDomains.Circles.list I don't know how to pass the second argument what is an object, I can not obtain several fields in the response, this is my code.
function getProfile() {
var userId = 'me';
var post = { maxResults: 2, fields:"title"};
var profile = PlusDomains.Circles.list(userId, post);
Logger.log('all: %s', JSON.stringify(profile));
}
this is the output,all: {"title":"Google+ List of Circles"}
if I try to get another field I don't know if this is correct, I put this:
var post = { maxResults: 2, fields:["title", "items"]};
but I get the same result:all: {"title":"Google+ List of Circles"}
If I try to get the result value for items, I get undefined. I don't how to pass the object correctly or if this is a bug in the Apps Script, somebody has idea??
I'm trying to get this working too. From the public google+ domain api docs it looks that the fields property expects a string with field names comma separated, i.e.
var post = { maxResults: 2, fields:"title,items"};
I don't seem to get the items populated (all the properties are undefined) in my google apps script. But when I use the API explorer "try it" console with OAuth2 enabled for scopes https://developers.google.com/+/domains/api/circles/list and https://developers.google.com/+/domains/api/circles/list I do see the items populated, so I'm thinking there may be an issue with my scripts authorization scopes or a bug in the google apps google+ domain service.
I am using the HTML5 version of Facebook Comment in my website. I have my own Facebook APP Id.
Using Graph-API, and FQL (I think this is how to do it), I want to list all the Comments posted in my website.
Example -
Page Title1
--Comment1
--Comment2
--Comment3
Page Title2
--Comment1
--Comment2
--Comment3
Page Title3
--Comment1
--Comment2
--Comment3
etc.
Please help me out.
It is possible, in two different ways, as long as you have a fixed set of sub-pages you want to fetch comments from.
If you have a large amount of sub-pages, or a variable amount, then you don't have a good scalable solution - and many have been looking for one:
Facebook fb:comments Graph API
How to display recent comments from Facebook Comments social plugin?
Facebook FQL query to return all comments against an application
Retrieve all comments with FQL by application ID
Facebook FQL query to return all comments against an application
fql query to get comment count no longer working
http://facebook.stackoverflow.com/questions/10023179/retrieve-all-the-comments-posted-using-fql
For a Fixed set of sub-pages in your website, you can either use a batch request, or an FQL query.
Batch Request
First, you need your access token. Just enter the following as a url in a browser (credit to this website ):
https://graph.facebook.com/oauth/access_token?type=client_cred&client_id=APP_ID&client_secret=APP_SECRET
And this is the javascript jquery code to make a batch request to fetch comments from several urls at once:
$.ajax({
url: 'https://graph.facebook.com/',
type : "POST",
data: {
access_token : 'YOUR_APP_ACCESS_TOKEN',
batch : '[ \
{"method":"GET","relative_url":"URL1"}, \
{"method":"GET","relative_url":"URL2"} \
]'
},
success: function(data) {
jdata = JSON.parse(data);
$.each(jdata, function(index,value){
jdata[index].body = JSON.parse(value.body);
console.log(value.body);
});
// Do whatever you want with jdata
}
});
FQL
inspired from this post
FB.api({
method: 'fql.query',
query: 'select text from comment where object_id in (select comments_fbid from link_stat where url="URL1" or url="URL2")'
}, function(response) {
// Do something with results
});
Conclusion
Because of this limitation of Facebook, I plan to switch to disqus.com, which apparently supports this feature (As you can see from this blog, for example. (search for 'recent comments')
Rather than list all the comments on your site, Facebook wants you to implement code to get notified when a new comment is posted anywhere on your site.
To make this happen, you have to put some Javascript into the page where the comment is posted to also notify yourself:
window.fbAsyncInit = function(){
console.log("subscribing to comment create");
FB.Event.subscribe('comment.create',function(response){
console.log("facbeook comment created: " + JSON.stringify(response));
var commentQuery = FB.Data.query('SELECT fromid, text FROM comment WHERE post_fbid=\'' + response.commentID + '\' AND object_id IN (SELECT comments_fbid FROM link_stat WHERE url=\'' + response.href + '\')');
FB.Data.waitOn([commentQuery], function () {
console.log("Facebook comment: " + JSON.stringify(commentQuery));
});
});
};
Where rather than just logging the comment to the console, you would need to implement some AJAX that would send the comment back to your site where you could store the comment in your database, or send yourself an email notifying you that the comment has been posted.
Reference: Facebook Comments Plugin
Say your website is http://mywebsite.com/blog.php?id=3 and you have a facebook comments plugin on it,
you can access comments this way
https://graph.facebook.com/comments/?ids={YOUR_URL}.
{YOUR_URL} becomes http://mywebsite.com/blog.php?id=3
Example 1: (Comments plugin installed on developers facebook doc website )
website: http://developers.facebook.com/docs/reference/plugins/comments
fetch comments: https://graph.facebook.com/comments/?ids=http://developers.facebook.com/docs/reference/plugins/comments
Example 2:
website: http://techcrunch.com/2011/04/08/the-seven-most-interesting-startups-at-500-startups-demo-day/
fetch comments: https://graph.facebook.com/comments/?ids=http://techcrunch.com/2011/04/08/the-seven-most-interesting-startups-at-500-startups-demo-day/
Check this too
Sample code for pulling comments can be found on this blog post