Is it possible to get the query parameters from google searches?
I.e. if someone googled bicycles the url becomes:
https://www.google.es/search?q=bicycles......
If you then come in the search results and someone clicks to your page you cannot see the query parameters with document.referrer it will only show
https://www.google.es/
Is there a way to know what a visitor searched before coming to your site?
Late response, but I have recently been doing some research and thought that this information may be interesting to other stackoverflow visitors who end up on this page.
Prior to 2016 you could get the url parameters using PHP from the referral data when visitors came to your site via organic search by using server variables:
function get_search_query(){
$query = $_SERVER['QUERY_STRING'];
return (strlen($query)? $query: 'none');
}
or
function get_search_query() {
$query = false;
$referrer = $_SERVER['HTTP_REFERER'];
if (!empty($referrer)) {
//Parse the referrer URL
$parts_url = parse_url($referrer);
// Check if a query string exists
$query = isset($parts_url['query']) ? $parts_url['query'] : '';
return (strlen($query)? $query: false);
}
return $query;
}
However, Google and other search engines have since made it impossible to view the query parameters from organic search and any script like that above returns a blank query string. This is true whether you are logged in to Google or not.
This is unfortunate because this removes valuable information like the the keywords used to find your site. The information still can be found in Google Search Console, but this provides Googles interpretation of what you should see and it is not as direct as getting the information immediately when a visitor hits a page after an organic search.
I am not aware of there being any way to still get organic query parameters.
Related
I want to build a chrome extension which can save the google search text (single English word) with its meaning. E.g. Suppose I have searched for the word "amount" i get the meaning.
"- a quantity of something, especially the total of a thing or things in number, size, value, or extent." Now i want these 2 to store as key value pair in my chrome's local storage. Hope I m cleared.
For this I m using curl and hosted my php page to a server.
$curl = curl_init('http://testing-ground.scraping.pro/textlist');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$page = curl_exec($curl);
if(curl_errno($curl)) // check for execution errors
{
echo 'Scraper error: ' . curl_error($curl);
exit;
}
curl_close($curl);
$regex = '/<div id="case_textlist">(.*?)<\/div>/s';
if ( preg_match($regex, $page, $list) )
echo $list[0];
else
print "Not found";
Code is working fine for static sites or 'dynamic sites with static div', until i got stuck using google search url. The question is how do go to that url for saving the meaning of particular word. Also sometimes I m not able to get the "div" using inspect, in which the actual meaning is written. If I have to go to that "div", then it is must to reach to the url first. Is there any possibility for achieving this. If not just help me out in saving the input value i type on search bar. Thanks.
The Google Web Search API is offcially deprecated, Google excourage developer to use the Custom Search API which may provide solution.
Custom Search Element is a control that provides the user interface for a Google Custom Search Engine. Using the Custom Search Element JavaScript API, you can configure the Google Custom Search UI, search parameters, and result rendering.
It implements the Custom Search Element, which comprises the Custom Search engine and UI. Calling the constructor initializes the Custom Search service and UI; other methods allow you to control search queries, results, and UI.
Syntax:
google.search.CustomSearchControl.CustomSearch
Control(cseId, opt_options);
You can use setOnKeepCallback method. Specifies the optional text label displayed under each search result. Clicking this label triggers a callback into the specified object/method.
google.search.SearchControl.KEEP_LABEL_SAVE sets a label value of save.
I am completely new to the Facebook API. I would like to incorporate Facebook login into my application. I am using the Javascript SDK on the front-end to log the user in and retrieve the user_id and signed_request from Facebook. I then plan to send these two pieces of information via AJAX to my server (either php/hack (hhvm), node, java, or whichever language I can determine is quickest for decoding) every time my logged in user does an action on my application to validate if the user is indeed logged in and is the person they say they are. For me to accomplish this, I need to decode the signed_request, for example in php:
function parse_signed_request($signed_request) {
list($encoded_sig, $payload) = explode('.', $signed_request, 2);
$secret = "appsecret"; // Use your app secret here
// decode the data
$sig = base64_url_decode($encoded_sig);
$data = json_decode(base64_url_decode($payload), true);
// confirm the signature
$expected_sig = hash_hmac('sha256', $payload, $secret, $raw = true);
if ($sig !== $expected_sig) {
error_log('Bad Signed JSON signature!');
return null;
}
return $data;
}
function base64_url_decode($input) {
return base64_decode(strtr($input, '-_', '+/'));
}
which then I will be able to extract the following JSON object:
{
"oauth_token": "{user-access-token}",
"algorithm": "HMAC-SHA256",
"expires": 1291840400,
"issued_at": 1291836800,
"user_id": "218471"
}
to be able to compare if the user_id the user sent over matches the one in the JSON object. Then if it matches I can complete my business logic (DB manipulation).
My big concern here is a user will be sending many requests to my server, so every time I will need to decode this signed_request which can really kill my server performance. I was thinking I maybe could call Facebook from my server, pass the user_id, and receive the signed_request string, which I can then match with the signed_request string the user sent over from the client_side and see if they match. This would be more efficient, but it does not seem Facebook offers anything like this. Is there any other methods besides the heavy performing decoding to validate a user? I have gone through quite a bit of the Facebook SDK's information but could not find a solution. If I must decode, which language/library would be the best performing at this type of operation?
PS. I plan on using cordova later to create a mobile app so I must use only Javascript on the front end and can't use a server language such as php to create html for the client.
Decoding the signed request will not kill your server. It's way fast than making an external request.
If you're using php you should look into the Facebook SDK for PHP and use this helper: https://developers.facebook.com/docs/php/FacebookJavaScriptLoginHelper/4.0.0
I want to use YQL to retrieve all 10-Q & 10-K files from SEC EDGAR database.
After ref to the discussions [1] & [2], I bump into some problem.
It seems that YQL cannot get search results from the search engine.
However, I can directly access the filing detail page.
Here is a jsfiddle shows the problem. Although both queries return success message, the query to the search engine returns a result of empty array.
Is there any other way to get all the html addresses of the detail filing pages without querying EDGAR search engine? Thanks.
Example code by using YQL shows below:
// results page from EDGAR search engine:
// fail to get data
var queryURL = "http://www.sec.gov/cgi-bin/browse-edgar?" +
"action=getcompany&CIK=0001326801&type=10-K&dateb=&owner=exclude&count=100";
// EDGAR 10-K detail filing page:
// success to fetch by YQL
var filingURL = "http://www.sec.gov/Archives/edgar/data/1326801/" +
"000132680114000007/0001326801-14-000007-index.htm";
$.get(queryURL).then(function() {
// get successful message, but get results of empty array
})
.then(function() {
$.get(filingURL).then(function() {
// get successful message, and get results of empty array
})
} )
The /cgi-bin URL is restricted by robots.txt, so YQL will honour that and not crawl the page.
You can see this happening by enabling diagnostics for the YQL query.
Add diagnostics=true to the YQL URL, like /v1/public/yql?diagnostics=true&callback=?
Look for the diagnostics field in the results. This contains information about the query and any URLs it visited.
I'm trying to load some php fql calls in to an ajax facebox (its like lightbox), i include a file with the following code in, but it does not work without the redirect ($my_url), is there any way to make the code ignore the redirect? Or do i need to use the javascript sdk from facebook?
$app_id = 'APP ID';
$app_secret = 'APP SECRET';
$my_url = 'LINK HERE';
$code = $_REQUEST["code"];
$bruger = "me()";
//auth user
if(empty($code)) {
$dialog_url = 'https://www.facebook.com/dialog/oauth?client_id='
. $app_id . '&redirect_uri=' . urlencode($my_url) ;
echo("<script>top.location.href='" . $dialog_url . "'</script>");
}
$token_url = 'https://graph.facebook.com/oauth/access_token?client_id='
. $app_id . '&redirect_uri=' . urlencode($my_url)
. '&client_secret=' . $app_secret
. '&code=' . $code;
// response is of the format "access_token=AAAC..."
$access_token = substr(file_get_contents($token_url), 13);
Have you tried using the Facebook PHP SDK? I highly recommend moving away from depreciated FQL as once you get to grips with the new API it is very easy to use
Once you have set up $facebook you can then just make API calls with $facebook->api('/query');
I'm not 100% sure what information you are trying to retrieve from your question however but I have put together a Lightbox / Facebook myself, even using tag data to pick people out of the picture so feel free to provide more detail and I will try to provide a more comprehensive answer
EDIT:
In order to build queries and test data returned you can use the Graph API Explorer and construct the query you want to test against your own Facebook Application
I'm trying to fetch friendlist
After you've setup $facebook then you can retrieve your friends by using
$fb_friends = $facebook->api("/me?fields=friends");
Next:
50 latest tagged photos and likes
To retrieve the latest 50 tagged photos and likes you'll need to do:
$fb_photos = $facebook->api("/me?fields=photos.limit(50)");
$fb_likes = $facebook->api("/me?fields=likes.limit(50)");
// I should mention that each time api() is called, the server makes a HTTPS request to Facebook, so it is worth condensing api queries to reduce load time:
$fb_data = $facebook->api("/me?fields=photos.limit(50),likes.limit(50)");
$fb_photos = $fb_data['photos'];
$fb_likes = $fb_data['likes'];
If you want the latest 50 of photos and likes chronologically sorted, you can array_merge() the two arrays and use a subval sort function for created_time, then just take the first 50.
I'm going to insert all the basic info into mysql (stuff like name, facebook id, email ect.) I'm going to use the fb id inserted into myslq, to view the friendlist, likes and photos
Right then, when you user logs on you will need to request extended permissions from them to access data like photos, likes and friends. When your user logs into facebook with the link you provide to them on your website, you need to call
$loginUrl = $facebook->getLoginUrl(array( 'scope' => 'read_stream,user_likes,user_photos,friends_photos,user_friends'));`
The scope is defined by the permissions stated here
Once you have been granted access to that information by the user you can retrieve profile information with $user = $facebook->api("/me"); and store the facebook ID, name etc in your database.
To get a better look at the data you'll be handing for photos then make sure that you click Get access token at the top of the page and grant permission to access photos and likes
Any more questions feel free to ask and I will provide more detail, hope that helps mate!
i would like to implement selective Framebursting for my iframe application.
My iframe is available at www.mywebsite.con/iframe.aspx?lic=1234
When the third party website hosting my iframe is (PayedWebsited1.con OR PayedWebsited2.con) AND the lic=1234 option also exists, display the iframe. For any other cheaters, display bananas!
How can i do it?
The thing is, that licence number won't help in any way - whether you will use server-side solution or in javascript. Cheaters will be able to see that licence number in PayedWebsite1.com.
As was said, you cannot get the parent frame location, but you can get the referrer - it equals to the parent frame, if your page is loaded in iframe.
if (window.top.location !== document.location) { // only if we're in iframe
// we get host of our referrer
var host = document.referrer.match(new RegExp("(http|https)://(.*?)/.*$"))[2];
host = host.toLowerCase(); // convert to lower case
var myHost = document.location.host.toLowerCase();
if (
host !== myHost // so we can click on links in an iframe
&& host !== 'payedwebsite1.com'
&& host !== 'payedwebsite2.com'
) {
window.top.location.href = document.location.href;
}
}
Be awared, that this technique can be beaten. More info at http://javascript.info/tutorial/clickjacking
For newer browsers, you can send special header:
X-Frame-Options: DENY
The logic keeps the same, only in server-side. Check Referrer, if PayedDomain or your own domain, just keep going. Otherwise, send this header.
If it is possible for your third party users to include a javascript file, or ideally send a request in ASP prior to drawing the page, this is what I would do:
Javascript
Build a ASP (I do PHP, so my example is in PHP) page on your server that checks the referrer and the license number to match an account in your database. The ASP file should then output javascript functions that will replace or insert into the element your specified iframe with a "one-time-use" key that you generate. The file might look similar to this:
<?php
$lic = $_GET['lic']; // Do better validation (short for demo purposes)
if (valid_license($lic, $_SERVER['HTTP_REFERER'])) {
$one_time_key = get_access_key($lic);
?>
function drawIframe() {
document.getElementById('iframe_target').innerHTML = "<iframe src='mysite.php?key=<?php echo $one_time_key;?>'></iframe>";
}
<?php
}
else {
echo "You are not authorized to use this service.";
}
Have your customer include this javascript code as a replacement of your iframe, in a fashion similar to this:
<script src="http://www.yoursite.com/preauth.php?lic=1234"></script>
<script>drawIframe();</script>
<div id="iframe_target"></div>
On the page that is loaded by the iframe, immediately check the key that you generated against the value passed to the iframe. If it is valid, immediately delete or change the status of the key so that you know it's been used. Then display appropriate application.
This javascript method will be the least painful method for your third party users, although it can be beat (users could change the "referer" that is sent to your server, although it is unlikely.)
ASP
If you can get your users to make a request to your url within their server, you will eliminate exposing any risky information like the license to the user. They could call something like $key = file_get_contents("http://www.yoursite.com/preauth.asp?lic=1234"); Immediately after they can output the iframe with the one time use key that you just generated.
Due to security, your browser will not allow you to use javascript to detect the URL of the parent page (i.e. the page that contains the iframe that displays your page).
The only solutions I can think of are:
Insist that users of your iframe.aspx page, include an additional GET param that states the domain that they are using.
Use the Request.UrlReferrer to get the referrer
On the page which you render, you should have a literal that, should you want to prevent the person from framing your page, you can simply add the javascript required to force the frames.
Unfortunately if Javascript is disabled, this will render your code useless...
Hope this helps?
protected void page_load(object sender, EventArgs e)
{
bool killFrames = false;
if(Request.QueryString["lic"] == null)
killFrames = true;
if(!killFrames && Request.UrlReferrer != null)
{
// do some database check against the LIC and Referrer
// and set killFrames accordingly.
}
if(killFrames)
{
literalFrame.Text = "<script type=\"text/javascript\">if(top.location != location) { top.location.href = document.location.href; }</script>";
// or show the bananas
}
else
{
// render the page accordingly.
}
}
I will try to point a solution for your general problem and not this particular technical problem, which as far as i know is impossible for security precautions done by all web browsers.
You need some sort of hand-shake between their app and yours and that haves to be done server-side.
Every PayedWebsite should have a password (or if they hava a static IP you could use that). Internally on their server (using CURL may be) they shold send you -via POST- their password; then you return a token that is used in the iframe.
iframe.aspx?lic=1234&token=d16evg5tr44e0trty45xp6es5
And the token only works once; so the process haves to be repeated every time the iframe needs to be opened. And you refuse every connection that doesn't include a valid token.
I'm not a .NET expert, but it looks like your solution could be easily solved by tracking the referral header that the client sends to your page when loading the iframe content.
You may want to refer to another question regarding refer headers:
how should we validate http header referrer in aspx .net
Basically, you would do the following
Use the referral header to get the domain name
Look up the domain name in your database (to see if there was a license for that site)
Send the real page, or the bananas depending on the result of the match.
Global.asax did the trick!
Sub Application_BeginRequest(ByVal sender As Object, ByVal e As EventArgs)
Dim ref As String = HttpContext.Current.Request.ServerVariables("HTTP_REFERER")
If Not ref.Contains("PayedWebsited1") And Not ref.Contains("PayedWebsited2") Then
Response.Redirect("MYDOMAIN", True)
End If
End Sub
Thanks to all!