Develop Reference

Facebook group chat - fetch complete chat content without paging

What I want
I want to be able to copy/paste the entire content of a chat to memory
so I can extract included YouTube urls from it.
What I know
As you may know, the group chat(s) run on a separate url and are loaded page by page. Normally you go to the previous page either by simply scrolling upwards, or by clicking on a show previous link (works differently on different devices I think).
Things I tried
Sadly I can't find the urls to either anymore, but ...
Add a script to Chrome console
The point was to add a script that went looking for the show previous link and clicked it.
Add a start=0 parameter to the url
This assumes you can find out the actual url, either manually or through something like Fiddler.
The idea was that you add something like ?start=0 to the url. This would cause the paging to start from the very first record and load all.
Both solutions didn't work.
Possibly this is because Facebook made these options obsolete. It's my impression that Facebook initially provided more dev options than it does now.
My question
What can I do to fully load chat content?

Not really sure what this has to do with C#, but i'll give a C# solution anyways. My solution would be to use something such as HTMLAgilityPack to get the InnerHTML from a page once it's loaded, although this will obviously require some type of authentication, so for this I suggest using something like a WebClient and sending along Auth credentials with whatever it is you're doing, OR just create a method to login, then use the same webclient to access chats via URL, use DownloadString() to get the contents of the page then using HTMLAgilityPack's methods to get the InnerHTML of whatever the chat box is called/indentified as.

Right now this is the nearest thing I can find:
https://www.facebook.com/help/community/question/?id=10200611181580779
There is a way to see your complete chat history on Facebook easily.
By this method you can also see Photos or videos you've shared on
Facebook. Your Wall posts etc. -- 'A copy of what you've shared on
Facebook' Follow these steps:
Go to 'Account Settings'
Click on 'Download a copy of your Facebook data' from bottom of General section
Then click 'Start My Archive' -- It may take a little while for gather your photos, wall posts, messages, and other information.
(Usually 20 to 60 minutes)
Once Archive generated Download it.
Extract and open 'index.html' from downloaded folder
Now you can see 'Messages' on bottom of the page, click it.
Done!
I got a response in my mail way faster than 20 minutes.
You will get a mail with a link to a zip file, containing your archive:
In the html folder you find: messages.htm
For that I can write a script that looks for YouTube URLs in that file.

PHP HttpRequest to create a web page - how to handle long response times?

I am currently using javascript and XMLHttpRequest on a static html page to create a view of a record in Zotero. This works nicely except for one thing: The page html title.
I can of course also change the <title>...</title> tag, but if someone wants to post the view to for example facebook the static title on the web page will be shown there.
I can't think of any way to fix this with just a static page with javascript. I believe I need a dynamically created page from a server that does something similar to XMLHttpRequest.
For PHP there is HTTPRequest. Now to the problem. In the javascript version I can use asynchronous calls. With PHP I think I need synchronous calls. Is that something to worry about?
Is there perhaps some other way to handle this that I am not aware of?
UPDATE: It looks like those trying to answer are not at all familiar with Zotero. I should have been more clear. Zotero is a reference db located at http://zotero.org/. It has an API that can be used through XMLHttpRequest (which is what I said above).
Now I can not use that in my scenario which I described above. So I want to call the Zotero server from my server instead. (Through PHP or something else.)
(If you are not familiar with the concepts it might be hard to understand and answer the question. Of course.)
UPDATE 2: For those interested in how Facebook scraps an URL you post there, please test here: https://developers.facebook.com/tools/debug
As you can see by testing there no javascript is run.

Sorry, im not sure if i understand what you are trying to ask, are you just wanting to change the pages title?
Why not use javascript?
document.title = newTitle

Facebook expects the title (or opengraph :title tags) to be present when it fetches the page. It won't execyte any JavaScript for you to fill in the blanks.
A cool workaround would be to detect the Facebook scraper with PHP by parsing the User Agent string, and serving a version of the page with the information already filled in by PHP instead of JavaScript.
As far as I know, the Facebook scraper uses this header for User Agent: "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
You can check to see if part of that string is present in the header and load the page accordingly.
if (strpos($_SERVER['HTTP_USER_AGENT'], 'facebookexternalhit') !== false)
{
//synchronously load the title and opengraph tags here.
}
else
{
//load the page normally
}

How to catch a certain part of a website?

Suppose that we have a website. We want to show a specified part of this site in another site, like a table of data that shows latest news, and we want to show this part in our website with javascript.
Is this possible? Are there any more information needed?
We all know that with this code:
<iframe src="http://www.XYZ.com">
</iframe>
we can load all website, but how to load a specific part of a website?

I think your best bet is JQuery.load(), but I'm not up to speed on whether there are crossdomain problems with that approach. • Just checked and there are.
You can use PHP to load the output of a page with file_get_contents() or similar and then scrape out what you want for your own output.
Example:
$str = file_get_contents("http://ninemsn.com.au/");
// Reallly shouldn't use regex with HTML.
$region = '/(\<div id\=\"tabloid\_2\" class\=\"tabloids\">)(.*)(\<\/div\>)/';
preg_match($region, $str, $matches);
echo $matches[0];
Finally, something to keep in the back of your mind is that many of the larger websites that developers may want to get content from for their own website offer APIs to easily and efficiently obtain information from their site. YouTube, Twitter and a number of photo sharing sites are good examples of these. JSON and XML are probably the most common data formats that you will receive from these APIs.
Here are some examples of APIs that produce usable JSON:
YouTube video feed: https://gdata.youtube.com/feeds/api/users/Martyaced/uploads?alt=json
Twitter feed: http://api.twitter.com/1/statuses/user_timeline.json?screen_name=Martyaced

Generate embed code given any url

My goal is to take user-submitted urls and generate video embed code. Any ideas on this could be done?
I've seen multiple sites do this including myvidster. Autoembed is a paid option written in php, however I"m using ruby/rails.

Have a look at oEmbed and its associated Ruby gem. It supports Youtube, Vimeo, Hulu, blip.tv out of the box.

If you are writing this yourself you will need to parse the URL for each site you are trying to accept video content from. For instance a Youtube link could be used something like this:
if url.match(/.*\.youtube\.com\/watch\?v=(.{11})/) || url.match(/youtu\.be\/(.{11})/)
%{<iframe width="440" height="275" src="https://www.youtube.com/embed/#{$1}?rel=0&wmode=transparent"></iframe>}.html_safe
else #other video site conditions here
end
Where url is the string passed in by the user and $1 is the the ID you parsed out and the iframe string code is specific to the site you are using.

Ajax page part load and Google

I have some div on page loaded from server by ajax, but in the scenario google and other search engine don't index the content of this div. The only solution I see, it's recognize when page get by search robot and return complete page without ajax.
1) Is there more simple way?
2) How distinguish humans and robots?

You could also provide a link to the non-ajax version in your sitemap, and when you serve that file (to the robot), you make sure to have included a canonical link-element to the "real" page you want users to see:
<html>
<head>
[...]
<link rel="canonical" href="YOUR_CANONICAL_URL_HERE" />
[...]
</head>
<body>
[...]
YOUR NON_AJAX_CONTENT_HERE
</body>
</html>
edit: if this solution is not appropriate (some comments below points out that that this solution is non-standard and only supported by the "big-three"), you might have to re-think whether you should make the non-ajax version the standard solution, and use JavaScript to hide/show the information instead of fetching it via AJAX. If it is business critical information that is fetched, you have to realize that not all users have JavaScript enabled, and thus they won't be able to see this information. A progressive enhancement approach might be more appropriate in this case.

Google gets antsy if you are trying to show different things to you users than to crawlers. I suggest simply caching your query or whatever it is that needs AJAX and then using AJAX to replace only what you need to change. You still haven't really explained what's in this div that only AJAX can provide. If you can do it without AJAX then you should be, not just for SEO but for braille readers, mobile devices and people without javascript.

You can specify a sitemap in your robots.txt. That sitemap should be a list of your static pages. You should not be giving to Google a different page at the same URL, so you should have a different URL with static and dynamic content. Typically, the static URL is .../blog/03/09/i-bought-a-puppy and dynamic URL is something like .../search/puppy.