I want to fetch this site
https://www.film-fish.com/modern-mindless-action
to fetch the IMDB IDs of all movies listed there.
The problem is that the page loads all movies listed there just after scrolling down. So, a simple wget doesn't work.
Even if I scroll to the bottom of the page and view the source code, I do not see the last movie in the list (Hard Kill (2020)).
So the problem seems to be that the content is being created via JavaScript.
Has anybody a tip on how to achieve that?
So the problem seems to be that the content is being created via a js
script. Has anybody a tip on how to achieve that?
Indeed, executing JavaScript code is beyond scope of GNU Wget. You would need browser automation tool. If you know some Node.js or JavaScript I suggest taking look at PhantomJS Quick Start, Page Automation. Please take look at first example in 2nd link, you should be probably able to rework to your needs, i.e. instruct page to scroll down using JavaScript then extract what you need using JavaScript.
I want to add twitter card meta tags to my website. i cannot add static tags since the content attribute in the meta tag has to change dynamically.
Plz help if someone has a solution.
You can't, at least not in any way that Twitter is going to recognise.
When Twitter fetches the page, it is always going to get the meta elements that are in the HTML, never ones added with client side JavaScript.
If you are changing content dynamically then make use of the history API to update the URI, and make sure that the server will generate all the content for each URI when the URI is used as an entry point (you can still use JavaScript to generate the content when moving from another page on the site).
You can generate pages dynamically (server side) as long as you don't intend to change them after. Think about the process in the same way you would build landing pages for SEO purposes. Twitterbot is in many ways really similar to Googlebot and similar indexers.
See https://dev.twitter.com/blog/twitter-cards-tips-tricks for a few examples of sites doing this.
I have tried to place the slider on every page on this website: http://atripathi.com
It works on the homepage, but doesn't work on any of the other pages (About, Services, etc.)
I know it's probably an easy fix, but I can't get it at the moment.
Thank you for any help or suggestions!
Looks like the original suggestion above is correct. I'm seeing slider javascript includes at the top of your homepage that aren't on the other pages.
Generally, a good way of troubleshooting is to make copies of both pages, index-c.php and about-c.php perhaps, and start removing everything that isn't pertinent to the trouble you're having (other HTML, css includes, etc.) until you get down to only the slider on the page. Once you've done that, you might notice that the one page is slightly different than the other, making it work. You can copy back and forth until it does.
The other possibility is that there's a relative path problem somewhere, because your one page is inside a folder (though I'm guessing you have a .htaccess redirect to a root folder page)? So if all else fails, move the reduced about-c.php to the root folder and see if that then works. If so, you know it's a path problem.
Hope these suggestions help.
I see that jQuery is being included on all your pages but the cycle plugin is only included on the home page. You should be able to update your template(s) to fix this.
I have developed a small component which can be put in to any website. Now, I want to develop a code that could demonstrate how would my component look like on any website.
So, the person would come to my page and put in his URL and then my code should embed my custom JS/CSS in to the downloaded HTML and display it. Something like this.
Here, like the feedback tab, I want to show my component any where on that page.
Try a bookmarklet.
Create a piece of javascript that adds your code into the page such as the following:
javascript:(function(){var%20script=document.createElement('script');script.src='http://www.example.org/js/example.js';document.getElementsByTagName('head')[0].appendChild(script);})()
Add it as the href of a link like so:
Link Text Here
Tell your users to drag the link to their bookmark toolbar and click on it on different websites to try your code out.
Some examples: http://www.reclaimprivacy.org/, http://www.readability.com/bookmarklets
In the example you linked, they are requesting the page specified in the url querystring parameter on the server, and then doing more or less the following steps:
In the <head> tag they are adding a <base href="url" /> tag to the document. The base tag will make any relative links in the document treat the value in the href attribute as their root. This is how they are getting around broken css / images. (The base tag is supported by all browsers)
At the end of the document (IE the </body> tag) they are injecting the javascript that runs their demos.
They serve the modified HTML requested to the browser.
All of this is pretty straight forward in implementation. You could use regular expressions to match the <head> and </body> tags for steps 1 and 2 respectively. Depending on the server platform how you actually request the page will vary, but here are some links to get you started:
C# - HttpWebRequest object documentation
PHP - HttpRequest::send
Nathan's answer is the closest to how we have done the demo feature at WebEngage. To make such a demo functional, you'll need to create a Javascript widget that can be embedded on third party sites. syserr0r's answer on creating a bookmarklet is the simplest approach to do so. Our's is a JAVA backend and we use HttpClient to fetch the responses. As Nathan suggested, we parse the response, sanitize it and add our widget Javascript to the response. The widget JS code takes it on from there to render the Feedback tab and load a demo short survey.
Disclosure: I am a co-founder and ceo at WebEngage.
You can not do this with JQuery due to cross site scripting restrictions.
I suggest you write a PHP script that downloads the URL specified by the user and includes your widget code and then echo it back to the user.
I recommend using bookmarklets. I've made a bookmarklet generator for adding jQuery-enabled bookmarklets to a page to make development easier.
There's a caliper bookmarklet on the page that you can mess around with just to show an example of it working.
Full disclosure, this is something I've made, I'm not trying to be spammy as I think it's relevant: zbooks
You could make an iframe page, which loads their page in the iframe, and uses javascript to inject your code into the iframe.
Here is my approach...
http://jsfiddle.net/L2kEf/
html
<iframe src="http://www.bing.com"></iframe>
<div>I am div</div>
css
div { background: red; position: absolute; top: 20px; width: 100px; left:20px;}
iframe{width: 100%; height: 500px;}
you can add javascript/jquery too, so you could do something like,
jQuery //not 100% sure it would work coz of cross browser thingy, but you know, worth a try.
$('div').click(function (){
$('iframe').contents().html('changed');///
});
if this can't change any of the contents, you can display a dialog, to say it would normally work if it was in your website, then use #syserr0r approach for bookmarked users, for better results, since you are offering this kinda services, to developers, im sure they would know about bookmarking, my approach would be rarely used :) so hope it helps.
I had a problem of a similiar nature, and the main obstacle is the cross-domain policy.
You have to ask the user to put your code in a <script src="..."> or create a proxy solution that would add your code for them.
I went for the proxy and here are my observations:
it's easy to create a basic proxy in php - there are some php proxies on sourceforge and Ben Alman has created a simple php proxy for AJAX. Based on those I was able to create a php proxy altering the content properly in one day.
after that I spent a lot of time making it work with more and more sites with issues. You can never create a perfect proxy.
As an alternative (sa long as you are non-commercial) you can use http://www.jmarshall.com/tools/cgiproxy/ and put the site in an iframe and then do whatever you want to do with the iframes document, as it's in your domain thanks to the proxy. You can access iframeDOMnode.contentWindow.document then, etc.
You can create a Crossrider extension which your users can download.
Then simply add this to your App/Extension code:
appAPI.dom.addRemoteJS("http://yourdomain.com/file.js")
Your users can then download the extension (it works cross-browser for Internet Explorer, Chrome and Firefox) and it will load your JS code on every page load.
You can get an approximation of what it will look like using a iframe. Take a look at that link for an example.
http://jsfiddle.net/jzaun/5PjRy/
The issue with this appoch is that you can't move your DIV(s) when the page scrolls, they are in effect just floating over the iframe. There is no way around this as cross-domain scripting wont let you access the iframe's document to monitor scroll events.
The only other option you have for a better fitting example would be to load the page from the server side in whatever scripting language you are using and load that into the iframe (or into a div, etc.) and you can use javascript all you want as the page is coming from your domain.
For your example of what will your widget look like I imagine floating your DIV(s) over an iframe would give enough of an idea.
Please note the example you gave is using the server side method, not the iframe method.
I agree with the bookmarklet strategy.
I'm a fan of http://bookmarklets.heroku.com/, which lets you generate bookmarklets easily, inject jQuery, etc.
I am trying to set up a masterpage that contains a javascript popup subroutine that can be used in multiple web pages. The popup already works in a single page environment. I now want to migrate it to a master page. Any ideas will be greatly appreciated. I already searched this site and tried a couple of the suggestions to no avail. W small working example would help.
Thanks
Bill
Just take what you have in your single page and place it in the master page. It is really that simple. :) Then just replace the main body of the HTML in the master page with the content sections and you should be ready to go.
One thing I did discover was that for some reason the derived page had to be in the same folder as the master page.
Most likely caused by the master page referencing a subdirectory that works for the master page, but not the internal pages themselves. One workaround would be using
ScriptManager.RegisterClientScriptInclude(string Key, string URL)
like so:
ScriptManager.RegisterClientScriptInclude("uniqueIdentifier", "~/javascript/myjs.js");
Alternatively, you can embed the file as a resource and use
ScriptManager.RegisterClientScriptResource(Page, Type, Key);
like so:
ScriptManager.RegisterClientScriptResource(this.Page, typeOf(Page), "resourceKey");
this is my current favourite, seems to be most robust and flexible. Simply include the js files in your master then create an isntance on your pages
be sure to use the scriptmanager class to add the javascript