How can one parse HTML/XML and extract information from it?
Native XML Extensions
I prefer using one of the native XML extensions since they come bundled with PHP, are usually faster than all the 3rd party libs and give me all the control I need over the markup.
DOM
The DOM extension allows you to operate on XML documents through the DOM API with PHP 5. It is an implementation of the W3C's Document Object Model Core Level 3, a platform- and language-neutral interface that allows programs and scripts to dynamically access and update the content, structure and style of documents.
DOM is capable of parsing and modifying real world (broken) HTML and it can do XPath queries. It is based on libxml.
It takes some time to get productive with DOM, but that time is well worth it IMO. Since DOM is a language-agnostic interface, you'll find implementations in many languages, so if you need to change your programming language, chances are you will already know how to use that language's DOM API then.
How to use the DOM extension has been covered extensively on StackOverflow, so if you choose to use it, you can be sure most of the issues you run into can be solved by searching/browsing Stack Overflow.
A basic usage example and a general conceptual overview are available in other answers.
XMLReader
The XMLReader extension is an XML pull parser. The reader acts as a cursor going forward on the document stream and stopping at each node on the way.
XMLReader, like DOM, is based on libxml. I am not aware of how to trigger the HTML Parser Module, so chances are using XMLReader for parsing broken HTML might be less robust than using DOM where you can explicitly tell it to use libxml's HTML Parser Module.
A basic usage example is available in another answer.
XML Parser
This extension lets you create XML parsers and then define handlers for different XML events. Each XML parser also has a few parameters you can adjust.
The XML Parser library is also based on libxml, and implements a SAX style XML push parser. It may be a better choice for memory management than DOM or SimpleXML, but will be more difficult to work with than the pull parser implemented by XMLReader.
SimpleXml
The SimpleXML extension provides a very simple and easily usable toolset to convert XML to an object that can be processed with normal property selectors and array iterators.
SimpleXML is an option when you know the HTML is valid XHTML. If you need to parse broken HTML, don't even consider SimpleXml because it will choke.
A basic usage example is available, and there are lots of additional examples in the PHP Manual.
3rd Party Libraries (libxml based)
If you prefer to use a 3rd-party lib, I'd suggest using a lib that actually uses DOM/libxml underneath instead of string parsing.
FluentDom
FluentDOM provides a jQuery-like fluent XML interface for the DOMDocument in PHP. Selectors are written in XPath or CSS (using a CSS to XPath converter). Current versions extend the DOM implementing standard interfaces and add features from the DOM Living Standard. FluentDOM can load formats like JSON, CSV, JsonML, RabbitFish and others. Can be installed via Composer.
HtmlPageDom
Wa72\HtmlPageDom is a PHP library for easy manipulation of HTML
documents using DOM. It requires DomCrawler from Symfony2
components for traversing
the DOM tree and extends it by adding methods for manipulating the
DOM tree of HTML documents.
phpQuery
phpQuery is a server-side, chainable, CSS3 selector driven Document Object Model (DOM) API based on jQuery JavaScript Library.
The library is written in PHP5 and provides additional Command Line Interface (CLI).
This is described as "abandonware and buggy: use at your own risk" but does appear to be minimally maintained.
laminas-dom
The Laminas\Dom component (formerly Zend_DOM) provides tools for working with DOM documents and structures. Currently, we offer Laminas\Dom\Query, which provides a unified interface for querying DOM documents utilizing both XPath and CSS selectors.
This package is considered feature-complete, and is now in security-only maintenance mode.
fDOMDocument
fDOMDocument extends the standard DOM to use exceptions at all occasions of errors instead of PHP warnings or notices. They also add various custom methods and shortcuts for convenience and to simplify the usage of DOM.
sabre/xml
sabre/xml is a library that wraps and extends the XMLReader and XMLWriter classes to create a simple "xml to object/array" mapping system and design pattern. Writing and reading XML is single-pass and can therefore be fast and require low memory on large xml files.
FluidXML
FluidXML is a PHP library for manipulating XML with a concise and fluent API.
It leverages XPath and the fluent programming pattern to be fun and effective.
3rd-Party (not libxml-based)
The benefit of building upon DOM/libxml is that you get good performance out of the box because you are based on a native extension. However, not all 3rd-party libs go down this route. Some of them listed below
PHP Simple HTML DOM Parser
An HTML DOM parser written in PHP5+ lets you manipulate HTML in a very easy way!
Require PHP 5+.
Supports invalid HTML.
Find tags on an HTML page with selectors just like jQuery.
Extract contents from HTML in a single line.
I generally do not recommend this parser. The codebase is horrible and the parser itself is rather slow and memory hungry. Not all jQuery Selectors (such as child selectors) are possible. Any of the libxml based libraries should outperform this easily.
PHP Html Parser
PHPHtmlParser is a simple, flexible, html parser which allows you to select tags using any css selector, like jQuery. The goal is to assiste in the development of tools which require a quick, easy way to scrape html, whether it's valid or not! This project was original supported by sunra/php-simple-html-dom-parser but the support seems to have stopped so this project is my adaptation of his previous work.
Again, I would not recommend this parser. It is rather slow with high CPU usage. There is also no function to clear memory of created DOM objects. These problems scale particularly with nested loops. The documentation itself is inaccurate and misspelled, with no responses to fixes since 14 Apr 16.
HTML 5
You can use the above for parsing HTML5, but there can be quirks due to the markup HTML5 allows. So for HTML5 you may want to consider using a dedicated parser. Note that these are written in PHP, so suffer from slower performance and increased memory usage compared to a compiled extension in a lower-level language.
HTML5DomDocument
HTML5DOMDocument extends the native DOMDocument library. It fixes some bugs and adds some new functionality.
Preserves html entities (DOMDocument does not)
Preserves void tags (DOMDocument does not)
Allows inserting HTML code that moves the correct parts to their proper places (head elements are inserted in the head, body elements in the body)
Allows querying the DOM with CSS selectors (currently available: *, tagname, tagname#id, #id, tagname.classname, .classname, tagname.classname.classname2, .classname.classname2, tagname[attribute-selector], [attribute-selector], div, p, div p, div > p, div + p, and p ~ ul.)
Adds support for element->classList.
Adds support for element->innerHTML.
Adds support for element->outerHTML.
HTML5
HTML5 is a standards-compliant HTML5 parser and writer written entirely in PHP. It is stable and used in many production websites, and has well over five million downloads.
HTML5 provides the following features.
An HTML5 serializer
Support for PHP namespaces
Composer support
Event-based (SAX-like) parser
A DOM tree builder
Interoperability with QueryPath
Runs on PHP 5.3.0 or newer
Regular Expressions
Last and least recommended, you can extract data from HTML with regular expressions. In general using Regular Expressions on HTML is discouraged.
Most of the snippets you will find on the web to match markup are brittle. In most cases they are only working for a very particular piece of HTML. Tiny markup changes, like adding whitespace somewhere, or adding, or changing attributes in a tag, can make the RegEx fails when it's not properly written. You should know what you are doing before using RegEx on HTML.
HTML parsers already know the syntactical rules of HTML. Regular expressions have to be taught for each new RegEx you write. RegEx are fine in some cases, but it really depends on your use-case.
You can write more reliable parsers, but writing a complete and reliable custom parser with regular expressions is a waste of time when the aforementioned libraries already exist and do a much better job on this.
Also see Parsing Html The Cthulhu Way
Books
If you want to spend some money, have a look at
PHP Architect's Guide to Webscraping with PHP
I am not affiliated with PHP Architect or the authors.
Try Simple HTML DOM Parser.
A HTML DOM parser written in PHP 5+ that lets you manipulate HTML in a very easy way!
Require PHP 5+.
Supports invalid HTML.
Find tags on an HTML page with selectors just like jQuery.
Extract contents from HTML in a single line.
Download
Note: as the name suggests, it can be useful for simple tasks. It uses regular expressions instead of an HTML parser, so will be considerably slower for more complex tasks. The bulk of its codebase was written in 2008, with only small improvements made since then. It does not follow modern PHP coding standards and would be challenging to incorporate into a modern PSR-compliant project.
Examples:
How to get HTML elements:
// Create DOM from URL or file
$html = file_get_html('http://www.example.com/');
// Find all images
foreach($html->find('img') as $element)
echo $element->src . '<br>';
// Find all links
foreach($html->find('a') as $element)
echo $element->href . '<br>';
How to modify HTML elements:
// Create DOM from string
$html = str_get_html('<div id="hello">Hello</div><div id="world">World</div>');
$html->find('div', 1)->class = 'bar';
$html->find('div[id=hello]', 0)->innertext = 'foo';
echo $html;
Extract content from HTML:
// Dump contents (without tags) from HTML
echo file_get_html('http://www.google.com/')->plaintext;
Scraping Slashdot:
// Create DOM from URL
$html = file_get_html('http://slashdot.org/');
// Find all article blocks
foreach($html->find('div.article') as $article) {
$item['title'] = $article->find('div.title', 0)->plaintext;
$item['intro'] = $article->find('div.intro', 0)->plaintext;
$item['details'] = $article->find('div.details', 0)->plaintext;
$articles[] = $item;
}
print_r($articles);
Just use DOMDocument->loadHTML() and be done with it. libxml's HTML parsing algorithm is quite good and fast, and contrary to popular belief, does not choke on malformed HTML.
Why you shouldn't and when you should use regular expressions?
First off, a common misnomer: Regexps are not for "parsing" HTML. Regexes can however "extract" data. Extracting is what they're made for. The major drawback of regex HTML extraction over proper SGML toolkits or baseline XML parsers are their syntactic effort and varying reliability.
Consider that making a somewhat dependable HTML extraction regex:
<a\s+class="?playbutton\d?[^>]+id="(\d+)".+? <a\s+class="[\w\s]*title
[\w\s]*"[^>]+href="(http://[^">]+)"[^>]*>([^<>]+)</a>.+?
is way less readable than a simple phpQuery or QueryPath equivalent:
$div->find(".stationcool a")->attr("title");
There are however specific use cases where they can help.
Many DOM traversal frontends don't reveal HTML comments <!--, which however are sometimes the more useful anchors for extraction. In particular pseudo-HTML variations <$var> or SGML residues are easy to tame with regexps.
Oftentimes regular expressions can save post-processing. However HTML entities often require manual caretaking.
And lastly, for extremely simple tasks like extracting <img src= urls, they are in fact a probable tool. The speed advantage over SGML/XML parsers mostly just comes to play for these very basic extraction procedures.
It's sometimes even advisable to pre-extract a snippet of HTML using regular expressions /<!--CONTENT-->(.+?)<!--END-->/ and process the remainder using the simpler HTML parser frontends.
Note: I actually have this app, where I employ XML parsing and regular expressions alternatively. Just last week the PyQuery parsing broke, and the regex still worked. Yes weird, and I can't explain it myself. But so it happened.
So please don't vote real-world considerations down, just because it doesn't match the regex=evil meme. But let's also not vote this up too much. It's just a sidenote for this topic.
Note, this answer recommends libraries that have now been abandoned for 10+ years.
phpQuery and QueryPath are extremely similar in replicating the fluent jQuery API. That's also why they're two of the easiest approaches to properly parse HTML in PHP.
Examples for QueryPath
Basically you first create a queryable DOM tree from an HTML string:
$qp = qp("<html><body><h1>title</h1>..."); // or give filename or URL
The resulting object contains a complete tree representation of the HTML document. It can be traversed using DOM methods. But the common approach is to use CSS selectors like in jQuery:
$qp->find("div.classname")->children()->...;
foreach ($qp->find("p img") as $img) {
print qp($img)->attr("src");
}
Mostly you want to use simple #id and .class or DIV tag selectors for ->find(). But you can also use XPath statements, which sometimes are faster. Also typical jQuery methods like ->children() and ->text() and particularly ->attr() simplify extracting the right HTML snippets. (And already have their SGML entities decoded.)
$qp->xpath("//div/p[1]"); // get first paragraph in a div
QueryPath also allows injecting new tags into the stream (->append), and later output and prettify an updated document (->writeHTML). It can not only parse malformed HTML, but also various XML dialects (with namespaces), and even extract data from HTML microformats (XFN, vCard).
$qp->find("a[target=_blank]")->toggleClass("usability-blunder");
.
phpQuery or QueryPath?
Generally QueryPath is better suited for manipulation of documents. While phpQuery also implements some pseudo AJAX methods (just HTTP requests) to more closely resemble jQuery. It is said that phpQuery is often faster than QueryPath (because of fewer overall features).
For further information on the differences see this comparison on the wayback machine from tagbyte.org. (Original source went missing, so here's an internet archive link. Yes, you can still locate missing pages, people.)
Advantages
Simplicity and Reliability
Simple to use alternatives ->find("a img, a object, div a")
Proper data unescaping (in comparison to regular expression grepping)
Simple HTML DOM is a great open-source parser:
simplehtmldom.sourceforge
It treats DOM elements in an object-oriented way, and the new iteration has a lot of coverage for non-compliant code. There are also some great functions like you'd see in JavaScript, such as the "find" function, which will return all instances of elements of that tag name.
I've used this in a number of tools, testing it on many different types of web pages, and I think it works great.
One general approach I haven't seen mentioned here is to run HTML through Tidy, which can be set to spit out guaranteed-valid XHTML. Then you can use any old XML library on it.
But to your specific problem, you should take a look at this project: http://fivefilters.org/content-only/ -- it's a modified version of the Readability algorithm, which is designed to extract just the textual content (not headers and footers) from a page.
For 1a and 2: I would vote for the new Symfony Componet class DOMCrawler ( DomCrawler ).
This class allows queries similar to CSS Selectors. Take a look at this presentation for real-world examples: news-of-the-symfony2-world.
The component is designed to work standalone and can be used without Symfony.
The only drawback is that it will only work with PHP 5.3 or newer.
This is commonly referred to as screen scraping, by the way. The library I have used for this is Simple HTML Dom Parser.
We have created quite a few crawlers for our needs before. At the end of the day, it is usually simple regular expressions that do the thing best. While libraries listed above are good for the reason they are created, if you know what you are looking for, regular expressions is a safer way to go, as you can handle also non-valid HTML/XHTML structures, which would fail, if loaded via most of the parsers.
I recommend PHP Simple HTML DOM Parser.
It really has nice features, like:
foreach($html->find('img') as $element)
echo $element->src . '<br>';
This sounds like a good task description of W3C XPath technology. It's easy to express queries like "return all href attributes in img tags that are nested in <foo><bar><baz> elements." Not being a PHP buff, I can't tell you in what form XPath may be available. If you can call an external program to process the HTML file you should be able to use a command line version of XPath.
For a quick intro, see http://en.wikipedia.org/wiki/XPath.
Third party alternatives to SimpleHtmlDom that use DOM instead of String Parsing: phpQuery, Zend_Dom, QueryPath and FluentDom.
Yes you can use simple_html_dom for the purpose. However I have worked quite a lot with the simple_html_dom, particularly for web scraping and have found it to be too vulnerable. It does the basic job but I won't recommend it anyways.
I have never used curl for the purpose but what I have learned is that curl can do the job much more efficiently and is much more solid.
Kindly check out this link:scraping-websites-with-curl
Advanced Html Dom is a simple HTML DOM replacement that offers the same interface, but it's DOM-based which means none of the associated memory issues occur.
It also has full CSS support, including jQuery extensions.
QueryPath is good, but be careful of "tracking state" cause if you didn't realise what it means, it can mean you waste a lot of debugging time trying to find out what happened and why the code doesn't work.
What it means is that each call on the result set modifies the result set in the object, it's not chainable like in jquery where each link is a new set, you have a single set which is the results from your query and each function call modifies that single set.
in order to get jquery-like behaviour, you need to branch before you do a filter/modify like operation, that means it'll mirror what happens in jquery much more closely.
$results = qp("div p");
$forename = $results->find("input[name='forename']");
$results now contains the result set for input[name='forename'] NOT the original query "div p" this tripped me up a lot, what I found was that QueryPath tracks the filters and finds and everything which modifies your results and stores them in the object. you need to do this instead
$forename = $results->branch()->find("input[name='forname']")
then $results won't be modified and you can reuse the result set again and again, perhaps somebody with much more knowledge can clear this up a bit, but it's basically like this from what I've found.
For HTML5, html5 lib has been abandoned for years now. The only HTML5 library I can find with a recent update and maintenance records is html5-php which was just brought to beta 1.0 a little over a week ago.
I created a library named PHPPowertools/DOM-Query, which allows you to crawl HTML5 and XML documents just like you do with jQuery.
Under the hood, it uses symfony/DomCrawler for conversion of CSS selectors to XPath selectors. It always uses the same DomDocument, even when passing one object to another, to ensure decent performance.
Example use :
namespace PowerTools;
// Get file content
$htmlcode = file_get_contents('https://github.com');
// Define your DOMCrawler based on file string
$H = new DOM_Query($htmlcode);
// Define your DOMCrawler based on an existing DOM_Query instance
$H = new DOM_Query($H->select('body'));
// Passing a string (CSS selector)
$s = $H->select('div.foo');
// Passing an element object (DOM Element)
$s = $H->select($documentBody);
// Passing a DOM Query object
$s = $H->select( $H->select('p + p'));
// Select the body tag
$body = $H->select('body');
// Combine different classes as one selector to get all site blocks
$siteblocks = $body->select('.site-header, .masthead, .site-body, .site-footer');
// Nest your methods just like you would with jQuery
$siteblocks->select('button')->add('span')->addClass('icon icon-printer');
// Use a lambda function to set the text of all site blocks
$siteblocks->text(function( $i, $val) {
return $i . " - " . $val->attr('class');
});
// Append the following HTML to all site blocks
$siteblocks->append('<div class="site-center"></div>');
// Use a descendant selector to select the site's footer
$sitefooter = $body->select('.site-footer > .site-center');
// Set some attributes for the site's footer
$sitefooter->attr(array('id' => 'aweeesome', 'data-val' => 'see'));
// Use a lambda function to set the attributes of all site blocks
$siteblocks->attr('data-val', function( $i, $val) {
return $i . " - " . $val->attr('class') . " - photo by Kelly Clark";
});
// Select the parent of the site's footer
$sitefooterparent = $sitefooter->parent();
// Remove the class of all i-tags within the site's footer's parent
$sitefooterparent->select('i')->removeAttr('class');
// Wrap the site's footer within two nex selectors
$sitefooter->wrap('<section><div class="footer-wrapper"></div></section>');
[...]
Supported methods :
[x] $ (1)
[x] $.parseHTML
[x] $.parseXML
[x] $.parseJSON
[x] $selection.add
[x] $selection.addClass
[x] $selection.after
[x] $selection.append
[x] $selection.attr
[x] $selection.before
[x] $selection.children
[x] $selection.closest
[x] $selection.contents
[x] $selection.detach
[x] $selection.each
[x] $selection.eq
[x] $selection.empty (2)
[x] $selection.find
[x] $selection.first
[x] $selection.get
[x] $selection.insertAfter
[x] $selection.insertBefore
[x] $selection.last
[x] $selection.parent
[x] $selection.parents
[x] $selection.remove
[x] $selection.removeAttr
[x] $selection.removeClass
[x] $selection.text
[x] $selection.wrap
Renamed 'select', for obvious reasons
Renamed 'void', since 'empty' is a reserved word in PHP
NOTE :
The library also includes its own zero-configuration autoloader for PSR-0 compatible libraries. The example included should work out of the box without any additional configuration. Alternatively, you can use it with composer.
You could try using something like HTML Tidy to cleanup any "broken" HTML and convert the HTML to XHTML, which you can then parse with a XML parser.
I have written a general purpose XML parser that can easily handle GB files. It's based on XMLReader and it's very easy to use:
$source = new XmlExtractor("path/to/tag", "/path/to/file.xml");
foreach ($source as $tag) {
echo $tag->field1;
echo $tag->field2->subfield1;
}
Here's the github repo: XmlExtractor
Another option you can try is QueryPath. It's inspired by jQuery, but on the server in PHP and used in Drupal.
XML_HTMLSax is rather stable - even if it's not maintained any more. Another option could be to pipe you HTML through Html Tidy and then parse it with standard XML tools.
There are many ways to process HTML/XML DOM of which most have already been mentioned. Hence, I won't make any attempt to list those myself.
I merely want to add that I personally prefer using the DOM extension and why :
iit makes optimal use of the performance advantage of the underlying C code
it's OO PHP (and allows me to subclass it)
it's rather low level (which allows me to use it as a non-bloated foundation for more advanced behavior)
it provides access to every part of the DOM (unlike eg. SimpleXml, which ignores some of the lesser known XML features)
it has a syntax used for DOM crawling that's similar to the syntax used in native Javascript.
And while I miss the ability to use CSS selectors for DOMDocument, there is a rather simple and convenient way to add this feature: subclassing the DOMDocument and adding JS-like querySelectorAll and querySelector methods to your subclass.
For parsing the selectors, I recommend using the very minimalistic CssSelector component from the Symfony framework. This component just translates CSS selectors to XPath selectors, which can then be fed into a DOMXpath to retrieve the corresponding Nodelist.
You can then use this (still very low level) subclass as a foundation for more high level classes, intended to eg. parse very specific types of XML or add more jQuery-like behavior.
The code below comes straight out my DOM-Query library and uses the technique I described.
For HTML parsing :
namespace PowerTools;
use \Symfony\Component\CssSelector\CssSelector as CssSelector;
class DOM_Document extends \DOMDocument {
public function __construct($data = false, $doctype = 'html', $encoding = 'UTF-8', $version = '1.0') {
parent::__construct($version, $encoding);
if ($doctype && $doctype === 'html') {
#$this->loadHTML($data);
} else {
#$this->loadXML($data);
}
}
public function querySelectorAll($selector, $contextnode = null) {
if (isset($this->doctype->name) && $this->doctype->name == 'html') {
CssSelector::enableHtmlExtension();
} else {
CssSelector::disableHtmlExtension();
}
$xpath = new \DOMXpath($this);
return $xpath->query(CssSelector::toXPath($selector, 'descendant::'), $contextnode);
}
[...]
public function loadHTMLFile($filename, $options = 0) {
$this->loadHTML(file_get_contents($filename), $options);
}
public function loadHTML($source, $options = 0) {
if ($source && $source != '') {
$data = trim($source);
$html5 = new HTML5(array('targetDocument' => $this, 'disableHtmlNsInDom' => true));
$data_start = mb_substr($data, 0, 10);
if (strpos($data_start, '<!DOCTYPE ') === 0 || strpos($data_start, '<html>') === 0) {
$html5->loadHTML($data);
} else {
#$this->loadHTML('<!DOCTYPE html><html><head><meta charset="' . $encoding . '" /></head><body></body></html>');
$t = $html5->loadHTMLFragment($data);
$docbody = $this->getElementsByTagName('body')->item(0);
while ($t->hasChildNodes()) {
$docbody->appendChild($t->firstChild);
}
}
}
}
[...]
}
See also Parsing XML documents with CSS selectors by Symfony's creator Fabien Potencier on his decision to create the CssSelector component for Symfony and how to use it.
The Symfony framework has bundles which can parse the HTML, and you can use CSS style to select the DOMs instead of using XPath.
With FluidXML you can query and iterate XML using XPath and CSS Selectors.
$doc = fluidxml('<html>...</html>');
$title = $doc->query('//head/title')[0]->nodeValue;
$doc->query('//body/p', 'div.active', '#bgId')
->each(function($i, $node) {
// $node is a DOMNode.
$tag = $node->nodeName;
$text = $node->nodeValue;
$class = $node->getAttribute('class');
});
https://github.com/servo-php/fluidxml
JSON and array from XML in three lines:
$xml = simplexml_load_string($xml_string);
$json = json_encode($xml);
$array = json_decode($json,TRUE);
Ta da!
There are several reasons to not parse HTML by regular expression. But, if you have total control of what HTML will be generated, then you can do with simple regular expression.
Above it's a function that parses HTML by regular expression. Note that this function is very sensitive and demands that the HTML obey certain rules, but it works very well in many scenarios. If you want a simple parser, and don't want to install libraries, give this a shot:
function array_combine_($keys, $values) {
$result = array();
foreach ($keys as $i => $k) {
$result[$k][] = $values[$i];
}
array_walk($result, create_function('&$v', '$v = (count($v) == 1)? array_pop($v): $v;'));
return $result;
}
function extract_data($str) {
return (is_array($str))
? array_map('extract_data', $str)
: ((!preg_match_all('#<([A-Za-z0-9_]*)[^>]*>(.*?)</\1>#s', $str, $matches))
? $str
: array_map(('extract_data'), array_combine_($matches[1], $matches[2])));
}
print_r(extract_data(file_get_contents("http://www.google.com/")));
I've created a library called HTML5DOMDocument that is freely available at https://github.com/ivopetkov/html5-dom-document-php
It supports query selectors too that I think will be extremely helpful in your case. Here is some example code:
$dom = new IvoPetkov\HTML5DOMDocument();
$dom->loadHTML('<!DOCTYPE html><html><body><h1>Hello</h1><div class="content">This is some text</div></body></html>');
echo $dom->querySelector('h1')->innerHTML;
The best method for parse xml:
$xml='http://www.example.com/rss.xml';
$rss = simplexml_load_string($xml);
$i = 0;
foreach ($rss->channel->item as $feedItem) {
$i++;
echo $title=$feedItem->title;
echo '<br>';
echo $link=$feedItem->link;
echo '<br>';
if($feedItem->description !='') {
$des=$feedItem->description;
} else {
$des='';
}
echo $des;
echo '<br>';
if($i>5) break;
}
There are many ways:
In General:
Native XML Extensions: they come bundled with PHP, are usually faster than all the 3rd party libs, and give me all the control you need over the markup.
DOM: DOM is capable of parsing and modifying real-world (broken) HTML and it can do XPath queries. It is based on libxml.
XML Reader: XMLReader, like DOM, is based on libxml. The XMLReader extension is an XML pull parser. The reader acts as a cursor going forward on the document stream and stopping at each node on the way
XML Parser: This extension lets you create XML parsers and then define handlers for different XML events. Each XML parser also has a few parameters you can adjust. It implements a SAX style XML push parser.
Simple XML: The SimpleXML extension provides a very simple and easily usable toolset to convert XML to an object that can be processed with normal property selectors and array iterators.
3rd Party Libraries [ libxml based ]:
FluentDom - Repo: FluentDOM provides a jQuery-like fluent XML interface for the DOMDocument in PHP. It can load formats like JSON, CSV, JsonML, RabbitFish and others. Can be installed via Composer.
HtmlPageDom: is a PHP library for easy manipulation of HTML documents using It requires DomCrawler from Symfony2 components for traversing the DOM tree and extends it by adding methods for manipulating the DOM tree of HTML documents.
ZendDOM: Zend_Dom provides tools for working with DOM documents and structures. Currently, they offer Zend_Dom_Query, which provides a unified interface for querying DOM documents utilizing both XPath and CSS selectors.
QueryPath: QueryPath is a PHP library for manipulating XML and HTML. It is designed to work not only with local files but also with web services and database resources. It implements much of the jQuery interface (including CSS-style selectors), but it is heavily tuned for server-side use. Can be installed via Composer.
fDOM Document: fDOMDocument extends the standard DOM to use exceptions at all occasions of errors instead of PHP warnings or notices. They also add various custom methods and shortcuts for convenience and to simplify the usage of DOM.
Sabre/XML: sabre/xml is a library that wraps and extends the XMLReader and XMLWriter classes to create a simple "xml to object/array" mapping system and design pattern. Writing and reading XML is single-pass and can therefore be fast and require low memory on large xml files.
FluidXML: FluidXML is a PHP library for manipulating XML with a concise and fluent API. It leverages XPath and the fluent programming pattern to be fun and effective.
3rd Party Libraries [ Not libxml based ]:
PHP Simple HTML DOM Parser: An HTML DOM parser written in PHP5+ lets you manipulate HTML in a very easy way, It Requires PHP 5+. Also Supports invalid HTML.
It Extracts contents from HTML in a single line. The codebase is horrible and very slow in working.
PHP Html Parser: HPHtmlParser is a simple, flexible, HTML parser that allows you to select tags using any CSS selector, like jQuery. The goal is to assist in the development of tools that require a quick, easy way to scrape HTML, whether it's valid or not. It is slow and takes too much CPU Power.
Ganon (recommended): A universal tokenizer and HTML/XML/RSS DOM Parser. It has the Ability to manipulate elements and their attributes. It Supports invalid HTML and UTF8. It Can perform advanced CSS3-like queries on elements (like jQuery -- namespaces supported). A HTML beautifier (like HTML Tidy). Minify CSS and Javascript. It Sort attributes, change character case, correct indentation, etc.
Extensible. The Operations separated into smaller functions for easy overriding and
Fast and Easy to use.
Web Services:
If you don't feel like programming PHP, you can also use Web services. ScraperWiki's external interface allows you to extract data in the form you want for use on the web or in your own applications. You can also extract information about the state of any scraper.
I have shared all the resources, you can choose according to your taste, usefulness, etc.
Related
I just started learning about the HTML custom elements, and through reading a series of intros, tutorials, and documentation, I think I have a good handle on how it works, but I have a philosophical question on the proper way to use or not use the <template> tag.
Custom elements give you the ability to encapsulate new functionality, simplifying the structure of your HTML document, and allowing you to simply insert a <my-custom-element>...</my-custom-element> tag instead of <div class="my-custom-element"><span class="part1">...</span><span class="part2">...</span></div>.
The class definition for the element then sets up the structure and functionality of that element. A bunch of the tutorials then describe how to use <template>...</template> and <slot>...</slot> to set up the contents of the custom element. You would then have to then include the template code in every HTML document in which you want to use the element rather than setting it up in the custom element class's constructor. Doesn't this run counter to the fact that custom elements help simplify and encapsulate functionality in a way that makes them more portable? Or am I misunderstanding the proper usage and/or placement of the template within the document?
Looking through SO, the closest I can find to addressing this is this question:
How to stamp out template in self contained custom elements with vanilla js?
But the answer essentially sidesteps this all together and says "Don't use <template>," and so doesn't really clear up my confusion.
Actually <template> elements can be imported from another document via HTML Imports, along with the Javascript code that will define the custom element:
<link rel="import" src="my-custom-element.html">
...
<custom-element></custom-element>
So it doesn't need to be included in a every HTML document. This post shows a minimal example.
HTML Imports are implemented only in Chrome and Opera. If you want to use them in the with Firefox and Safari you'll need to use the HTML Imports polyfill.
On the other hand and for the moment, Mozilla and Apple don't intend to implement HTML Imports natively in their respective browsers. Therefore they recommend to define custom elements with pure Javascript modules (with import or <script src="...">), and promote template literals strings instead, which offer some advantages (variables, functions), but are sometimes more complicated to code in an IDE (because of their string representation).
Maybe in the future standard HTML modules will be adopted by all browsers, and <template> will come back in the spotlight...
Note that without HTML Imports you can still import yourself some HTML documents with fetch():
fetch( "template.html" )
.then( stream => stream.text() )
.then( text =>
customElements.define( "c-e", class extends HTMLElement {
constructor() {
super()
this.attachShadow( { mode: 'open'} )
.innerHTML = text
}
} )
)
Update 2019
HTML Imports won't be supported natively after Chrome 73. You should then use the other solutions listed above (the polyfill, an alternate module loader, JS import, or a direct download with fetch).
Disclaimer: I'm an author of the rich-component library mentioned below.
After some time of experimenting with custom elements and recently raising a full blown project based solely upon them I'd like to share my insights on this:
any component tiny as it is, is a candidate to grow to some beast
HTML part of it may grow to a point where it is very non-convenient to keep it within JS
do use template, built and parsed once and from that point cloned and injected into the shadow root - this is the same best practice as to use document fragment instead of mutating a living DOM
if the template contents should be changed from component's instance to instance - some kind of data binding framework may be used, and if minimalist approach on those is taken - it might still be easier and more performant to deal with a cloned-from-template document fragment than operate on string or template literals
In order to not write the same dozens of lines over and over again I've prepared rich-component library, which:
normalizes some API for template provisioning and all those 'clone template, create shadow, inject template's content into it' lines of repeating code
known to fetch html contents when html URL is provided
caches the templates so the fetch is done only once
I have been using a DOMParser object to parse a text string to an XML tree. However it is not available in the context of a Web Worker (and neither is, of course, document.ELEMENT_NODE or the various other constants that would be needed). Is there any other way to do that?
Please note that I do not want to manipulate the DOM of the current page. The XML file won't contain HTML elements or anything of the sort. In fact, I do not want to touch the document object at all. I simply want to provide a text string like the following:
<car color="blue"><driver/></car>
...and get back a suitable tree structure and a way to traverse it. I also do not care about schema validation or anything fancy. I know about XML for <SCRIPT>, which many may find useful (hence I'm linking to it here), however its licensing is not really suitable for me. I'm not sure if jQuery includes an XML parser (I'm fairly new to this stuff), but even if it does (and it is usable inside a Worker), I would not include an extra ~50K lines of code just for this function.
I suppose I could write a simple XML parser in JavaScript, I'm just wondering if I'm missing a quicker option.
according to the spec
The DOM APIs (Node objects, Document objects, etc) are not available to workers in this version of this specification.
I guess thats why DOMParser is not availlable, but I don't really understand why that decision was made. (fetching and processing an XML document in a WebWorker does not seems unreasonnable)
but you can import other tools available: a "Cross Platform XML Parsing in JavaScript"
At this point I like to share my parser: https://github.com/tobiasnickel/tXml
with its tXml() method you can parse a string into an object and it takes only 0.5kb minified + gzipped
When I look in the dojo documentation for template all I get is for dijit and examples only show you been able to use them in a widgit. I'm looking for the equivalent of the below methods in js prototype
var tmpl = new Template(url)
tmpl.evaluate(templateObj)
Does dojo have a template method that you can use in a dojo.declare( class ){} like you can do in js prototype. If not how could I go about similar functionality
Thanks
You may be interested in dojo.string.substitute (you'll need to dojo.require("dojo.string")).
http://dojotoolkit.org/api/dojo/string/substitute
[Edit] Also, if you're interested in acquiring a template for use in substitution from a URL on the same server, you may also want to look into dojo.cache (which is also what is often used to fetch widget templates):
http://dojotoolkit.org/reference-guide/dojo/cache.html
To clarify missingno's response, I don't think dojo.parser is what you're interested in right now; its job is to scan the DOM and transform DOM nodes into widgets and other Dojo components. dijit._Templated only uses dojo.parser when child widgets are involved (i.e. widgetsInTemplate is true); on the other hand, it uses dojo.string.substitute in all cases, to initially parse ${...} strings (e.g. ${id}) in the template.
I don't know Prototype, but this sounds like dojo.parser stuff. It is what is used by dijit._Templated behind the scenes (you can chack that in the source code if you want...)
Just note that you probably wouldn't need to cal this yourself - there is parseOnLoad=true for automatically parsing your initial HTML.
I want to write a web application that allows users to enter any HTML that can occur inside a <div> element. This HTML will then end up being displayed to other users, so I want to make sure that the site doesn't open people up to XSS attacks.
Is there a nice library in Python that will clean out all the event handler attributes, <script> elements and other Javascript cruft from HTML or a DOM tree?
I am intending to use Beautiful Soup to regularize the HTML to make sure it doesn't contain unclosed tags and such. But, as far as I can tell, it has no pre-packaged way to strip all Javascript.
If there is a nice library in some other language, that might also work, but I would really prefer Python.
I've done a bunch of Google searching and hunted around on pypi, but haven't been able to find anything obvious.
Related
Sanitising user input using Python
As Klaus mentions, the clear consensus in the community is to use BeautifulSoup for these tasks:
soup = BeautifulSoup.BeautifulSoup(html)
for script_elt in soup.findAll('script'):
script_elt.extract()
html = str(soup)
Whitelist approach to allowed tags, attributes and their values is the only reliable way. Take a look at Recipe 496942: Cross-site scripting (XSS) defense
What is wrong with existing markup languages such as used on this very site?
You could use BeautifulSoup. It allows you to traverse the markup structure fairly easily, even if it's not well-formed. I don't know that there's something made to order that works only on script tags.
I would honestly look at using something like bbcode or some other alternative markup with it.
Eric,
Have you thought about using a 'SAX' type parser for the HTML? I'm really not sure
though that it would ignore the events properly though. It would also be a bit harder to construct than using something like Beautiful Soup. Handling syntax errors may be a problem with SAX as well.
What I like to do in situations like this is to construct python objects (subclassed from an XML_Element class) from the parsed HTML. Then remove any undesired objects from the tree, and finally re-serialize the objects back to html. It's not all that hard in python.
Regards,
I'm using Django and Apache to serve webpages. My JavaScript code currently includes a data object with values to be displayed in various HTML widgets based on the user's selection from a menu of choices. I want to derive these data from a Python dictionary. I think I know how to embed the JavaScript code in the HTML, but how do I embed the data object in that script (on the fly) so the script's functions can use it?
Put another way, I want to create a JavaScript object or array from a Python dictionary, then insert that object into the JavaScript code, and then insert that JavaScript code into the HTML.
I suppose this structure (e.g., data embedded in variables in the JavaScript code) is suboptimal, but as a newbie I don't know the alternatives. I've seen write-ups of Django serialization functions, but these don't help me until I can get the data into my JavaScript code in the first place.
I'm not (yet) using a JavaScript library like jQuery.
n.b. see 2018 update at the bottom
I recommend against putting much JavaScript in your Django templates - it tends to be hard to write and debug, particularly as your project expands. Instead, try writing all of your JavaScript in a separate script file which your template loads and simply including just a JSON data object in the template. This allows you to do things like run your entire JavaScript app through something like JSLint, minify it, etc. and you can test it with a static HTML file without any dependencies on your Django app. Using a library like simplejson also saves you the time spent writing tedious serialization code.
If you aren't assuming that you're building an AJAX app this might simply be done like this:
In the view:
from django.utils import simplejson
def view(request, …):
js_data = simplejson.dumps(my_dict)
…
render_template_to_response("my_template.html", {"my_data": js_data, …})
In the template:
<script type="text/javascript">
data_from_django = {{ my_data }};
widget.init(data_from_django);
</script>
Note that the type of data matters: if my_data is a simple number or a string from a controlled source which doesn't contain HTML, such as a formatted date, no special handling is required. If it's possible to have untrusted data provided by a user you will need to sanitize it using something like the escape or escapejs filters and ensure that your JavaScript handles the data safely to avoid cross-site scripting attacks.
As far as dates go, you might also want to think about how you pass dates around. I've almost always found it easiest to pass them as Unix timestamps:
In Django:
time_t = time.mktime(my_date.timetuple())
In JavaScript, assuming you've done something like time_t = {{ time_t }} with the results of the snippet above:
my_date = new Date();
my_date.setTime(time_t*1000);
Finally, pay attention to UTC - you'll want to have the Python and Django date functions exchange data in UTC to avoid embarrassing shifts from the user's local time.
EDIT : Note that the setTime in javascript is in millisecond whereas the output of time.mktime is seconds. That's why we need to multiply by 1000
2018 Update: I still like JSON for complex values but in the intervening decade the HTML5 data API has attained near universal browser support and it's very convenient for passing simple (non-list/dict) values around, especially if you might want to have CSS rules apply based on those values and you don't care about unsupported versions of Internet Explorer.
<div id="my-widget" data-view-mode="tabular">…</div>
let myWidget = document.getElementById("my-widget");
console.log(myWidget.dataset.viewMode); // Prints tabular
somethingElse.addEventListener('click', evt => {
myWidget.dataset.viewMode = "list";
});
This is a neat way to expose data to CSS if you want to set the initial view state in your Django template and have it automatically update when JavaScript updates the data- attribute. I use this for things like hiding a progress widget until the user selects something to process or to conditionally show/hide errors based on fetch outcomes or even something like displaying an active record count using CSS like #some-element::after { content: attr(data-active-transfers); }.
For anyone who might be having a problems with this, be sure you are rendering your json object under safe mode in the template. You can manually set this like this
<script type="text/javascript">
data_from_django = {{ my_data|safe }};
widget.init(data_from_django);
</script>
As of mid-2018 the simplest approach is to use Python's JSON module, simplejson is now deprecated. Beware, that as #wilblack mentions you need to prevent Django's autoescaping either using safe filter or autoescape tag with an off option. In both cases in the view you add the contents of the dictionary to the context
viewset.py
import json
def get_context_data(self, **kwargs):
context['my_dictionary'] = json.dumps(self.object.mydict)
and then in the template you add as #wilblack suggested:
template.html
<script>
my_data = {{ my_dictionary|safe }};
</script>
Security warning:
json.dumps does not escape forward slashes: an attack is {'</script><script>alert(123);</script>': ''}. Same issue as in other answers. Added another answer hopefully fixing it.
You can include <script> tags inside your .html templates, and then build your data structures however is convenient for you. The template language isn't only for HTML, it can also do Javascript object literals.
And Paul is right: it might be best to use a json module to create a JSON string, then insert that string into the template. That will handle the quoting issues best, and deal with deep structures with ease.
It is suboptimal. Have you considered passing your data as JSON using django's built in serializer for that?
See the related response to this question. One option is to use jsonpickle to serialize between Python objects and JSON/Javascript objects. It wraps simplejson and handles things that are typically not accepted by simplejson.
Putting Java Script embedded into Django template is rather always bad idea.
Rather, because there are some exceptions from this rule.
Everything depends on the your Java Script code site and functionality.
It is better to have seperately static files, like JS, but the problem is that every seperate file needs another connect/GET/request/response mechanism. Sometimes for small one, two liners code os JS to put this into template, bun then use django templatetags mechanism - you can use is in other templates ;)
About objects - the same. If your site has AJAX construction/web2.0 like favour - you can achieve very good effect putting some count/math operation onto client side. If objects are small - embedded into template, if large - response them in another connection to avoid hangind page for user.
Fixing the security hole in the answers by #willblack and #Daniel_Kislyuk.
If the data is untrusted, you cannot just do
viewset.py
def get_context_data(self, **kwargs):
context['my_dictionary'] = json.dumps(self.object.mydict)
template.html
<script>
my_data = {{ my_dictionary|safe }};
</script>
because the data could be something like
{"</script><script>alert(123);</script>":""}
and forward slashes aren't escaped by default. Clearly the escaping by json.dumps may not 100% match the escaping in Javascript, which is where the problems come from.
Fixed solution
As far as I can tell, the following fixes the problem:
<script>
my_data = JSON.parse("{{ my_dictionary|escapejs }}");
</script>
If there are still issues, please post in the comments.