Package org.apache.nutch.parse.tika
Class DOMContentUtils
- java.lang.Object
-
- org.apache.nutch.parse.tika.DOMContentUtils
-
public class DOMContentUtils extends Object
A collection of methods for extracting content from DOM trees. This class holds a few utility methods for pulling content out of DOM nodes, such as getOutlinks, getText, etc.
-
-
Constructor Summary
Constructors Constructor Description DOMContentUtils(Configuration conf)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description StringgetBase(Node node)If Node contains a BASE tag then it's HREF is returned.voidgetOutlinks(URL base, ArrayList<Outlink> outlinks, List<org.apache.tika.sax.Link> tikaExtractedOutlinks)voidgetOutlinks(URL base, ArrayList<Outlink> outlinks, Node node)voidgetText(StringBuffer sb, Node node)This is a convinience method, equivalent togetText(sb, node, false).booleangetTitle(StringBuffer sb, Node node)This method takes aStringBufferand a DOMNode, and will append the content text found beneath the firsttitlenode to theStringBuffer.voidsetConf(Configuration conf)
-
-
-
Constructor Detail
-
DOMContentUtils
public DOMContentUtils(Configuration conf)
-
-
Method Detail
-
setConf
public void setConf(Configuration conf)
-
getText
public void getText(StringBuffer sb, Node node)
This is a convinience method, equivalent togetText(sb, node, false).- Parameters:
sb- aStringBufferused to store content text found beneath the DOM node... if any existsnode- a DOMNodeto check for content text
-
getTitle
public boolean getTitle(StringBuffer sb, Node node)
This method takes aStringBufferand a DOMNode, and will append the content text found beneath the firsttitlenode to theStringBuffer.- Parameters:
sb- aStringBufferused to store content text found beneath the DOM node... if any existsnode- a DOMNodeto check for content text- Returns:
- true if a title node was found, false otherwise
-
getBase
public String getBase(Node node)
If Node contains a BASE tag then it's HREF is returned.- Parameters:
node- a DOMNodeto check for a BASE tag- Returns:
- HREF if one exists
-
getOutlinks
public void getOutlinks(URL base, ArrayList<Outlink> outlinks, Node node)
This method finds all anchors below the supplied DOMnode, and creates appropriateOutlinkrecords for each (relative to the suppliedbaseURL), and adds them to theoutlinksArrayList.Links without inner structure (tags, text, etc) are discarded, as are links which contain only single nested links and empty text nodes (this is a common DOM-fixup artifact, at least with nekohtml).
-
-