Package org.apache.nutch.parse
Class ParseSegment
- java.lang.Object
-
- org.apache.hadoop.conf.Configured
-
- org.apache.nutch.util.NutchTool
-
- org.apache.nutch.parse.ParseSegment
-
- All Implemented Interfaces:
Configurable,Tool
public class ParseSegment extends NutchTool implements Tool
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classParseSegment.ParseSegmentMapperstatic classParseSegment.ParseSegmentReducer
-
Field Summary
Fields Modifier and Type Field Description static StringSKIP_TRUNCATED-
Fields inherited from class org.apache.nutch.util.NutchTool
currentJob, currentJobNum, numJobs, results, status
-
-
Constructor Summary
Constructors Constructor Description ParseSegment()ParseSegment(Configuration conf)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static booleanisTruncated(Content content)Checks if the page's content is truncated.static voidmain(String[] args)voidparse(Path segment)intrun(String[] args)Map<String,Object>run(Map<String,Object> args, String crawlId)Runs the tool, using a map of arguments.-
Methods inherited from class org.apache.nutch.util.NutchTool
getProgress, getStatus, killJob, setConf, stopJob
-
Methods inherited from class org.apache.hadoop.conf.Configured
getConf
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.hadoop.conf.Configurable
getConf, setConf
-
-
-
-
Field Detail
-
SKIP_TRUNCATED
public static final String SKIP_TRUNCATED
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ParseSegment
public ParseSegment()
-
ParseSegment
public ParseSegment(Configuration conf)
-
-
Method Detail
-
isTruncated
public static boolean isTruncated(Content content)
Checks if the page's content is truncated.- Parameters:
content- the responseContent- Returns:
- If the page is truncated
true. When it is not, or when it couldn't be determined,false.
-
parse
public void parse(Path segment) throws IOException, InterruptedException, ClassNotFoundException
-
run
public Map<String,Object> run(Map<String,Object> args, String crawlId) throws Exception
Description copied from class:NutchToolRuns the tool, using a map of arguments. May return results, or null.- Specified by:
runin classNutchTool- Parameters:
args- aMapof arguments to be run with the toolcrawlId- a crawl identifier to associate with the tool invocation- Returns:
- Map results object if tool executes successfully otherwise null
- Throws:
Exception- if there is an error during the tool execution
-
-