代码之家 › 专栏 › 技术社区 › Piyush Jindal

清管器输出位置验证失败

snappy apache-pig gruntjs hadoop

3

Piyush Jindal · 技术社区 · 11 年前

在执行下面给定的脚本时,我得到了下面提到的以下错误。我尝试过使用现有目录和非现有目录,但仍然收到相同的错误。由于我刚开始写猪脚本,请帮我解决这个问题。我尝试在应用快速转换后加载文件并存储在不同的位置。

密码

SET output.compression.enabled true;
SET mapred.map.output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
A = LOAD '/abc/def/' using PigStorage();
Store A into '/abc/def/snappy/' using PigStorage();

错误

Pig Stack Trace
---------------
ERROR 2116:
<file snappy_conversion.pig, line 4, column 0> Output Location Validation Failed for: '/abc/def/snappy/

org.apache.pig.impl.plan.VisitorException: ERROR 2116:
<file snappy_conversion.pig, line 4, column 0> Output Location Validation Failed for: '/abc/def/snappy/
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
        at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:285)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1324)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
        at org.apache.pig.PigServer.execute(PigServer.java:1241)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:604)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.NullPointerException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.pig.builtin.PigStorage.setStoreLocation(PigStorage.java:396)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:68)
        ... 22 more
================================================================================

1 回复 | 直到 11 年前

1

zsxwing 11 年前

问题是您设置了错误的配置。正确的设置是

SET output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;

mapred.map.output.compression.codec 用于设置映射器输出的中间文件的压缩格式。