代码之家  ›  专栏  ›  技术社区  ›  srinivasa

在hadoop中处理非结构化和多行CSV

  •  -2
  • srinivasa  · 技术社区  · 12 年前

    我希望在Hadoop Mapreduce中处理数据,我的数据格式低于非结构化、多行和未终止的报价。

        2/1/2013 5:16,Edward Felton,2,8/1/2012 3:57,Working on all the digital elements for our big event in Sydney in a couple of weeks... for more visit http://www.xy.com/au/geworks/,324005862,2,18200695
        12/28/2012 19:28,Laura McCullum,2,7/26/2012 18:03,"The Day You Give Them Jive  <br>
    <a href="http://youtu.be/qfq9LVD2Qr4" > http://youtu.be/qfq9LVD2Qr4 <br>
     <br>
    'Like' if you have always wanted to destroy a cube!",502114904,2,18400313
        11/21/2012 13:35,Timothy Widdowson,4,8/17/2012 12:38,"Can a table really replace a laptop...
    
    With the new Windows tablets on the horizon and the Apple / Android devices out there I have been wondering if it is possible to really work with just and tablet. 
    
    My mission:
    -For one whole week I will be working with just my iPad. 
    
    Hardware:
    -Apple iPad
    -Apple keyboard.
    -Apple to HDMI connector.
    -HDMI capable monitor.
    - InCase iPad stand.
    
    :-)",105001439,1,19301609
        3/15/2013 13:43,Mary Romeo,3,8/16/2012 22:23,"HOW TO SHORTEN LONG LINKS YOU'RE POSTING <br>
    The attached image describes how to shorten a long url before posting it.  In 4 easy steps the 3-4 line urls can become a tiny link to post.",213022329,1,19901561
        11/30/2012 2:17,Lu Yin Zhong,3,8/29/2012 1:29,working on 2013 comms plan...need big ideas!!,302014449,2,20300666
        3/5/2013 22:15,Tim Steigert,12,8/29/2012 15:36,"Looking up 1024 email addresses. Manually? Probably a day! Doing it with SSOget, the add-in for  #[&quot;excel&quot;]? 5 minutes! Effort saved and  #[&quot;productivity&quot;] gained? Priceless! Now go get it and enjoy it for yourself! :)<br>http://sc.xy.com/*SSOget @@@data@@@{&quot;image&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;}",100011871,11,20400713
        11/1/2012 20:46,Pranay Jain,2,8/30/2012 14:26,Do people agree with the iCloud restrictions that Airwatch will put on Personal iOS devices that have email?,212065316,0,20700913
        11/9/2012 18:32,Monica Sharma,5,9/7/2012 11:42,hhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh gh gh gh gh ghghghghgghhhghghghghgh hg h gh gh,502000192,5,21400516
    

    请给我提供如何处理提到的数据的代码片段? 提前感谢!!!!!!!

    1 回复  |  直到 12 年前
        1
  •  0
  •   harpun    12 年前

    因为要处理多行数据,所以不能使用简单的 TextInputFormat 访问您的数据。因此,您需要使用自定义 InputFormat CSV文件。

    目前Hadoop中没有内置的处理多行CSV文件的方法(请参见 https://issues.apache.org/jira/browse/MAPREDUCE-2208 ),但幸运的是,github上有代码,您可以尝试: https://github.com/mvallebr/CSVInputFormat .

    就未终止的报价而言,可能有必要首先对数据进行预处理和清理。一个简单的规则是,如果在报价之前或之后没有分隔符,则对报价进行转义( " ):

    • 逃生: a"b => a\"b
    • 保持不变: a;"b a";b

    另一个选项是纠正生成无效CSV的应用程序,以正确的方式转义数据。