代码之家  ›  专栏  ›  技术社区  ›  tree em

将从文本文件加载的列表传递到Spark sql中的sql查询

  •  2
  • tree em  · 技术社区  · 6 年前

    , 作为分隔符:

    val csv = spark.read.text("src/main/resources/in/insight/account_issues.txt")
    
    //implicits
    import spark.sqlContext.implicits._
    
    val string_account = csv.map(_.getString(0)).collect.toList.toString()
    //print(string_account)
    
    val query = s"""(SELECT
                   |    ACCOUNT_NUMBER,
                   |    CASE WHEN STMT.CRF_TYPE='CREDIT' THEN STMT.AMOUNT_LCY
                   |        ELSE NULL
                   |    END as 'CreditAmount',
                   |    CASE WHEN STMT.CRF_TYPE='DEBIT' THEN STMT.AMOUNT_LCY
                   |        ELSE  NULL
                   |    END as 'DebitAmount',
                   |    STMT.BOOKING_DATE,
                   |    STMT.VALUE_DATE,
                   |    CRF_TYPE
                   |FROM [InsightLanding].[dbo].[v_STMT_ENTRY] AS STMT
                   |    LEFT JOIN [InsightWarehouse].[dbo].[v_Account] AS A ON a.AccountNum = STMT.ACCOUNT_NUMBER
                   |
                   |WHERE STMT.MIS_DATE='$BusinessDate'
                   | AND STMT.ACCOUNT_NUMBER IN ($string_account) ) tmp """.stripMargin
    
    val responseWithSelectedColumns = spark
      .read
      .format("jdbc")
      .option("url", url)
      .option("driver", driver)
      .option("dbtable", query)
      .load()
    

    我无法获取作品,而是获取错误:

    : 'List' is not a recognized built-in function name
    

    我的代码有什么问题?

    1 回复  |  直到 6 年前
        1
  •  4
  •   Shaido MadHadders    6 年前

    当您创建 string_account 你用 toString() List(...)

    scala> List(1,2,3).toString()
    res0: String = List(1, 2, 3)
    

    你想用的是 mkString(",") :

    scala> List(1,2,3).mkString(",")
    res1: String = "1,2,3"
    

    在这种情况下,这将是:

    val string_account = csv.map(_.getString(0)).collect.toList.mkString(",")
    

    注意:可以很容易地将括号添加到 字符串帐户 mkString("(", ",", ")") .