代码之家 › 专栏 › 技术社区 › Simon Breton

如何在python中比较Databricks笔记本中的两种模式

azure-databricks databricks pyspark python

1

Simon Breton · 技术社区 · 4 年前

我将使用databricks笔记本接收数据。我想根据我所期望的这些数据的模式来验证摄取的数据的模式。

    validation_schema = StructType([
      StructField("a", StringType(), True),
      StructField("b", IntegerType(), False),
      StructField("c", StringType(), False),
      StructField("d", StringType(), False)
    ])

    data_ingested_good = [("foo",1,"blabla","36636"),
     ("foo",2,"booboo","40288"),
     ("bar",3,"fafa","42114"),
     ("bar",4,"jojo","39192"),
     ("baz",5,"jiji","32432")
    ]

    data_ingested_bad = [("foo","1","blabla","36636"),
     ("foo","2","booboo","40288"),
     ("bar","3","fafa","42114"),
     ("bar","4","jojo","39192"),
     ("baz","5","jiji","32432")
    ]
     
    data_ingested_good.printSchema()
    data_ingested_bad.printSchema()
    validation_schema.printSchema()

我见过类似的问题,但答案总是用scala。

1 回复 | 直到 4 年前

1

Alex Ott 4 年前

这实际上取决于您的确切要求&要比较的模式的复杂性—例如,忽略可空性标志与将其考虑在内、列的顺序、对映射/结构/数组的支持等。此外,如果模式匹配与否,您希望看到差异还是只看到一个标志。

在最简单的情况下,它可以像以下一样简单-只需比较模式的字符串表示:

def compare_schemas(df1, df2):
  return df1.schema.simpleString() == df2.schema.simpleString()

Chispa 它具有更高级的模式比较功能-您可以调整检查,它将在安装后显示差异等(您只需 %pip install chispa )-如果架构不同,这将引发异常:

from chispa.schema_comparer import assert_schema_equality

assert_schema_equality(df1.schema, df2.schema)

2

0

Karthikeyan Rasipalay Durairaj 4 年前

list 比较。

dept = [("Finance",10), 
        ("Marketing",20), 
        ("Sales",30), 
        ("IT",40) 
      ]
deptColumns = ["dept_name","dept_id"]

dept1 = [("Finance",10,'999'), 
        ("Marketing",20,'999'), 
        ("Sales",30,'999'), 
        ("IT",40,'999') 
      ]
deptColumns1 = ["dept_name","dept_id","extracol"]

deptDF = spark.createDataFrame(data=dept, schema = deptColumns)
dept1DF = spark.createDataFrame(data=dept1, schema = deptColumns1)
deptDF_columns=deptDF.schema.names
dept1DF_columns=dept1DF.schema.names

list_difference = []
for item in dept1DF_columns:
  if item not in deptDF_columns:
     list_difference.append(item)

print(list_difference)