从列表中的元素中删除尾随空格

我有一个Spark数据框架,其中给定的列是一些文本。我正试图清除文本并用逗号分隔,这样会输出一个包含单词列表的新列。

我遇到的问题是,该列表中的某些元素包含要删除的尾随空格。

代码 以下内容:

# Libraries
# Standard Libraries
from typing import Dict, List, Tuple

# Third Party Libraries
import pyspark
from pyspark.ml.feature import Tokenizer
from pyspark.sql import SparkSession
import pyspark.sql.functions as s_function


def tokenize(sdf, input_col="text", output_col="tokens"):
    # Remove email 
    sdf_temp = sdf.withColumn(
        colName=input_col,
        col=s_function.regexp_replace(s_function.col(input_col), "[\w\.-]+@[\w\.-]+\.\w+", ""))
    # Remove digits
    sdf_temp = sdf_temp.withColumn(
        colName=input_col,
        col=s_function.regexp_replace(s_function.col(input_col), "\d", ""))
    # Remove one(1) character that is not a word character except for
    # commas(,), since we still want to split on commas(,)
    sdf_temp = sdf_temp.withColumn(
        colName=input_col,
        col=s_function.regexp_replace(s_function.col(input_col), "[^a-zA-Z0-9,]+", " ")) 
    # Split the affiliation string based on a comma
    sdf_temp = sdf_temp.withColumn(
        colName=output_col,
        col=s_function.split(sdf_temp[input_col], ", "))

    return sdf_temp


if __name__ == "__main__":
    # Sample data
    a_1 = "Department of Bone and Joint Surgery, Ehime University Graduate"\
        " School of Medicine, Shitsukawa, Toon 791-0295, Ehime, Japan."\
        " shinyama@m.ehime-u.ac.jp." 
    a_2 = "Stroke Pharmacogenomics and Genetics, Fundació Docència i Recerca"\
        " Mútua Terrassa, Hospital Mútua de Terrassa, 08221 Terrassa, Spain."
    a_3 = "Neurovascular Research Laboratory, Vall d'Hebron Institute of Research,"\
        " Hospital Vall d'Hebron, 08035 Barcelona, Spain;catycarrerav@gmail.com"\
        " (C.C.). catycarrerav@gmail.com."

    data = [(1, a_1), (2, a_2), (3, a_3)]

    spark = SparkSession\
        .builder\
        .master("local[*]")\
        .appName("My_test")\
        .config("spark.ui.port", "37822")\
        .getOrCreate()
    sc = spark.sparkContext
    sc.setLogLevel("WARN")

    af_data = spark.createDataFrame(data, ["index", "text"])
    sdf_tokens = tokenize(af_data)
    # sdf_tokens.select("tokens").show(truncate=False)

输出

|[Department of Bone and Joint Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon , Ehime, Japan ]                                                |
|[Stroke Pharmacogenomics and Genetics, Fundaci Doc ncia i Recerca M tua Terrassa, Hospital M tua de Terrassa, Terrassa, Spain ]                                       |
|[Neurovascular Research Laboratory, Vall d Hebron Institute of Research, Hospital Vall d Hebron, Barcelona, Spain C C ]  

期望输出 以下内容:

|[Department of Bone and Joint Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon, Ehime, Japan]                                                |
|[Stroke Pharmacogenomics and Genetics, Fundaci Doc ncia i Recerca M tua Terrassa, Hospital M tua de Terrassa, Terrassa, Spain]                                       |
|[Neurovascular Research Laboratory, Vall d Hebron Institute of Research, Hospital Vall d Hebron, Barcelona, Spain C C]  

所以在

1. 第1行: 'Toon ' -> 'Toon' ,请 'Japan ' -> 'Japan' .
2. 第二行: 'Spain ' -> 'Spain'
3. 第三行: 'Spain C C ' -> 'Spain C C'

注意

尾随空格不仅与列表的最后一个元素一起出现,而且可以与任何元素一起出现。