代码之家  ›  专栏  ›  技术社区  ›  Muhammad Affan

如何在短时间内通过django模型在mongodb中批量插入550万条记录?

  •  0
  • Muhammad Affan  · 技术社区  · 4 年前

    它带我四处转转 6.5至7小时 在从的数据帧插入批量记录时 550万记录 对我的 mongodb数据库 通过 django模型 具有 12-GB内存 .下面是引发查询的代码。我想让它更快地四处走动 30分钟到1小时 时间,可能吗?

    # Split DF
    
    listDf = []
    listDf = split_dataframe(completeDF02)
    
    ## This completeDF02 is the full dataframe having 5.5M records, So I split it with having 0.1M record in one batch in listDf list
    
    for item in listDf:
        df_records = item.to_dict('records')
        model_instances = [MyModel(
            Field1=record['Field1'],
            Field2=record['Field2'],
            Field3=record['Field3'],
            Field4=record['Field4'],
            Field5=record['Field5'],
            Field6=record['Field6'],
            Field7=record['Field7'],
            Field8=record['Field8'],
            Field9=record['Field9'],
            Field10=record['Field10'],
            Field11=record['Field11'],
            Field12=record['Field12'],
            Field13=record['Field13'],
            Field14=record['Field14'],
            Field15=record['Field15'],
            Field16=record['Field16'],
            Field17=record['Field17'],
            Field18=record['Field18'],
            Field19=record['Field19'],
            Field20=record['Field20'],
            Field21=record['Field21'],
            Field22=record['Field22'],
            Field23=record['Field23'],
            Field24=record['Field24'],
            Field25=record['Field25']
        ) for record in df_records]
        
        MyModel.objects.bulk_create(model_instances)
    
    

    有没有其他方法可以进行批量插入,以提高速度和效率?如果是,请告诉我。

    0 回复  |  直到 4 年前