代码之家 › 专栏 › 技术社区 › user366312

保存时数据帧未垂直堆叠

csv pandas python

user366312 · 技术社区 · 2 年前

数据1.dat

   1 ASN C  7.042   9.118  0.000 1 1 1 1  1  0
   2 LEU H  5.781   5.488  7.470 0 0 0 0  1  0
   3 THR H  5.399   5.166  6.452 0 0 0 0  0  0
   4 GLU H  5.373   4.852  6.069 0 0 0 0  1  0
   5 LEU H  5.423   5.164  6.197 0 0 0 0  2  0
   6 LYS H  5.247   4.943  6.434 0 0 0 0  1  0
   7 ASN C  5.485   8.103  8.264 0 0 0 0  1  0
   8 THR C  6.675   9.152  9.047 0 0 0 0  1  0
   9 PRO C  6.372   8.536 11.954 0 0 0 0  0  0
  10 VAL H  5.669   5.433  6.703 0 0 0 0  0  0
  11 SER H  5.304   4.924  6.407 0 0 0 0  0  0
  12 GLU H  5.461   5.007  6.088 0 0 0 0  1  0
  13 LEU H  5.265   5.057  6.410 0 0 0 0  3  0
  14 ILE H  5.379   5.026  6.206 0 0 0 0  1  0
  15 THR H  5.525   5.154  6.000 0 0 0 0  1  0
  16 LEU H  5.403   5.173  6.102 0 0 0 0  1  0
  17 GLY H  5.588   5.279  6.195 0 0 0 0  1  0
  18 GLU H  5.381   5.238  6.675 0 0 0 0  1  0
  19 ASN H  5.298   5.287  6.668 0 0 0 0  1  0
  20 MSE H  5.704   7.411  4.926 0 0 0 0  1  0

data2.dat

  21 GLY C  5.978   9.254  9.454 0 0 0 0  1  0
  22 LEU C  6.778  10.534 12.640 0 0 1 2  2  0
  23 GLU C  7.187   7.217 10.728 0 0 0 0  2  0
  24 ASN C  5.392   8.296 10.702 0 0 0 0  0  0
  25 LEU C  5.657   6.064  9.609 0 0 0 1  3  0
  26 ALA C  5.446   5.528  7.503 0 0 0 0  2  0
  27 ARG C  5.656   8.071  8.419 0 0 0 0  0  0
  28 MSE C  6.890   9.157  8.728 0 0 0 0  1  0
  29 ARG C  6.330   7.993 11.562 0 0 0 0  0  0
  30 LYS H  5.428   5.207  5.897 0 0 0 0  1  0
  31 GLN H  5.402   5.046  6.349 0 0 0 0  1  0
  32 ASP H  5.426   5.093  6.226 0 0 0 1  1  0
  33 ILE H  5.361   5.004  6.194 0 0 0 0  6  0
  34 ILE H  5.443   5.150  6.190 0 0 0 0  5  0
  35 PHE H  5.403   5.181  6.293 0 0 0 0  1  0
  36 ALA H  5.533   5.357  6.193 0 0 0 0  3  0
  37 ILE H  5.634   5.167  6.025 0 0 0 1  5  0
  38 LEU H  5.402   5.121  6.104 0 0 0 0  3  0
  39 LYS H  5.470   5.092  6.101 0 0 0 0  1  0
  40 GLN H  5.491   5.210  6.054 0 0 0 0  2  0

import os
import pandas as pd
from src.utils.get_root_dir import get_root_directory

def save_dataframe_to_ascii(df, filepath):
    df.to_csv(filepath, sep=',', index=False)

def getDataFrame(dataDirectoryPathString: str) -> pd.DataFrame:
    dataframes = []
    for filename in os.listdir(dataDirectoryPathString):
        if filename.endswith('.dat'):
            filepath = os.path.join(dataDirectoryPathString, filename)
            df = pd.read_csv(filepath, sep='\t')
            dataframes.append(df)
    concatenated_df = pd.concat(dataframes, ignore_index=True)
    return concatenated_df


if __name__ == "__main__":
    dataFrame = getDataFrame(get_root_directory() + "/data/")
    save_dataframe_to_ascii(dataFrame, get_root_directory() + "/save/save.txt")

输出:

   1 ASN C  7.042   9.118  0.000 1 1 1 1  1  0,  21 GLY C  5.978   9.254  9.454 0 0 0 0  1  0
   2 LEU H  5.781   5.488  7.470 0 0 0 0  1  0,
   3 THR H  5.399   5.166  6.452 0 0 0 0  0  0,
   4 GLU H  5.373   4.852  6.069 0 0 0 0  1  0,
   5 LEU H  5.423   5.164  6.197 0 0 0 0  2  0,
   6 LYS H  5.247   4.943  6.434 0 0 0 0  1  0,
   7 ASN C  5.485   8.103  8.264 0 0 0 0  1  0,
   8 THR C  6.675   9.152  9.047 0 0 0 0  1  0,
   9 PRO C  6.372   8.536 11.954 0 0 0 0  0  0,
  10 VAL H  5.669   5.433  6.703 0 0 0 0  0  0,
  11 SER H  5.304   4.924  6.407 0 0 0 0  0  0,
  12 GLU H  5.461   5.007  6.088 0 0 0 0  1  0,
  13 LEU H  5.265   5.057  6.410 0 0 0 0  3  0,
  14 ILE H  5.379   5.026  6.206 0 0 0 0  1  0,
  15 THR H  5.525   5.154  6.000 0 0 0 0  1  0,
  16 LEU H  5.403   5.173  6.102 0 0 0 0  1  0,
  17 GLY H  5.588   5.279  6.195 0 0 0 0  1  0,
  18 GLU H  5.381   5.238  6.675 0 0 0 0  1  0,
  19 ASN H  5.298   5.287  6.668 0 0 0 0  1  0,
  20 MSE H  5.704   7.411  4.926 0 0 0 0  1  0,
,  22 LEU C  6.778  10.534 12.640 0 0 1 2  2  0
,  23 GLU C  7.187   7.217 10.728 0 0 0 0  2  0
,  24 ASN C  5.392   8.296 10.702 0 0 0 0  0  0
,  25 LEU C  5.657   6.064  9.609 0 0 0 1  3  0
,  26 ALA C  5.446   5.528  7.503 0 0 0 0  2  0
,  27 ARG C  5.656   8.071  8.419 0 0 0 0  0  0
,  28 MSE C  6.890   9.157  8.728 0 0 0 0  1  0
,  29 ARG C  6.330   7.993 11.562 0 0 0 0  0  0
,  30 LYS H  5.428   5.207  5.897 0 0 0 0  1  0
,  31 GLN H  5.402   5.046  6.349 0 0 0 0  1  0
,  32 ASP H  5.426   5.093  6.226 0 0 0 1  1  0
,  33 ILE H  5.361   5.004  6.194 0 0 0 0  6  0
,  34 ILE H  5.443   5.150  6.190 0 0 0 0  5  0
,  35 PHE H  5.403   5.181  6.293 0 0 0 0  1  0
,  36 ALA H  5.533   5.357  6.193 0 0 0 0  3  0
,  37 ILE H  5.634   5.167  6.025 0 0 0 1  5  0
,  38 LEU H  5.402   5.121  6.104 0 0 0 0  3  0
,  39 LYS H  5.470   5.092  6.101 0 0 0 0  1  0
,  40 GLN H  5.491   5.210  6.054 0 0 0 0  2  0

各行本应垂直堆放。

为什么输出中断?

我该怎么修?

1 回复 | 直到 2 年前

Zach Young mo1010 2 年前

如果你不需要使用Pandas,而且我认为你除了级联之外(到目前为止)还没有使用它,我建议你使用Python的csv模块来堆叠类似csv的文件。它将比Pandas更快,并且使用几乎为零的内存:

import csv

TAB = "\t"

# Get files first, using maybe just use `glob.glob(get_root_directory()+'/data/*.dat')`
dat_files = ["input1.dat", "input2.dat"]

with open("output.dat", "w", newline="", encoding="utf-8") as f_out:
    writer = csv.writer(f_out, delimiter=TAB)

    for dat_file in dat_files:
        with open(dat_file, newline="", encoding="utf-8") as f_in:
            reader = csv.reader(f_in, delimiter=TAB)

            writer.writerows(reader)

这个 reader 变量是用于创建它的文件中所有行的迭代器。使命感 writer.writerows(reader) 将导致阅读器中的每一行都被迭代,并直接交给编写器进行编写。这种方法也几乎不使用内存,因为行只是从输入直接传递到输出。

我还展示了先获取文件以避免额外的缩进/复杂性。

使用您共享的示例文件(并将任意数量的空格转换为TAB),我的output.dat看起来如下:

1   ASN C   7.042   9.118   0.000   1   1   1   1   1   0
2   LEU H   5.781   5.488   7.470   0   0   0   0   1   0
3   THR H   5.399   5.166   6.452   0   0   0   0   0   0
...
38  LEU H   5.402   5.121   6.104   0   0   0   0   3   0
39  LYS H   5.470   5.092   6.101   0   0   0   0   1   0
40  GLN H   5.491   5.210   6.054   0   0   0   0   2   0