代码之家  ›  专栏  ›  技术社区  ›  AlwaysLookBright

python中具有巨大矩阵的矩阵运算[重复]

  •  5
  • AlwaysLookBright  · 技术社区  · 12 年前

    有人知道如何在python中处理巨大的矩阵吗?我必须处理形状的邻接矩阵(10^6,10^6),并执行包括加法、缩放和点积在内的操作。使用numpy数组时,ram出现了问题。

    1 回复  |  直到 12 年前
        1
  •  6
  •   spencerlyon2    11 年前

    这样的怎么样。。。

    import numpy as np
    
    # Create large arrays x and y.
    # Note they are 1e4 not 1e6 b/c of memory issues creating random numpy matrices (CookieOfFortune) 
    # However, the same principles apply to larger arrays
    x = np.random.randn(10000, 10000)
    y = np.random.randn(10000, 10000)
    
    # Create memory maps for x and y arrays
    xmap = np.memmap('xfile.dat', dtype='float32', mode='w+', shape=x.shape)
    ymap = np.memmap('yfile.dat', dtype='float32', mode='w+', shape=y.shape)
    
    # Fill memory maps with data
    xmap[:] = x[:]
    ymap[:] = y[:]
    
    # Create memory map for out of core dot product result
    prodmap = np.memmap('prodfile.dat', dtype='float32', mode='w+', shape=x.shape)
    
    # Due out of core dot product and write data
    prodmap[:] = np.memmap.dot(xmap, ymap)
    
    # Create memory map for out of core addition result
    addmap = np.memmap('addfile.dat', dtype='float32', mode='w+', shape=x.shape)
    
    # Due out of core addition and write data
    addmap[:] = xmap + ymap
    
    # Create memory map for out of core scaling result
    scalemap = np.memmap('scalefile.dat', dtype='float32', mode='w+', shape=x.shape)
    
    # Define scaling constant
    scale = 1.3
    
    # Do out of core  scaling and write data
    scalemap[:] = scale * xmap
    

    此代码将创建包含二进制格式数组的文件xfile.dat、yfile.dat等。要稍后访问它们,只需执行以下操作 np.memmap(filename) 。其他论据 np.memmap 是可选的,但建议使用(参数如dtype、shape等)。