Java中的稀疏矩阵/数组

对于频繁读取的数据,使用哈希映射构建的稀疏阵列效率非常低。最高效的实现使用Trie,允许访问段分布的单个向量。

Trie可以通过只执行只读两个数组索引来计算表中是否存在元素,以获取元素存储的有效位置,或者知道其是否不在基础存储中。

它还可以在备份存储中为稀疏数组的默认值提供默认位置,这样您就不需要对返回的索引进行任何测试,因为Trie保证所有可能的源索引将至少映射到备份存储中的默认位置(经常存储零、空字符串或空对象)。

此外,Java Hashmaps只能对对象进行索引,并且为每个哈希源索引创建一个整数对象(每次读取都需要创建该对象,而不仅仅是写入)在内存操作方面代价高昂,因为它会给垃圾收集器带来压力。

我真的希望JRE包含一个IntegerTrieMap<对象>作为慢速HashMap的默认实现<整数,对象>或LongTrieMap<对象>作为更慢的HashMap的默认实现<长,对象>。。。但事实并非如此。

你可能想知道什么是Trie?

它只是一个小整数数组(范围小于矩阵的整个坐标范围),允许将坐标映射到向量中的整数位置。

例如,假设您想要一个仅包含几个非零值的1024*1024矩阵。与其将该矩阵存储在包含1024*1024个元素(超过100万个)的数组中,不如将其拆分为大小为16*16的子范围,并且只需要64*64这样的子范围。

在这种情况下,Trie索引将只包含64*64个整数(4096),并且将至少有16*16个数据元素(包含默认的零或稀疏矩阵中最常见的子范围)。

matrix[i][j] ,您将使用如下语法:

trie.values[trie.subrangePositions[(i & ~15) + (j >> 4)] +
            ((i & 15) << 4) + (j & 15)]

下面是一个内置于注释类中的示例(我希望它编译正常,因为它被简化了;如果有错误需要更正,请通知我):

/**
 * Implement a sparse matrix. Currently limited to a static size
 * (<code>SIZE_I</code>, <code>SIZE_I</code>).
 */
public class DoubleTrie {

    /* Matrix logical options */        
    public static final int SIZE_I = 1024;
    public static final int SIZE_J = 1024;
    public static final double DEFAULT_VALUE = 0.0;

    /* Internal splitting options */
    private static final int SUBRANGEBITS_I = 4;
    private static final int SUBRANGEBITS_J = 4;

    /* Internal derived splitting constants */
    private static final int SUBRANGE_I =
        1 << SUBRANGEBITS_I;
    private static final int SUBRANGE_J =
        1 << SUBRANGEBITS_J;
    private static final int SUBRANGEMASK_I =
        SUBRANGE_I - 1;
    private static final int SUBRANGEMASK_J =
        SUBRANGE_J - 1;
    private static final int SUBRANGE_POSITIONS =
        SUBRANGE_I * SUBRANGE_J;

    /* Internal derived default values for constructors */
    private static final int SUBRANGES_I =
        (SIZE_I + SUBRANGE_I - 1) / SUBRANGE_I;
    private static final int SUBRANGES_J =
        (SIZE_J + SUBRANGE_J - 1) / SUBRANGE_J;
    private static final int SUBRANGES =
        SUBRANGES_I * SUBRANGES_J;
    private static final int DEFAULT_POSITIONS[] =
        new int[SUBRANGES](0);
    private static final double DEFAULT_VALUES[] =
        new double[SUBRANGE_POSITIONS](DEFAULT_VALUE);

    /* Internal fast computations of the splitting subrange and offset. */
    private static final int subrangeOf(
            final int i, final int j) {
        return (i >> SUBRANGEBITS_I) * SUBRANGE_J +
               (j >> SUBRANGEBITS_J);
    }
    private static final int positionOffsetOf(
            final int i, final int j) {
        return (i & SUBRANGEMASK_I) * MAX_J +
               (j & SUBRANGEMASK_J);
    }

    /**
     * Utility missing in java.lang.System for arrays of comparable
     * component types, including all native types like double here.
     */
    public static final int arraycompare(
            final double[] values1, final int position1,
            final double[] values2, final int position2,
            final int length) {
        if (position1 >= 0 && position2 >= 0 && length >= 0) {
            while (length-- > 0) {
                double value1, value2;
                if ((value1 = values1[position1 + length]) !=
                    (value2 = values2[position2 + length])) {
                    /* Note: NaN values are different from everything including
                     * all Nan values; they are are also neigher lower than nor
                     * greater than everything including NaN. Note that the two
                     * infinite values, as well as denormal values, are exactly
                     * ordered and comparable with <, <=, ==, >=, >=, !=. Note
                     * that in comments below, infinite is considered "defined".
                     */
                    if (value1 < value2)
                        return -1;        /* defined < defined. */
                    if (value1 > value2)
                        return 1;         /* defined > defined. */
                    if (value1 == value2)
                        return 0;         /* defined == defined. */
                    /* One or both are NaN. */
                    if (value1 == value1) /* Is not a NaN? */
                        return -1;        /* defined < NaN. */
                    if (value2 == value2) /* Is not a NaN? */
                        return 1;         /* NaN > defined. */
                    /* Otherwise, both are NaN: check their precise bits in
                     * range 0x7FF0000000000001L..0x7FFFFFFFFFFFFFFFL
                     * including the canonical 0x7FF8000000000000L, or in
                     * range 0xFFF0000000000001L..0xFFFFFFFFFFFFFFFFL.
                     * Needed for sort stability only (NaNs are otherwise
                     * unordered).
                     */
                    long raw1, raw2;
                    if ((raw1 = Double.doubleToRawLongBits(value1)) !=
                        (raw2 = Double.doubleToRawLongBits(value2)))
                        return raw1 < raw2 ? -1 : 1;
                    /* Otherwise the NaN are strictly equal, continue. */
                }
            }
            return 0;
        }
        throw new ArrayIndexOutOfBoundsException(
                "The positions and length can't be negative");
    }

    /**
     * Utility shortcut for comparing ranges in the same array.
     */
    public static final int arraycompare(
            final double[] values,
            final int position1, final int position2,
            final int length) {
        return arraycompare(values, position1, values, position2, length);
    }

    /**
     * Utility missing in java.lang.System for arrays of equalizable
     * component types, including all native types like double here.
     */ 
    public static final boolean arrayequals(
            final double[] values1, final int position1,
            final double[] values2, final int position2,
            final int length) {
        return arraycompare(values1, position1, values2, position2, length) ==
            0;
    }

    /**
     * Utility shortcut for identifying ranges in the same array.
     */
    public static final boolean arrayequals(
            final double[] values,
            final int position1, final int position2,
            final int length) {
        return arrayequals(values, position1, values, position2, length);
    }

    /**
     * Utility shortcut for copying ranges in the same array.
     */
    public static final void arraycopy(
            final double[] values,
            final int srcPosition, final int dstPosition,
            final int length) {
        arraycopy(values, srcPosition, values, dstPosition, length);
    }

    /**
     * Utility shortcut for resizing an array, preserving values at start.
     */
    public static final double[] arraysetlength(
            double[] values,
            final int newLength) {
        final int oldLength =
            values.length < newLength ? values.length : newLength;
        System.arraycopy(values, 0, values = new double[newLength], 0,
            oldLength);
        return values;
    }

    /* Internal instance members. */
    private double values[];
    private int subrangePositions[];
    private bool isSharedValues;
    private bool isSharedSubrangePositions;

    /* Internal method. */
    private final reset(
            final double[] values,
            final int[] subrangePositions) {
        this.isSharedValues =
            (this.values = values) == DEFAULT_VALUES;
        this.isSharedsubrangePositions =
            (this.subrangePositions = subrangePositions) ==
                DEFAULT_POSITIONS;
    }

    /**
     * Reset the matrix to fill it with the same initial value.
     *
     * @param initialValue  The value to set in all cell positions.
     */
    public reset(final double initialValue = DEFAULT_VALUE) {
        reset(
            (initialValue == DEFAULT_VALUE) ? DEFAULT_VALUES :
                new double[SUBRANGE_POSITIONS](initialValue),
            DEFAULT_POSITIONS);
    }

    /**
     * Default constructor, using single default value.
     *
     * @param initialValue  Alternate default value to initialize all
     *                      positions in the matrix.
     */
    public DoubleTrie(final double initialValue = DEFAULT_VALUE) {
        this.reset(initialValue);
    }

    /**
     * This is a useful preinitialized instance containing the
     * DEFAULT_VALUE in all cells.
     */
    public static DoubleTrie DEFAULT_INSTANCE = new DoubleTrie();

    /**
     * Copy constructor. Note that the source trie may be immutable
     * or not; but this constructor will create a new mutable trie
     * even if the new trie initially shares some storage with its
     * source when that source also uses shared storage.
     */
    public DoubleTrie(final DoubleTrie source) {
        this.values = (this.isSharedValues =
            source.isSharedValues) ?
            source.values :
            source.values.clone();
        this.subrangePositions = (this.isSharedSubrangePositions =
            source.isSharedSubrangePositions) ?
            source.subrangePositions :
            source.subrangePositions.clone());
    }

    /**
     * Fast indexed getter.
     *
     * @param i  Row of position to set in the matrix.
     * @param j  Column of position to set in the matrix.
     * @return   The value stored in matrix at that position.
     */
    public double getAt(final int i, final int j) {
        return values[subrangePositions[subrangeOf(i, j)] +
                      positionOffsetOf(i, j)];
    }

    /**
     * Fast indexed setter.
     *
     * @param i      Row of position to set in the sparsed matrix.
     * @param j      Column of position to set in the sparsed matrix.
     * @param value  The value to set at this position.
     * @return       The passed value.
     * Note: this does not compact the sparsed matric after setting.
     * @see compact(void)
     */
    public double setAt(final int i, final int i, final double value) {
       final int subrange       = subrangeOf(i, j);
       final int positionOffset = positionOffsetOf(i, j);
       // Fast check to see if the assignment will change something.
       int subrangePosition, valuePosition;
       if (Double.compare(
               values[valuePosition =
                   (subrangePosition = subrangePositions[subrange]) +
                   positionOffset],
               value) != 0) {
               /* So we'll need to perform an effective assignment in values.
                * Check if the current subrange to assign is shared of not.
                * Note that we also include the DEFAULT_VALUES which may be
                * shared by several other (not tested) trie instances,
                * including those instanciated by the copy contructor. */
               if (isSharedValues) {
                   values = values.clone();
                   isSharedValues = false;
               }
               /* Scan all other subranges to check if the position in values
                * to assign is shared by another subrange. */
               for (int otherSubrange = subrangePositions.length;
                       --otherSubrange >= 0; ) {
                   if (otherSubrange != subrange)
                       continue; /* Ignore the target subrange. */
                   /* Note: the following test of range is safe with future
                    * interleaving of common subranges (TODO in compact()),
                    * even though, for now, subranges are sharing positions
                    * only between their common start and end position, so we
                    * could as well only perform the simpler test <code>
                    * (otherSubrangePosition == subrangePosition)</code>,
                    * instead of testing the two bounds of the positions
                    * interval of the other subrange. */
                   int otherSubrangePosition;
                   if ((otherSubrangePosition =
                           subrangePositions[otherSubrange]) >=
                           valuePosition &&
                           otherSubrangePosition + SUBRANGE_POSITIONS <
                           valuePosition) {
                       /* The target position is shared by some other
                        * subrange, we need to make it unique by cloning the
                        * subrange to a larger values vector, copying all the
                        * current subrange values at end of the new vector,
                        * before assigning the new value. This will require
                        * changing the position of the current subrange, but
                        * before doing that, we first need to check if the
                        * subrangePositions array itself is also shared
                        * between instances (including the DEFAULT_POSITIONS
                        * that should be preserved, and possible arrays
                        * shared by an external factory contructor whose
                        * source trie was declared immutable in a derived
                        * class). */
                       if (isSharedSubrangePositions) {
                           subrangePositions = subrangePositions.clone();
                           isSharedSubrangePositions = false;
                       }
                       /* TODO: no attempt is made to allocate less than a
                        * fully independant subrange, using possible
                        * interleaving: this would require scanning all
                        * other existing values to find a match for the
                        * modified subrange of values; but this could
                        * potentially leave positions (in the current subrange
                        * of values) unreferenced by any subrange, after the
                        * change of position for the current subrange. This
                        * scanning could be prohibitively long for each
                        * assignement, and for now it's assumed that compact()
                        * will be used later, after those assignements. */
                       values = setlengh(
                           values,
                           (subrangePositions[subrange] =
                            subrangePositions = values.length) +
                           SUBRANGE_POSITIONS);
                       valuePosition = subrangePositions + positionOffset;
                       break;
                   }
               }
               /* Now perform the effective assignment of the value. */
               values[valuePosition] = value;
           }
       }
       return value;
    }

    /**
     * Compact the storage of common subranges.
     * TODO: This is a simple implementation without interleaving, which
     * would offer a better data compression. However, interleaving with its
     * O(N²) complexity where N is the total length of values, should
     * be attempted only after this basic compression whose complexity is
     * O(n²) with n being SUBRANGE_POSITIIONS times smaller than N.
     */
    public void compact() {
        final int oldValuesLength = values.length;
        int newValuesLength = 0;
        for (int oldPosition = 0;
                 oldPosition < oldValuesLength;
                 oldPosition += SUBRANGE_POSITIONS) {
            int oldPosition = positions[subrange];
            bool commonSubrange = false;
            /* Scan values for possible common subranges. */
            for (int newPosition = newValuesLength;
                    (newPosition -= SUBRANGE_POSITIONS) >= 0; )
                if (arrayequals(values, newPosition, oldPosition,
                        SUBRANGE_POSITIONS)) {
                    commonSubrange = true;
                    /* Update the subrangePositions|] with all matching
                     * positions from oldPosition to newPosition. There may
                     * be several index to change, if the trie has already
                     * been compacted() before, and later reassigned. */
                    for (subrange = subrangePositions.length;
                         --subrange >= 0; )
                        if (subrangePositions[subrange] == oldPosition)
                            subrangePositions[subrange] = newPosition;
                    break;
                }
            if (!commonSubrange) {
                /* Move down the non-common values, if some previous
                 * subranges have been compressed when they were common.
                 */
                if (!commonSubrange && oldPosition != newValuesLength) {
                    arraycopy(values, oldPosition, newValuesLength,
                        SUBRANGE_POSITIONS);
                    /* Advance compressed values to preserve these new ones. */
                    newValuesLength += SUBRANGE_POSITIONS;
                }
            }
        }
        /* Check the number of compressed values. */
        if (newValuesLength < oldValuesLength) {
            values = values.arraysetlength(newValuesLength);
            isSharedValues = false;
        }
    }

}

注意:此代码不完整,因为它处理单个矩阵大小,并且其压缩程序仅限于检测公共子范围,而不交错它们。

int indexOf(int, int) 和 int offsetOf(int, int) 内部方法),独立于两个坐标,并达到矩阵的最大宽度或高度 compact() 方法应能确定最佳配合尺寸。

SUBRANGE_POSITIONS ,并制作静态方法 int subrangeOf(int i, int j) 和 int positionOffsetOf(int i, int j) 变为非静态;以及初始化数组 DEFAULT_POSITIONS 和 DEFAULT_VALUES

如果您想支持交错,基本上您将首先将现有值分成两个大小大致相同的值(两个值都是最小子范围大小的倍数,第一个子集可能比第二个子集多一个子范围),然后在所有连续位置扫描较大的值以找到匹配的交错;然后您将尝试匹配这些值。然后通过将子集分成两半(也是最小子范围大小的倍数)递归循环,然后再次扫描以匹配这些子集(这会将子集的数量乘以2:您必须想一想,与现有值的大小相比,子范围位置索引的两倍大小是否值得,以查看它是否提供了有效的压缩(如果没有,您就到此为止:您已经直接从交错压缩过程中找到了最佳子范围大小).在这种情况下,在压实过程中,子范围大小将是可变的。

但这段代码显示了如何分配非零值和重新分配 data 在使用 setAt(int i, int j, double value) 方法)当数据中存在可统一的重复子范围,并在数据中的相同位置重新编制索引时,存储此数据 subrangePositions 大堆

无论如何,trie的所有原则都在那里实施:

1. 使用单个向量而不是双索引数组(每个数组单独分配)表示矩阵总是更快(内存更紧凑,意味着更好的局部性)。这一改进在以下方面是显而易见的: double getAt(int, int) 方法

3. 通过检测公共子范围,可以将初始大矩阵自动转换为更紧凑的矩阵。然后,一个典型的实现将包含一个方法,例如 紧凑的

4. 公共子范围在数据中使用公共存储,因此此共享数据必须是只读的。如果必须在不更改矩阵其余部分的情况下更改单个值,则必须首先确保该值在矩阵中仅被引用一次 子范围位置 values 向量,然后将此新子范围的位置存储到 指数

请注意,通用Colt库虽然非常好,但在处理稀疏矩阵时却不太好,因为它使用了哈希(或行压缩)技术,目前还不支持尝试,尽管它是一种非常好的优化,既节省空间,也节省空间 节省时间,特别是对于最频繁的getAt()操作。

即使是此处描述的用于trys的setAt()操作也可以节省大量时间(此处采用的方法是,设置后不进行自动压缩,这仍然可以根据需求和估计的时间来实现,压缩仍将以时间为代价节省大量存储空间):节省的时间与子范围中的单元数成正比,节省的空间与每个子范围中的单元数成反比。如果要使用子范围大小,每个子范围的单元数是2D矩阵中单元总数的平方根(使用3D矩阵时,它将是一个立方根),则最好进行压缩。

我只是希望Colt在将来会被更新,以实现另一个使用Tries的实现(即TrieSparseMatrix,而不仅仅是HashSparseMatrix和RCSparseMatrix)。这些想法都在本文中。

Trove实现(基于int->int映射)也基于类似于Colt的HashedSparseMatrix的散列技术,即它们具有相同的不便。尝试的速度会快得多,只需消耗少量的额外空间(但在延迟的时间内,使用最终的compact()离子操作对结果矩阵/trie进行操作,可以优化此空间,甚至比Trove和Colt更好)。

注意:这个Trie实现绑定到一个特定的本机类型(这里是double)。这是自愿的,因为使用装箱类型的泛型实现具有巨大的空间开销(并且访问时间要慢得多)。在这里,它只使用double的原生一维数组,而不是泛型向量。但它当然也有可能派生出一个通用的实现来尝试。。。不幸的是,Java仍然不允许编写具有本机类型所有优点的真正通用类,除非编写多个实现(针对通用对象类型或每个本机类型),并通过类型工厂为所有这些操作提供服务。该语言应该能够自动实例化本机实现并自动构建工厂(目前,即使在Java 7中,情况也不是这样,在这一点上,.Net对于与本机类型一样快的真正泛型类型仍然保持其优势)。

下面是测试Java矩阵库的框架,也提供了一个很好的列表! https://lessthanoptimal.github.io/Java-Matrix-Benchmark/

测试库:

* Colt
* Commons Math
* Efficient Java Matrix Library (EJML)
* Jama
* jblas
* JScience (Older benchmarks only)
* Matrix Toolkit Java (MTJ)
* OjAlgo
* Parallel Colt
* Universal Java Matrix Package (UJMP) 

这似乎很简单。

可以使用row*maxcolums+column作为索引的二叉树数据。

可能不是最快的运行时解决方案,但我能想出的最快的解决方案似乎是可行的。创建索引类并将其用作SortedMap的键,如:

    SortedMap<Index, Object> entries = new TreeMap<Index, Object>();
    entries.put(new Index(1, 4), "1-4");
    entries.put(new Index(5555555555l, 767777777777l), "5555555555l-767777777777l");
    System.out.println(entries.size());
    System.out.println(entries.get(new Index(1, 4)));
    System.out.println(entries.get(new Index(5555555555l, 767777777777l)));

public static class Index implements Comparable<Index>
{
    private long x;
    private long y;

    public Index(long x, long y)
    {
        super();
        this.x = x;
        this.y = y;
    }

    public int compareTo(Index index)
    {
        long ix = index.x;
        if (ix == x)
        {
            long iy = index.y;
            if (iy == y)
            {
                return 0;
            }
            else if (iy < y)
            {
                return -1;
            }
            else
            {
                return 1;
            }
        }
        else if (ix < x)
        {
            return -1;
        }
        else
        {
            return 1;
        }
    }

    public int hashCode()
    {
        final int PRIME = 31;
        int result = 1;
        result = PRIME * result + (int) (x ^ (x >>> 32));
        result = PRIME * result + (int) (y ^ (y >>> 32));
        return result;
    }

    public boolean equals(Object obj)
    {
        if (this == obj)
            return true;
        if (obj == null)
            return false;
        if (getClass() != obj.getClass())
            return false;
        final Index other = (Index) obj;
        if (x != other.x)
            return false;
        if (y != other.y)
            return false;
        return true;
    }

    public long getX()
    {
        return x;
    }

    public long getY()
    {
        return y;
    }
}

你可以看看 la4j (Java线性代数)库。它支持 CRS (Compressed Row Storage) CCS (Compressed Column Storage) 稀疏矩阵的内部表示。因此,这些是稀疏数据最有效、最快速的内部结构。

la4j :

Matrix a = new CRSMatrix(new double[][]{ // 'a' - CRS sparse matrix
   { 1.0, 0.0, 3.0 },
   { 0.0, 5.0, 0.0 },
   { 7.0, 0.0. 9.0 }
});

Matrix b = a.transpose(); // 'b' - CRS sparse matrix

Matrix c = b.multiply(a, Matrices.CCS_FACTORY); // 'c' = 'b' * 'a'; 
                                                // 'c' - CCS sparse matrix

您可以只使用嵌套映射,尽管如果需要对其执行矩阵演算,这可能不是最佳选择

 Map<Integer, Map<integer, Object>> matrix;

可能不使用对象,而是对实际数据使用一些元组,以便在提取后可以更轻松地使用它,例如:

class Tuple<T extends yourDataObject> {
  public final int x;
  public final int y;
  public final T object;
}

class Matrix {
  private final Map<Integer, Map<interger, Tupple>> data = new...;

 void add(int x, int y, Object object) {
     data.get(x).put(new Tupple(x,y,object);
 }
}


//etc

为简洁起见,省略了空检查等