代码之家  ›  专栏  ›  技术社区  ›  Kiquenet user385990

使用文件流aspnet发送大文件500MB时内存不足异常

  •  8
  • Kiquenet user385990  · 技术社区  · 15 年前

    我正在使用filestream读取大文件(>500 MB),并获得了out-ofmemoryException。

    我使用ASP.NET,.NET 3.5,Win2003,IIS 6.0

    我想在我的应用程序中使用:

    从Oracle读取数据

    使用文件流和bzip2解压缩文件

    读取未压缩的文件并将其发送到ASP.NET页进行下载。

    当我从磁盘读取文件时,失败了!!!!从记忆中消失…

    . 我的代码是:

    using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read)) 
            { 
              byte[] b2 = ReadFully(fs3, 1024); 
            } 
    
     // http://www.yoda.arachsys.com/csharp/readbinary.html
     public static byte[] ReadFully(Stream stream, int initialLength) 
      { 
        // If we've been passed an unhelpful initial length, just 
        // use 32K. 
        if (initialLength < 1) 
        { 
          initialLength = 32768; 
        } 
    
        byte[] buffer = new byte[initialLength]; 
        int read = 0; 
    
        int chunk; 
        while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0) 
        { 
          read += chunk; 
    
          // If we've reached the end of our buffer, check to see if there's 
          // any more information 
          if (read == buffer.Length) 
          { 
            int nextByte = stream.ReadByte(); 
    
            // End of stream? If so, we're done 
            if (nextByte == -1) 
            { 
              return buffer; 
            } 
    
            // Nope. Resize the buffer, put in the byte we've just 
            // read, and continue 
            byte[] newBuffer = new byte[buffer.Length * 2]; 
            Array.Copy(buffer, newBuffer, buffer.Length); 
            newBuffer[read] = (byte)nextByte; 
            buffer = newBuffer; 
            read++; 
          } 
        } 
        // Buffer is now too big. Shrink it. 
        byte[] ret = new byte[read]; 
        Array.Copy(buffer, ret, read); 
        return ret; 
      } 
    

    现在,我更详细地说明我的问题。

    使用filestream和bzip2解压文件可以,一切正常。

    问题如下:

    读取磁盘(>500 MB)中字节为[]的胖大文件,并将字节发送到响应(ASP.NET)进行下载。

    使用时

    http://www.yoda.arachsys.com/csharp/readbinary.html

    public static byte[] ReadFully
    

    我得到错误:内存不足异常…

    如果缓冲流比流(filestream、memorystream,…)更好??

    使用bufferedstream,我可以读取700MB的大文件吗??(任何使用bufferedstream下载大文件的示例代码源)

    我认为,这是一个问题:而不是“如何将500MB文件读取到内存中?”,但“如何将大文件发送到aspnet响应流?”

    我找到了Cheeso的代码:

    using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))  
    {  
       Response.BufferOutput= false;   // to prevent buffering 
       byte[] buffer = new byte[1024]; 
       int bytesRead = 0; 
       while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)  
       { 
           Response.OutputStream.Write(buffer, 0, bytesRead); 
       } 
    }
    

    这是好代码吗??对高性能有什么改进吗??

    一个学院说我,用

    Response.TransmitFile(filePath);
    

    现在,另一个问题,更好地由Cheeso传输文件或代码??

    很多年前,在msdn杂志上发表了一篇很棒的文章,但我无法访问。 http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/ ,

    更新 :您可以使用 网络档案 在链接中: https://web.archive.org/web/20070627063111/http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/

    有什么建议,评论,示例代码源吗??

    3 回复  |  直到 7 年前
        1
  •  16
  •   Pavel Morshenyuk    15 年前

    我已经创建了下载页面,允许用户在几个月前最多下载4GB(可能更多)。这是我的工作片段:

      private void TransmitFile(string fullPath, string outFileName)
        {
            System.IO.Stream iStream = null;
    
            // Buffer to read 10K bytes in chunk:
            byte[] buffer = new Byte[10000];
    
            // Length of the file:
            int length;
    
            // Total bytes to read:
            long dataToRead;
    
            // Identify the file to download including its path.
            string filepath = fullPath;
    
            // Identify the file name.
            string filename = System.IO.Path.GetFileName(filepath);
    
            try
            {
                // Open the file.
                iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
                            System.IO.FileAccess.Read, System.IO.FileShare.Read);
    
    
                // Total bytes to read:
                dataToRead = iStream.Length;
    
                Response.Clear();
                Response.ContentType = "application/octet-stream";
                Response.AddHeader("Content-Disposition", "attachment; filename=" + outFileName);
                Response.AddHeader("Content-Length", iStream.Length.ToString());
    
                // Read the bytes.
                while (dataToRead > 0)
                {
                    // Verify that the client is connected.
                    if (Response.IsClientConnected)
                    {
                        // Read the data in buffer.
                        length = iStream.Read(buffer, 0, 10000);
    
                        // Write the data to the current output stream.
                        Response.OutputStream.Write(buffer, 0, length);
    
                        // Flush the data to the output.
                        Response.Flush();
    
                        buffer = new Byte[10000];
                        dataToRead = dataToRead - length;
                    }
                    else
                    {
                        //prevent infinite loop if user disconnects
                        dataToRead = -1;
                    }
                }
            }
            catch (Exception ex)
            {
                throw new ApplicationException(ex.Message);
            }
            finally
            {
                if (iStream != null)
                {
                    //Close the file.
                    iStream.Close();
                }
                Response.Close();
            }
        }
    
        2
  •  2
  •   Giorgi    15 年前

    您不需要将整个文件保存在内存中,只需读取它并在循环中写入响应流即可。

        3
  •  0
  •   Ebram    7 年前

    有多个解决方案

    1-使用可回收的MemoryStream而不是MemoryStream解决方案

    您可以在此处阅读有关可回收内存流的更多信息: http://www.philosophicalgeek.com/2015/02/06/announcing-microsoft-io-recycablememorystream/

    https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

    2-使用memoryTributary而不是memoryStream

    您可以在此处阅读更多关于MemoryTributary的信息:

    https://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream?msg=5257615#xx5257615xx

        using System;
        using System.Collections.Generic;
        using System.IO;
        using System.Runtime.InteropServices;
    
       namespace LiquidEngine.Tools
           {
    /// <summary>
    /// MemoryTributary is a re-implementation of MemoryStream that uses a dynamic list of byte arrays as a backing store, instead of a single byte array, the allocation
    /// of which will fail for relatively small streams as it requires contiguous memory.
    /// </summary>
    public class MemoryTributary : Stream       /* http://msdn.microsoft.com/en-us/library/system.io.stream.aspx */
    {
        #region Constructors
    
        public MemoryTributary()
        {
            Position = 0;
        }
    
        public MemoryTributary(byte[] source)
        {
            this.Write(source, 0, source.Length);
            Position = 0;
        }
    
        /* length is ignored because capacity has no meaning unless we implement an artifical limit */
        public MemoryTributary(int length)
        {
            SetLength(length);
            Position = length;
            byte[] d = block;   //access block to prompt the allocation of memory
            Position = 0;
        }
    
        #endregion
    
        #region Status Properties
    
        public override bool CanRead
        {
            get { return true; }
        }
    
        public override bool CanSeek
        {
            get { return true; }
        }
    
        public override bool CanWrite
        {
            get { return true; }
        }
    
        #endregion
    
        #region Public Properties
    
        public override long Length
        {
            get { return length; }
        }
    
        public override long Position { get; set; }
    
        #endregion
    
        #region Members
    
        protected long length = 0;
    
        protected long blockSize = 65536;
    
        protected List<byte[]> blocks = new List<byte[]>();
    
        #endregion
    
        #region Internal Properties
    
        /* Use these properties to gain access to the appropriate block of memory for the current Position */
    
        /// <summary>
        /// The block of memory currently addressed by Position
        /// </summary>
        protected byte[] block
        {
            get
            {
                while (blocks.Count <= blockId)
                    blocks.Add(new byte[blockSize]);
                return blocks[(int)blockId];
            }
        }
        /// <summary>
        /// The id of the block currently addressed by Position
        /// </summary>
        protected long blockId
        {
            get { return Position / blockSize; }
        }
        /// <summary>
        /// The offset of the byte currently addressed by Position, into the block that contains it
        /// </summary>
        protected long blockOffset
        {
            get { return Position % blockSize; }
        }
    
        #endregion
    
        #region Public Stream Methods
    
        public override void Flush()
        {
        }
    
        public override int Read(byte[] buffer, int offset, int count)
        {
            long lcount = (long)count;
    
            if (lcount < 0)
            {
                throw new ArgumentOutOfRangeException("count", lcount, "Number of bytes to copy cannot be negative.");
            }
    
            long remaining = (length - Position);
            if (lcount > remaining)
                lcount = remaining;
    
            if (buffer == null)
            {
                throw new ArgumentNullException("buffer", "Buffer cannot be null.");
            }
            if (offset < 0)
            {
                throw new ArgumentOutOfRangeException("offset",offset,"Destination offset cannot be negative.");
            }
    
            int read = 0;
            long copysize = 0;
            do
            {
                copysize = Math.Min(lcount, (blockSize - blockOffset));
                Buffer.BlockCopy(block, (int)blockOffset, buffer, offset, (int)copysize);
                lcount -= copysize;
                offset += (int)copysize;
    
                read += (int)copysize;
                Position += copysize;
    
            } while (lcount > 0);
    
            return read;
    
        }
    
        public override long Seek(long offset, SeekOrigin origin)
        {
            switch (origin)
            {
                case SeekOrigin.Begin:
                    Position = offset;
                    break;
                case SeekOrigin.Current:
                    Position += offset;
                    break;
                case SeekOrigin.End:
                    Position = Length - offset;
                    break;
            }
            return Position;
        }
    
        public override void SetLength(long value)
        {
            length = value;
        }
    
        public override void Write(byte[] buffer, int offset, int count)
        {
            long initialPosition = Position;
            int copysize;
            try
            {
                do
                {
                    copysize = Math.Min(count, (int)(blockSize - blockOffset));
    
                    EnsureCapacity(Position + copysize);
    
                    Buffer.BlockCopy(buffer, (int)offset, block, (int)blockOffset, copysize);
                    count -= copysize;
                    offset += copysize;
    
                    Position += copysize;
    
                } while (count > 0);
            }
            catch (Exception e)
            {
                Position = initialPosition;
                throw e;
            }
        }
    
        public override int ReadByte()
        {
            if (Position >= length)
                return -1;
    
            byte b = block[blockOffset];
            Position++;
    
            return b;
        }
    
        public override void WriteByte(byte value)
        {
            EnsureCapacity(Position + 1);
            block[blockOffset] = value;
            Position++;
        }
    
        protected void EnsureCapacity(long intended_length)
        {
            if (intended_length > length)
                length = (intended_length);
        }
    
        #endregion
    
        #region IDispose
    
        /* http://msdn.microsoft.com/en-us/library/fs2xkftw.aspx */
        protected override void Dispose(bool disposing)
        {
            /* We do not currently use unmanaged resources */
            base.Dispose(disposing);
        }
    
        #endregion
    
        #region Public Additional Helper Methods
    
        /// <summary>
        /// Returns the entire content of the stream as a byte array. This is not safe because the call to new byte[] may 
        /// fail if the stream is large enough. Where possible use methods which operate on streams directly instead.
        /// </summary>
        /// <returns>A byte[] containing the current data in the stream</returns>
        public byte[] ToArray()
        {
            long firstposition = Position;
            Position = 0;
            byte[] destination = new byte[Length];
            Read(destination, 0, (int)Length);
            Position = firstposition;
            return destination;
        }
    
        /// <summary>
        /// Reads length bytes from source into the this instance at the current position.
        /// </summary>
        /// <param name="source">The stream containing the data to copy</param>
        /// <param name="length">The number of bytes to copy</param>
        public void ReadFrom(Stream source, long length)
        {
            byte[] buffer = new byte[4096];
            int read;
            do
            {
                read = source.Read(buffer, 0, (int)Math.Min(4096, length));
                length -= read;
                this.Write(buffer, 0, read);
    
            } while (length > 0);
        }
    
        /// <summary>
        /// Writes the entire stream into destination, regardless of Position, which remains unchanged.
        /// </summary>
        /// <param name="destination">The stream to write the content of this stream to</param>
        public void WriteTo(Stream destination)
        {
            long initialpos = Position;
            Position = 0;
            this.CopyTo(destination);
            Position = initialpos;
        }
    
        #endregion
    }
    

    }

    推荐文章