代码之家 › 专栏 › 技术社区 › pookie

使用Flask_socketio将视频流式传输到Flask应用程序

flask-socketio flask opencv python

pookie · 技术社区 · 4 年前

我有一个 video 元素whos-src被设置为我的网络摄像头的提要。这是使用OpenCV(opencvjs)完成的。

使用opencv,我读取一个帧并将其渲染为 canvas .

然后,我从画布上读取数据,并使用 socketio 一、 emit 到我的服务器应用程序,这是一个Flask应用程序,作为base64。。。

服务器端,我捕获传入的视频帧,从base64解码,做一些处理,重新编码为base64,并将其发送回客户端。

这整个base64->图像->base64是无稽之谈,效率极低。然而,这是我的第一个Flask应用程序,我已经很多年没有真正接触过web开发了,所以我在这里有点困惑。

我如何将视频从浏览器发送到服务器,进行处理,然后在不进行所有base64转换的情况下将其发送回客户端?我正在寻找一种更有效的方法来做到这一点。

这是我的代码:

index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <script type="text/javascript" src="../static/jquery-3.5.1.min.js" ></script>
    <script type="text/javascript" src="../static/index.js"></script>
    <script type="text/javascript" src="../static/socket.io.min.js"></script>
</head>
<body>
<div class="container">
    <div class="row">
        <div class="col-lg-8  offset-lg-2">
            <h3 class="mt-5">Live Streaming</h3>
            <video style="display: none;" id="videoInput"></video>
        </div>
    </div>
    <div class="row">
        <div class="col-lg-8  offset-lg-2">
            <h3 class="mt-5">Edited video</h3>
            <canvas class="center-block" id="canvasOutput" width=320 height=240></canvas>
        </div>
    </div>
    <div class="row">
        <div class="col-lg-8  offset-lg-2">
            <h3 class="mt-5">SOCKETIO IMAGES</h3>
            <img id="imagebox" src="" width="320" height="240">
        </div>
    </div>
</div>
<script src="../static/opencv.js" onload="onOpenCvReady();" type="text/javascript"></script>
</body>
</html>

index.js

let video = null; 

let width = 320;
let height = 0;

let src = null;
let dst = null;

let streaming = false;
let stream = null;
let vc = null;

let socket = null;

function processVideo() {

    vc.read(src);
    cv.imshow("canvasOutput", src);

    var type = "image/png"
    var data = document.getElementById("canvasOutput").toDataURL(type);
    data = data.replace('data:' + type + ';base64,', ''); 

    socket.emit('image', data);

    requestAnimationFrame(processVideo);
}

function stopVideoProcessing() {
    if (src != null && !src.isDeleted()) src.delete();
}

function stopCamera() {
    if (!streaming) return;
    stopVideoProcessing();
    document.getElementById("canvasOutput").getContext("2d").clearRect(0, 0, width, height);
    video.pause();
    video.srcObject=null;
    stream.getVideoTracks()[0].stop();
    streaming = false;
}

function startVideoProcessing() {
    if (!streaming) { console.warn("Please startup your webcam"); return; }
    stopVideoProcessing();
    src = new cv.Mat(height, width, cv.CV_8UC4);
    requestAnimationFrame(processVideo);
}

function startCamera() {
    if (streaming) return;
    navigator.mediaDevices.getUserMedia({video: true, audio: false})
        .then(function(s) {
            stream = s;
            video.srcObject = s;
            video.play();
        })
        .catch(function(err) {
            console.log("An error occured! " + err);
        });

    video.addEventListener("canplay", function(ev){
        if (!streaming) {
            height = video.videoHeight / (video.videoWidth/width);
            video.setAttribute("width", width);
            video.setAttribute("height", height);
            streaming = true;
            vc = new cv.VideoCapture(video);
        }
        startVideoProcessing();
    }, false);
}

function onOpenCvReady() {
   console.log('OpenCV.js is readyyyyy.');
}

$( document ).ready(function() {
    console.log('document ready');

    video = document.getElementById("videoInput"); 
    video.width = 640;
    video.height = 480;
    startCamera();

    socket = io('http://localhost:8080');
    socket.on('connect', function(){
        console.log("Connected...!", socket.connected)
    });
    socket.on('response_back', function(image){
        console.log('got response back!');
        const image_id = document.getElementById('imagebox');
        image_id.src = image;
    });
});

服务器.py

#Import necessary libraries
from flask import Flask, render_template, Response, request , jsonify
import cv2
import random
import json
import numpy as np
import base64
import os , io , sys
from PIL import Image
from flask_socketio import SocketIO, emit
import imutils


#Initialize the Flask app
app = Flask(__name__)
socketio = SocketIO(app=app,logger=True)


@socketio.on('image')
def image(image):

    sbuf = io.StringIO()
    sbuf.write(image)

    # decode and convert into image
    b = io.BytesIO(base64.b64decode(image))
    pimg = Image.open(b)

    ## converting RGB to BGR, as opencv standards
    frame = cv2.cvtColor(np.array(pimg), cv2.COLOR_RGB2BGR)

    # Process the image frame
    # frame = imutils.resize(frame, width=700)
    frame = cv2.flip(frame, 1)
    imgencode = cv2.imencode('.jpg', frame)[1]

    # base64 encode
    stringData = base64.b64encode(imgencode).decode('utf-8')
    b64_src = 'data:image/jpg;base64,'
    stringData = b64_src + stringData

    # emit the frame back
    emit('response_back', stringData)


@app.route('/')
def index():
    return render_template('index.html')

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=8080, debug=False)

0 回复 | 直到 4 年前