How do I put videos on the screen?

[chlghdwns]

https://slint.dev/success/sksignet-evcharger

The picture above is one of the screens on the address. If you look at the center of the picture on the left, it looks like a video, so I wonder how to implement it.

I tried to implement using gstreamerrate, but I noticed that I had to install LLVM, VCPKG, etc., and I tried to implement it after installing LLVM, VCPKG, etc., but I couldn't implement it for various reasons, such as the error that I couldn't keep finding the .dll file in vscode.

Can I ask you to share if you know how to implement a video in Slint or if anyone has sample examples?

[ogoffart]

The way to display video is with an Image that has a changing source.

We have an example using ffmpeg in https://github.com/slint-ui/slint/tree/master/examples/ffmpeg and using gstreamer in https://github.com/slint-ui/slint/tree/master/examples/gstreamer-player

[chlghdwns]

The way to display video is with an Image that has a changing source.

We have an example using ffmpeg in https://github.com/slint-ui/slint/tree/master/examples/ffmpeg and using gstreamer in https://github.com/slint-ui/slint/tree/master/examples/gstreamer-player

Thank you so much for your reply. I'll give it a try!!

[chlghdwns]

The way to display video is with an Image that has a changing source.

We have an example using ffmpeg in https://github.com/slint-ui/slint/tree/master/examples/ffmpeg and using gstreamer in https://github.com/slint-ui/slint/tree/master/examples/gstreamer-player

I ran it using the given ffmpeg example sample and checked it.

But there's a problem. The sound and the video don't match.

It was confirmed that the current video plays about a second faster than the sound.

Which part should I fix to fix this part?

[chlghdwns]

The way to display video is with an Image that has a changing source.

We have an example using ffmpeg in https://github.com/slint-ui/slint/tree/master/examples/ffmpeg and using gstreamer in https://github.com/slint-ui/slint/tree/master/examples/gstreamer-player

/// Cargo.toml

[package]
name = "test1234"
version = "0.1.0"
edition = "2021"

[dependencies]
slint = { version = "1.6.0" }
anyhow = { version = "1.0" }
futures = { version = "0.3.28" }

gst = {package = "gstreamer", version = "0.22.6"}
gst-audio = {package = "gstreamer-audio", version = "0.22.6" }
gst-video = {package = "gstreamer-video", version = "0.22.6" }
gst-app = {package = "gstreamer-app", version = "0.22.6" }

[build-dependencies]
slint-build = { version = "1.6.0" }

/// main.rs

// Copyright © SixtyFPS GmbH [email protected]
// SPDX-License-Identifier: MIT

slint::include_modules!();

use anyhow::{bail, Result};
use gst::prelude::*;
use gst_video::VideoFrameExt;
use std::env;

fn try_gstreamer_video_frame_to_pixel_buffer(
frame: &gst_video::VideoFrame<gst_video::video_frame::Readable>,
) -> Result<slint::SharedPixelBufferslint::Rgb8Pixel> {
match frame.format() {
gst_video::VideoFormat::Rgb => {
let mut slint_pixel_buffer =
slint::SharedPixelBuffer::slint::Rgb8Pixel::new(frame.width(), frame.height());
frame
.buffer()
.copy_to_slice(0, slint_pixel_buffer.make_mut_bytes())
.expect("Unable to copy to slice!"); // Copies!
Ok(slint_pixel_buffer)
}
_ => {
bail!(
"Cannot convert frame to a slint RGB frame because it is format {}",
frame.format().to_str()
)
}
}
}

fn main() {
let app = App::new().unwrap();
let app_weak = app.as_weak();

gst::init().unwrap();

// Get the current working directory path and set the relative path of the video.mp4 file
// And if you want to use a local file, you have to erase _ before video_uri.
let current_dir = env::current_dir().expect("Could not get current directory");
let video_path = current_dir.join("src/video/video.mp4");
let _video_uri = format!("file:///{}", video_path.to_str().expect("Could not convert path to string"));

// Create a new playbin element
let source = gst::ElementFactory::make("playbin")
    .build()
    .expect("Could not create playbin element.");

// Set the URI property to the path of the video file
//source.set_property("uri", video_uri.as_str());
source.set_property("uri", "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4");

let width: u32 = 1024;
let height: u32 = 1024;

let appsink = gst_app::AppSink::builder()
    .caps(
        &gst_video::VideoCapsBuilder::new()
            .format(gst_video::VideoFormat::Rgb)
            .width(width as i32)
            .height(height as i32)
            .build(),
    )
    .build();

// No need to create a custom pipeline, playbin handles it internally
let pipeline = source.downcast::<gst::Pipeline>().unwrap();

appsink.set_callbacks(
    gst_app::AppSinkCallbacks::builder()
        .new_sample(move |appsink| {
            let sample = appsink.pull_sample().map_err(|_| gst::FlowError::Eos)?;
            let buffer = sample.buffer_owned().unwrap(); // Probably copies!
            let video_info =
                gst_video::VideoInfo::builder(gst_video::VideoFormat::Rgb, width, height)
                    .build()
                    .expect("couldn't build video info!");
            let video_frame =
                gst_video::VideoFrame::from_buffer_readable(buffer, &video_info).unwrap();
            let slint_frame = try_gstreamer_video_frame_to_pixel_buffer(&video_frame)
                .expect("Unable to convert the video frame to a slint video frame!");

            app_weak
                .upgrade_in_event_loop(|app| {
                    app.set_video_frame(slint::Image::from_rgb8(slint_frame))
                })
                .unwrap();

            Ok(gst::FlowSuccess::Ok)
        })
        .build(),
);

// Connect the appsink to the playbin element
pipeline.set_property("video-sink", &appsink);

pipeline
    .set_state(gst::State::Playing)
    .expect("Unable to set the pipeline to the `Playing` state");

app.run().unwrap();

}

/// scene.slint

// Copyright © SixtyFPS GmbH [email protected]
// SPDX-License-Identifier: MIT

import { VerticalBox, StyleMetrics, Palette } from "std-widgets.slint";

export component App inherits Window {
in property video-frame <=> image.source;
in property playing;

pure callback toggle-pause-play();

preferred-width: 960px;
preferred-height: 540px;
min-width: 500px;
min-height: 300px;
title: "Slint GStreamer Video Playback Example";
background: #000000;

states [
    shown when area.has-hover || animation-tick() < 5s : {
        controls.opacity: 1;
        in {
            animate controls.opacity {
                duration: 50ms;
            }
        }
    }
    hidden when !area.has-hover: {
        controls.opacity: 0;
        in {
            animate controls.opacity {
                delay: 3s;
                duration: 500ms;
            }
        }
    }
]

VerticalBox {
    image := Image {
        width: 960px;
        height: 540px;
    }
}

area := TouchArea {
    width: 50%;
    height: self.preferred-height;
    y: root.height - self.height - 40px;
    controls := Rectangle {
        border-radius: 4px;
        background: Palette.color-scheme == ColorScheme.dark ? #3737378c : #ffffff82;

        Image {
            width: 64px;
            height: 64px;
            source: root.playing ? @image-url("img/pause.svg") : @image-url("img/play.svg");
        }

        TouchArea {
            clicked => {
                root.toggle-pause-play();
            }
        }
    }
}

}

When I worked on the latest version of gstreamer, this made it work. I hope it helps a little :)

[wuanzhuan]

Currently, the performance of using the image element to play videos doesn't seem ideal. Are there any plans to provide a video element?

[tronical]

We've got an example that uses gstreamer to generate OpenGL textures and import them into slint::Image. This isn't perfect as there is still an extra render pass for the yuv to rgb conversion, but this is a good start and shows a path of how we can make this even better in the future.

Eventually I'm thinking we should add the ability to import dma buffers (rgb and common yuv formats), iosurfaces, etc.

https://github.com/slint-ui/slint/blob/master/examples/gstreamer-player/egl_integration.rs

[wuanzhuan]

We've got an example that uses gstreamer to generate OpenGL textures and import them into slint::Image. This isn't perfect as there is still an extra render pass for the yuv to rgb conversion, but this is a good start and shows a path of how we can make this even better in the future.

Eventually I'm thinking we should add the ability to import dma buffers (rgb and common yuv formats), iosurfaces, etc.

https://github.com/slint-ui/slint/blob/master/examples/gstreamer-player/egl_integration.rs

I believe a better approach would be to directly render the video stream in a specified window, rather than playing frame-by-frame image textures. Therefore, I think a separate video element is still necessary.

[tronical]

It's possible to implement this on the application level. We have a customer who did this by using the rendering notifier API, punched a whole into the scene using a clearing fragment shader in the AfterRendering phase, and then placed a separate video window underneath.

Could you elaborate what advantages this approach offers? Are you concerned about embedded devices where the video decoder can't decode into a GPU accessible buffer?

[wuanzhuan]

It's possible to implement this on the application level. We have a customer who did this by using the rendering notifier API, punched a whole into the scene using a clearing fragment shader in the AfterRendering phase, and then placed a separate video window underneath.

Could you elaborate what advantages this approach offers? Are you concerned about embedded devices where the video decoder can't decode into a GPU accessible buffer?

According to my understanding, rendering a video stream through image involves first decoding and rendering the video stream into individual frames, and then displaying those frames on the screen via the image element. In contrast, directly rendering the video stream to the screen offers the following advantages:

1. Reduced processing steps.
2. Partial updates for video stream rendering frames.
3. easy to use