Issue with thumbnail generation failure for extremely large images

[guiyanakuang]

Reported by @sunxiang0918

In addition, I also found a BUG related to generating thumbnails.
When using skia to generate pictures (thumbnails), if the size of the picture is too large, the generation will fail and become an empty file with a small size.
On the CrossPaste interface, it is just a blank block.

I roughly looked for the reason, and what was said on the Internet may be related to CPU and memory:
https://groups.google.com/g/skia-discuss/c/A47-OtBOnKU
https://groups.google.com/g/skia-discuss/c/886WuVcQUGo
https://stackoverflow.com/questions/52206463/large-image-resizing-shows-blank

I tried it a little bit myself.
In the environment of my macmini (intel-i5 + 32g), the maximum size is 2317023170 (sqrt(uint32/8)), and the program reports an error if it exceeds the size.
In my mbp (apple m2 + 16g) environment, I tested 128k128k and it still works...
Therefore, I have not specifically tested what the boundaries and influencing factors are.

@Test
fun testCreateImage() {
    val width = 23170
    val height = 23170
    
    val surface = Surface.makeRasterN32Premul(width, height)
    val canvas = surface.canvas

    // fill gray background
    val paint = Paint().apply { color = Color.makeRGB(128, 128, 128) }
    canvas.drawRect(Rect.makeXYWH(0f, 0f, width.toFloat(), height.toFloat()), paint)

    val img = surface.makeImageSnapshot()

    val result = File("/Users/sun/Downloads/create1111.png").toOkioPath(true)
    
    img.encodeToData()?.bytes?.let {
        result.toNioPath().writeBytes(it)
    }
}

[sunxiang0918]

我又做了一些测试. 我使用了一个 30000 * 23756的 梵高的星空的 图片 作为输入源 7天有效

在JAVA中, 使用 skia 就会出现这个问题.

如果使用的是 thumbnailator 库, 是能成功生成缩略图的,不过比较慢.

    Thumbnails.of("/Users/sun/Downloads/Van_Gogh_-_Starry_Night.jpg")
        .size(200,200)
        .toFile("/Users/sun/Downloads/preview1.jpg")

如果使用的是OpenCV, 能生成成功,比thumbnailator稍快一点.

        <dependency>
            <groupId>org.openpnp</groupId>
            <artifactId>opencv</artifactId>
            <version>4.9.0-0</version>
        </dependency>

fun main() {

    OpenCV.loadLocally()

    val imagePath = "/Users/sun/Downloads/Van_Gogh_-_Starry_Night.jpg"
    
    val image = Imgcodecs.imread(imagePath)

    // 定义缩略图的尺寸
    val thumbnailWidth = 200
    val thumbnailHeight = 200
    val targetImage = Mat(200, 200, image.type())

    Imgproc.resize(image, targetImage, org.opencv.core.Size(thumbnailWidth.toDouble(), thumbnailHeight.toDouble()))
    
    Imgcodecs.imwrite("photo_1_opencv.png", targetImage)
}

在python中, 使用 Pillow 框架,能比较快速的生成缩略图.

from PIL import Image

def create_thumbnail(input_image_path, output_image_path, size=(200, 200)):
    with Image.open(input_image_path) as img:
        img.thumbnail(size)
        img.save(output_image_path)

if __name__ == '__main__':
    input_path = '/Users/sun/Downloads/Van_Gogh_-_Starry_Night.jpg'  # 输入图片路径
    output_path = '/Users/sun/Downloads/thumbnail.jpg'  # 输出缩略图路径
    create_thumbnail(input_path, output_path)

在swift中, 使用了 Cocoa,也能生成缩略图.

import Cocoa

func createThumbnail(for image: NSImage, size: NSSize) -> NSImage? {
    let thumbnail = NSImage(size: size)
    thumbnail.lockFocus()
    image.draw(in: NSRect(origin: .zero, size: size))
    thumbnail.unlockFocus()
    return thumbnail
}

func saveImage(_ image: NSImage, to url: URL) {
    guard let data = image.tiffRepresentation,
          let bitmap = NSBitmapImageRep(data: data),
          let pngData = bitmap.representation(using: .png, properties: [:]) else {
        print("Failed to convert image to PNG")
        return
    }

    do {
        try pngData.write(to: url)
        print("Thumbnail saved to \(url.path)")
    } catch {
        print("Error saving thumbnail: \(error)")
    }
}


let imagePath = "/Users/sun/Downloads/Van_Gogh_-_Starry_Night.jpg" // 完整路径
let thumbnailSize = NSSize(width: 200, height: 200)

if let image = NSImage(contentsOfFile: imagePath) {
    if let thumbnail = createThumbnail(for: image, size: thumbnailSize) {
        let desktopURL = FileManager.default.urls(for: .desktopDirectory, in: .userDomainMask).first!
        let fileURL = desktopURL.appendingPathComponent("thumbnail.png")
        saveImage(thumbnail, to: fileURL)
    } else {
        print("Failed to create thumbnail")
    }
} else {
    print("Image not found: \(imagePath)")
}

所以,有没有可能,当判断到图片尺寸比较大的时候, 就使用MacosApi.swift里面的方法 来生成缩略图?

[guiyanakuang]

所以,有没有可能,当判断到图片尺寸比较大的时候, 就使用MacosApi.swift里面的方法 来生成缩略图?

Certainly, if using the native API on Mac is faster, I even support using MacosApi by default on Mac to implement this.

[guiyanakuang]

• ✨ Implement native macOS API for thumbnail generation #1907

Improving Thumbnail Creation on Windows and Linux

we recognize the need for similar improvements on Windows and Linux platforms. This issue is to track potential enhancements for these operating systems.

Goals:

1. Faster thumbnail generation
2. Support for extremely large images

Potential Approaches:

• Windows: Investigate using Windows Imaging Component (WIC) or other native Windows APIs
• Linux: Explore options like libvips or ImageMagick for efficient image processing

Current Status:

This is not currently a high-priority item for the core team, but we welcome contributions from the community.

[sunxiang0918]

我对比了一下 ImageMagick和libvips. 在我的环境中, 相同的文件和相同的设置, libvips比ImageMagick快了3倍 (2.64s VS 8.29s)

因此,我着重测试了一下通过JAVA调用 LibVips的方法和性能.
发现,使用JNA调用( jlibvips、 jvips) 非常的慢 (平均18s).
但是如果使用JDK22的FFM(Foreign Function and Memory) API, 性能几乎和命令行是一样的. vips-ffm
供参考~

[guiyanakuang]

Exciting feature! It looks like JDK 22 can directly access off-heap memory, avoiding copying (which should be the main reason for slowness).

[guiyanakuang]

CrossPaste is currently bound to JBR on the desktop because JBR has fixed many clipboard-related bugs, while other open-source JDKs like OpenJDK have largely ignored these issues as they focus on the server side. The latest JBR supports JDK 21, so to use this new feature, you may need to wait for the next long-term support version.

[sunxiang0918]

是的, 所以可能ffm暂时还行不通.
另一条路, 使用 Runtime.getRuntime,通过java调用shell命令, 避免图片的读取.

[CompileFuture2024]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Yes, so maybe ffm won't work for the time being.
Another way is to use Runtime.getRuntime to call the shell command through java to avoid reading the image.