Seamless horizontal tiling depthmap

[thygate]

Request made in AUTOMATIC1111/stable-diffusion-webui#4252 (comment)

@Ladypoly's Technique : https://imgur.com/a/PMDJF4S

My understanding :

• Load input1
• generate additional rgb input image from input1 with horizontal halves flipped -> input2
• generate depthmaps (manually) -> depth1, depth2
• swap left and right in depth2
• blend center of depth1 onto depth2
• generate mask with a horizontal gradient, 1 in middle, falling off to 0 at each side
• blend depth1 and depth2 with mask

Proof of concept, but not quite there yet I assume.

EDIT: Include extra step (3b, 3c) to blend center of depth1 into lr-swapped depth2
The tapermult in htile_gradientmask(intput_array, tapermult=1) allows for making a narrower mask.
(Using a mask that is too narrow for this step results in a visible seam.)

# LadyPoly's technique for horizontal tiling depth maps (https://imgur.com/a/PMDJF4S)
# implementation attempt by thygate

import os
from PIL import Image
import numpy as np

# swap left and right halves in PIL image
def htile_swap_lr_pil(input_image):
    # Get the width and height of the input image
    width, height = input_image.size

    # Split the input image into two halves
    left_half = input_image.crop((0, 0, width//2, height))
    right_half = input_image.crop((width//2, 0, width, height))

    # Swap the left and right halves
    swapped_image = Image.new("RGB", (width, height))
    swapped_image.paste(right_half, (0, 0))
    swapped_image.paste(left_half, (width//2, 0))

    return swapped_image

# swap left and right halves in numpy depthmap
def htile_swap_lr_np(input_array):
    # Get the dimensions of the input array
    height, width = input_array.shape

    # Split the input array into two halves
    left_half = input_array[:, :width//2]
    right_half = input_array[:, width//2:]

    # the left and right halves
    swapped_array = np.concatenate((right_half, left_half), axis=1)

    return swapped_array

# create a horizontal linear gradient, 1 in middle, taper off to 0 at each side
def htile_gradientmask(intput_array, tapermult=1):
    # Get the dimensions of the input array
    height, width = intput_array.shape

    # Generate a meshgrid of the same size as the image
    x, y = np.meshgrid(np.linspace(0, 1, width), np.linspace(0, 1, height))
    # Define the center point of the gradient
    center_x = 0.5
    # Calculate the distance from the center point
    distance = np.abs(x - center_x)
    # Calculate the blend factors by linearly interpolating between 1 and 0
    blend_factors = 1 - np.clip(2 * distance * tapermult, 0, 1)
    # Normalize the blend factors so that the maximum value is 1
    blend_factors /= np.max(blend_factors)

    return blend_factors

# blend two depthmaps 
def htile_blend_np(input1, input2, blend_factors):
    blended_array = (1 - blend_factors) * input1 + blend_factors * input2

    return blended_array



# load input1
input1 = Image.open("input1.png")

# 1.   generate additional rgb input image with horizontal halves flipped -> input2
input2 = htile_swap_lr_pil(input1)
input2.save("input2.png")

# 2.   generate depthmaps
if not os.path.isfile("depth1.png") or not os.path.isfile("depth2.png"):
    print("Please generate depth1.png and depth2.png from input1.png and input2.png (16 bit png's !)\n")
    exit()
depth1 = np.asarray(Image.open("depth1.png"), dtype="float")
depth2 = np.asarray(Image.open("depth2.png"), dtype="float")

# 3.   swap left and right in numpy depthmap2
depth2 = htile_swap_lr_np(depth2)

# 3b.  generate narrower(?) mask
blend_factors = htile_gradientmask(depth1, 2)
# uncomment to save blend mask 
Image.fromarray(np.uint8(blend_factors * 255)).save("blend_factors1.png")

# 3c.  blend center of depth1 into depth2
depth2 = htile_blend_np(depth2, depth1, blend_factors)
Image.fromarray(np.uint16(depth2)).save("depth2_swapped.png")

# 4.   generate full width mask
blend_factors = htile_gradientmask(depth1)
# uncomment to save blend mask 
Image.fromarray(np.uint8(blend_factors * 255)).save("blend_factors2.png")

# 5.   final blend 
depth = htile_blend_np(depth1, depth2, blend_factors)
Image.fromarray(np.uint16(depth)).save("output.png")

[thygate]

@Ladypoly Can you take a look and maybe play around with this code and tell me what's missing, because my output still has a seam in the middle ..

EDIT: I've included the extra step of blending the center of depth1 into swapped depth2 in the posted code, new results are better, but the masks might still require some fine tuning ..

I am using the Asymmetric Tiling extension to generate the initial image.

input 1 :

depth2_swapped :

output :

blend_factors1 :

blend_factors2 :

[Ladypoly]

How can I test the script?
For me both depth maps looked very good without any seam.

[thygate]

It's looking pretty good, but I still have a small seam on the tile border.

It was easier to prototype this outside of the webui, so the script is standalone, simply save it to a .py file, and run it from the commandline with python scriptname.py. It expects input1.png it the same folder, and will ask to generate depth1 & depth2 before it can continue.

[thygate]

I can't tell if it's better than the original depthmap .. I must still be doing something wrong ..

[Ladypoly]

This seams like a problem on the base shape of the 3d object. Are you using this addon in blender?
https://github.com/Ladypoly/Serpens-Bledner-Addons/blob/main/Serpens3/importdepthmap_1.1.1.zip?raw=true

When I run the py script it will directly generate a input2.png file which is just completely white.