Dynamic prompt updates in screen example

[jcelerier]

I'm trying to make the prompt in the screen example evolve over time.
However, if I move

        stream.prepare(
            prompt=prompt,
            negative_prompt=negative_prompt,
            num_inference_steps=50,
            guidance_scale=guidance_scale,
            delta=delta,
        )

from before the main loop to within it, just before the call to

        output_images = stream.stream(
            input_batch.to(device=stream.device, dtype=stream.dtype)
        ).cpu()

then my output becomes pretty much some random uniform color.

If I add a timer to call stream.prepare for instance once every second, then once every second I get some grey / blank frame in the middle of the stream. How can I make this more fluid and continuous ?

[olegchomp]

There is function for updating prompt only without preparing the whole stream

StreamDiffusion/src/streamdiffusion/pipeline.py

Lines 255 to 262 in 765d710

	def update_prompt(self, prompt: str) -> None:
	encoder_output = self.pipe.encode_prompt(
	prompt=prompt,
	device=self.device,
	num_images_per_prompt=1,
	do_classifier_free_guidance=False,
	)
	self.prompt_embeds = encoder_output[0].repeat(self.batch_size, 1, 1)

[jcelerier]

hmm does not seem to work very well here if I replace my call to prepare:

[E] 3: [executionContext.cpp::validateInputBindings::2046] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::validateInputBindings::2046, condition: profileMinDims.d[i] <= dimensions.d[i]. Supplied binding dimension [2,77,1024] for bindings[2] exceed min ~ max range at index 0, maximum dimension in profile is 4, minimum dimension in profile is 4, but supplied dimension is 2.

this gives me a lot of visual artifacts too

[olegchomp]

i mean, you need to prepare stream anyway, but after you can change prompt with function above.

[jcelerier]

I do, I first have a call to prepare() unchanged from screen.py, and then I call stream.update_prompt("blah") whenever I get a new prompt