Windows: Generated command exceeds windows 32k limit and then fails #1522
Comments
|
I'm constantly hitting this every day. |
|
I‘d do myself a favour and develop on WSL. |
|
We have users. https://github.com/fastled/fastled |
|
agree with @zackees it is a Platformio issue. When providing for Windows it has to work. |
|
Today I read somewhere platformio is „just“ a package manager, now it is a build system :) AFAIK platformio uses scons for that. |
|
I built all of FastLED's massive testing infrastructure on platformio. I built a web compiler for FastLED using platformio. I've tuned platformio nearly to the maximum performance on linux through docker. It's so much faster than running on windows. Especially when you throw on ccache, then pre-warm it, then when it's ready to go full steam, the image get's frozen, and shipped to the cloud for distribution. It's an amazing piece of technology - way better than Arduino in pretty much every conceivable way. I make no hesitation to say that FastLED would not be where it is today if it weren't for platformio. |
Since espressif doesn't want to pay Ivan for espressif support I think this issue will remain a constant problem. What would the possibility be that espressif can fix this and push a patch? |
|
@zackees I can't speak for espressif nor for Platformio. If you want to see this fixed, you have to do yourself, or wait and hope Platformio is doing for there IDF users. |
|
It's a WinAPI limitation. The The only workaround I'm aware of is to use special include flags that utilize a common include prefix. To test it out you can create an extra script, for example import os
from platformio import fs
Import("env")
platform = env.PioPlatform()
FRAMEWORK_DIR = fs.to_unix_path(
platform.get_package_dir("framework-arduinoespressif32")
)
IS_INTEGRATION_DUMP = env.IsIntegrationDump()
def shorten_includes(env, node):
if IS_INTEGRATION_DUMP:
# Don't shorten include paths for IDE integrations
return node
includes = [fs.to_unix_path(inc) for inc in env.get("CPPPATH", [])]
shortened_includes = []
generic_includes = []
for inc in includes:
if inc.startswith(FRAMEWORK_DIR):
shortened_includes.append(
"-iwithprefix/"
+ fs.to_unix_path(os.path.relpath(inc, FRAMEWORK_DIR))
)
else:
generic_includes.append(inc)
return env.Object(
node,
CPPPATH=generic_includes,
CCFLAGS=env["CCFLAGS"] + ["-iprefix", FRAMEWORK_DIR] + shortened_includes
)
env.AddBuildMiddleware(shorten_includes)Then enable this script in your [env:lolin32]
platform = espressif32
framework = arduino
board = lolin32
monitor_speed = 115200
extra_scripts =
pre:shorten_includes.py |
|
@valeros Thx for the code snipplet. The approach is to invasive. More complex includes in libs are not working anymore. |
@Jason2866 Mind sharing an example of a complex include that breaks the code above? |
|
@valeros The include |
|
@Jason2866 I believe I know what went wrong, but it's a bit hard to confirm based only on a scarce error message from the log, especially when a big project like Tasmota is compiled. Any chance you could put together a minimal example to reproduce the issue? |
|
@valeros I need a while. But yes i will try to setup a small example where the error happens. |
|
@valeros Here is a small example https://github.com/Jason2866/pio-shortpath |
|
@Jason2866 Thanks for the example, helped a lot. Here is an improved version of the script: import os
from platformio import fs
Import("env")
platform = env.PioPlatform()
target = env.BoardConfig().get("build.mcu", "esp32")
FRAMEWORK_SDK_DIR = fs.to_unix_path(
os.path.join(
platform.get_package_dir("framework-arduinoespressif32"),
"tools",
"sdk",
target,
"include",
)
)
IS_INTEGRATION_DUMP = env.IsIntegrationDump()
def is_framework_subfolder(potential_subfolder):
if (
os.path.splitdrive(FRAMEWORK_SDK_DIR)[0]
!= os.path.splitdrive(potential_subfolder)[0]
):
return False
return os.path.commonpath([FRAMEWORK_SDK_DIR]) == os.path.commonpath(
[FRAMEWORK_SDK_DIR, potential_subfolder]
)
def shorthen_includes(env, node):
if IS_INTEGRATION_DUMP:
# Don't shorten include paths for IDE integrations
return node
includes = [fs.to_unix_path(inc) for inc in env.get("CPPPATH", [])]
shortened_includes = []
generic_includes = []
for inc in includes:
if is_framework_subfolder(inc):
shortened_includes.append(
"-iwithprefix/"
+ fs.to_unix_path(os.path.relpath(inc, FRAMEWORK_SDK_DIR))
)
else:
generic_includes.append(inc)
return env.Object(
node,
CPPPATH=generic_includes,
CCFLAGS=env["CCFLAGS"]
+ ["-iprefix", FRAMEWORK_SDK_DIR]
+ shortened_includes,
ASFLAGS=env["ASFLAGS"]
+ ["-iprefix", FRAMEWORK_SDK_DIR]
+ shortened_includes,
)
env.AddBuildMiddleware(shorthen_includes) |
|
@valeros Thx for the advanced script. Included directly in the platform of my test fork branch. The code where the error happens https://github.com/Jason2866/platform-espressif32/actions/runs/12816490475/job/35737551124 File: https://github.com/Jason2866/platform-espressif32/blob/short_path/builder/frameworks/arduino.py |
|
@Jason2866 I'm not sure how that relative path looks like, but a simple check should suffice: def is_framework_subfolder(potential_subfolder):
if not os.path.isabs(potential_subfolder):
return False
if (
os.path.splitdrive(FRAMEWORK_SDK_DIR)[0]
!= os.path.splitdrive(potential_subfolder)[0]
):
return False
return os.path.commonpath([FRAMEWORK_SDK_DIR]) == os.path.commonpath(
[FRAMEWORK_SDK_DIR, potential_subfolder]
) |
|
@valeros Great! Tasmota does compile successfully with the script! Going to test the Windows example where compile always failed with the script! Thx so far. Will come back when i have done more tests! |
|
All tests successfull. Now code compiles on Windows where it failed before without using this script. Big thank you @valeros import os
from platformio import fs
Import("env")
platform = env.PioPlatform()
target = env.BoardConfig().get("build.mcu", "esp32")
FRAMEWORK_SDK_DIR = fs.to_unix_path(
os.path.join(
platform.get_package_dir("framework-arduinoespressif32"),
"tools",
"sdk",
target,
"include",
)
)
IS_INTEGRATION_DUMP = env.IsIntegrationDump()
def is_framework_subfolder(potential_subfolder):
if not os.path.isabs(potential_subfolder):
return False
if (
os.path.splitdrive(FRAMEWORK_SDK_DIR)[0]
!= os.path.splitdrive(potential_subfolder)[0]
):
return False
return os.path.commonpath([FRAMEWORK_SDK_DIR]) == os.path.commonpath(
[FRAMEWORK_SDK_DIR, potential_subfolder]
)
def shorthen_includes(env, node):
if IS_INTEGRATION_DUMP:
# Don't shorten include paths for IDE integrations
return node
includes = [fs.to_unix_path(inc) for inc in env.get("CPPPATH", [])]
shortened_includes = []
generic_includes = []
for inc in includes:
if is_framework_subfolder(inc):
shortened_includes.append(
"-iwithprefix/"
+ fs.to_unix_path(os.path.relpath(inc, FRAMEWORK_SDK_DIR))
)
else:
generic_includes.append(inc)
return env.Object(
node,
CPPPATH=generic_includes,
CCFLAGS=env["CCFLAGS"]
+ ["-iprefix", FRAMEWORK_SDK_DIR]
+ shortened_includes,
ASFLAGS=env["ASFLAGS"]
+ ["-iprefix", FRAMEWORK_SDK_DIR]
+ shortened_includes,
)
env.AddBuildMiddleware(shorthen_includes) |
|
Use the script only for Windows. On Mac i see a dramatic increase in compile time. Probably the same happens under Linux (did not test). |
Hi there, I'm a dev on FastLED. I'm cannot build esp32c6 on windows due to command limit of 32k being exceeded:
Repro steps.
git clone https://github.com/fastled/fastled5442a57c6603baac823bac885f2f10753bca2ff9pio runResult
This is the generated command, running this with
wc -mshows that the size is 34k, larger than the 32k windows limit.It would be helpful if a lot of these same path prefixes were instead folded into an environmental variable.
The text was updated successfully, but these errors were encountered: