Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(prod/jenkins): debug jenkins start oom #1356

Merged

Conversation

purelind
Copy link
Contributor

@purelind purelind commented Dec 1, 2024

debug jenkins start oom

Copy link
Contributor

ti-chi-bot bot commented Dec 1, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from purelind, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot requested a review from wuhuizuo December 1, 2024 06:35
@ti-chi-bot ti-chi-bot bot added area/apps env/prod will deploy on the main product cluster labels Dec 1, 2024
Copy link
Contributor

ti-chi-bot bot commented Dec 1, 2024

I have already done a preliminary review for you, and I hope to help you do a better job.

Based on the title and description of the PR, it seems that the main change is to add a JVM flag -XX:+HeapDumpOnOutOfMemoryError to the Jenkins values file to generate heap dump when the Jenkins JVM runs out of memory.

The change itself looks fine, but there are a few potential issues that should be addressed before merging.

  1. Is there any reason why the Jenkins JVM is running out of memory? This should be investigated and fixed if possible. Adding the heap dump flag is a good way to diagnose the issue, but it should not be considered a permanent solution.

  2. The heap dump file generated by this flag can be quite large, so it's important to ensure that there is enough disk space available to store the file.

  3. The log file generated by the -Xlog flag can also consume a lot of disk space, so it's important to monitor the log files and rotate them regularly.

As for suggestions, it would be good to add some comments to the Jenkins values file explaining why this flag was added and how to use the heap dump file for debugging. Additionally, it might be helpful to add some monitoring/alerting to notify when the heap dump or log files are taking up too much disk space.

@ti-chi-bot ti-chi-bot bot added the size/XS label Dec 1, 2024
@purelind purelind merged commit ee13f9f into PingCAP-QE:main Dec 1, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/apps env/prod will deploy on the main product cluster size/XS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant